Mostafa Mokhtar created IMPALA-6792:
---------------------------------------
Summary: Appears to be a memory leak in orphaned fragments
Key: IMPALA-6792
URL: https://issues.apache.org/jira/browse/IMPALA-6792
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Mostafa Mokhtar
Assignee: Tim Armstrong
Attachments: Screen Shot 2018-04-02 at 12.08.54 PM.png, memory_leak.txt
On Impala daemons that have orphaned fragments it appears that memory continued
to grow after query got cancelled, indicating of a memory leak.
Memz page indicate that some fragments continue to run.
{code}
Memory Usage
Memory consumption / limit: 101.48 GB / 64.00 GB
Breakdown
Process: memory limit exceeded. Limit=64.00 GB Total=101.51 GB Peak=101.51 GB
Buffer Pool: Free Buffers: Total=0
Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=-71.09 MB
Data Stream Service Queue: Limit=3.20 GB Total=0 Peak=875.26 KB
Data Stream Manager Early RPCs: Total=0 Peak=23.03 MB
TCMalloc Overhead: Total=760.53 MB
Free Disk IO Buffers: Total=1.24 GB Peak=1.30 GB
RequestPool=root.default: Total=909.59 MB Peak=11.91 GB
Query(6c4aa5355486f3f6:5125570400000000): Limit=6.00 GB Reservation=69.50
MB ReservationLimit=4.80 GB OtherMemory=67.90 MB Total=137.40 MB Peak=400.25 MB
Unclaimed reservations: Reservation=34.00 MB OtherMemory=0 Total=34.00 MB
Peak=90.14 MB
Fragment 6c4aa5355486f3f6:5125570400000845: Reservation=35.50 MB
OtherMemory=31.87 MB Total=67.37 MB Peak=67.37 MB
Runtime Filter Bank: Reservation=512.00 KB ReservationLimit=512.00 KB
OtherMemory=0 Total=512.00 KB Peak=512.00 KB
AGGREGATION_NODE (id=12): Total=4.00 KB Peak=4.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
AGGREGATION_NODE (id=11): Total=17.12 KB Peak=17.12 KB
Exprs: Total=17.12 KB Peak=17.12 KB
HASH_JOIN_NODE (id=10): Reservation=17.00 MB OtherMemory=66.25 KB
Total=17.06 MB Peak=17.06 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB
Peak=13.12 KB
EXCHANGE_NODE (id=19): Reservation=10.59 MB OtherMemory=6.69 MB
Total=17.29 MB Peak=17.29 MB
KrpcDeferredRpcs: Total=6.69 MB Peak=6.69 MB
HASH_JOIN_NODE (id=9): Reservation=18.00 MB OtherMemory=34.25 KB
Total=18.03 MB Peak=18.10 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=9): Total=13.12 KB Peak=45.12 KB
Hash Join Builder (join_node_id=9) Exprs: Total=13.12 KB Peak=13.12
KB
EXCHANGE_NODE (id=17): Reservation=14.44 MB OtherMemory=0 Total=14.44
MB Peak=14.45 MB
KrpcDeferredRpcs: Total=0 Peak=2.42 MB
EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=776.00
KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=20): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=18.24 KB Peak=2.89 MB
Fragment 6c4aa5355486f3f6:5125570400000047: Reservation=0 OtherMemory=0
Total=0 Peak=41.25 MB
HDFS_SCAN_NODE (id=4): Total=0 Peak=18.75 MB
KrpcDataStreamSender (dst_id=17): Total=0 Peak=22.19 MB
CodeGen: Total=0 Peak=222.50 KB
Fragment 6c4aa5355486f3f6:512557040000033f: Reservation=0 OtherMemory=0
Total=0 Peak=241.80 MB
HASH_JOIN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=2.00 MB
Hash Join Builder (join_node_id=8): Total=0 Peak=21.12 KB
HASH_JOIN_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=1.99 MB
Hash Join Builder (join_node_id=7): Total=0 Peak=21.12 KB
HASH_JOIN_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=128.08
MB
Hash Join Builder (join_node_id=6): Total=0 Peak=21.12 KB
HDFS_SCAN_NODE (id=0): Total=0 Peak=106.87 MB
EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=2.46 MB
KrpcDeferredRpcs: Total=0 Peak=388.27 KB
EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=18): Total=0 Peak=2.61 MB
CodeGen: Total=0 Peak=3.61 MB
Fragment 6c4aa5355486f3f6:51255704000002e8: Reservation=0 OtherMemory=0
Total=0 Peak=6.71 MB
HDFS_SCAN_NODE (id=2): Total=0 Peak=3.44 MB
KrpcDataStreamSender (dst_id=14): Total=0 Peak=3.23 MB
CodeGen: Total=0 Peak=241.50 KB
Fragment 6c4aa5355486f3f6:51255704000005ea: Reservation=0
OtherMemory=36.02 MB Total=36.02 MB Peak=46.83 MB
HDFS_SCAN_NODE (id=5): Total=9.55 MB Peak=18.17 MB
KrpcDataStreamSender (dst_id=19): Total=26.46 MB Peak=28.87 MB
CodeGen: Total=234.00 B Peak=52.50 KB
Query(b44c5962ad16140:4bd217e100000000): Limit=6.00 GB Reservation=660.28
MB ReservationLimit=4.80 GB OtherMemory=111.92 MB Total=772.20 MB Peak=778.99 MB
Unclaimed reservations: Reservation=3.94 MB OtherMemory=0 Total=3.94 MB
Peak=120.66 MB
Fragment b44c5962ad16140:4bd217e100000bbb: Reservation=0
OtherMemory=49.74 KB Total=49.74 KB Peak=1.41 MB
SORT_NODE (id=16): Total=8.00 KB Peak=8.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
AGGREGATION_NODE (id=27): Total=25.12 KB Peak=25.12 KB
Exprs: Total=25.12 KB Peak=25.12 KB
EXCHANGE_NODE (id=26): Reservation=0 OtherMemory=0 Total=0 Peak=0
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=28): Total=3.88 KB Peak=3.88 KB
CodeGen: Total=4.74 KB Peak=1.37 MB
Fragment b44c5962ad16140:4bd217e100000934: Reservation=332.39 MB
OtherMemory=15.81 MB Total=348.20 MB Peak=358.59 MB
Runtime Filter Bank: Reservation=2.52 MB ReservationLimit=2.52 MB
OtherMemory=0 Total=2.52 MB Peak=2.52 MB
AGGREGATION_NODE (id=15): Total=25.12 KB Peak=25.12 KB
Exprs: Total=25.12 KB Peak=25.12 KB
HASH_JOIN_NODE (id=14): Reservation=1.94 MB OtherMemory=82.25 KB
Total=2.02 MB Peak=2.03 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=14): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=14) Exprs: Total=13.12 KB
Peak=13.12 KB
HASH_JOIN_NODE (id=13): Reservation=1.94 MB OtherMemory=74.25 KB
Total=2.01 MB Peak=2.03 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=13): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=13) Exprs: Total=13.12 KB
Peak=13.12 KB
HASH_JOIN_NODE (id=12): Reservation=160.00 MB OtherMemory=134.25 KB
Total=160.13 MB Peak=160.15 MB
Exprs: Total=47.12 KB Peak=47.12 KB
Hash Join Builder (join_node_id=12): Total=47.12 KB Peak=55.12 KB
Hash Join Builder (join_node_id=12) Exprs: Total=47.12 KB
Peak=47.12 KB
HASH_JOIN_NODE (id=11): Reservation=160.00 MB OtherMemory=62.25 KB
Total=160.06 MB Peak=160.08 MB
Exprs: Total=17.12 KB Peak=17.12 KB
Hash Join Builder (join_node_id=11): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=11) Exprs: Total=13.12 KB
Peak=13.12 KB
HASH_JOIN_NODE (id=10): Reservation=6.00 MB OtherMemory=54.25 KB
Total=6.05 MB Peak=6.07 MB
Exprs: Total=17.12 KB Peak=17.12 KB
Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB
Peak=13.12 KB
EXCHANGE_NODE (id=20): Reservation=14.34 MB OtherMemory=0 Total=14.34
MB Peak=14.34 MB
KrpcDeferredRpcs: Total=0 Peak=1.16 MB
EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=2.11 MB
KrpcDeferredRpcs: Total=0 Peak=232.95 KB
EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=10.97 MB
KrpcDeferredRpcs: Total=0 Peak=805.37 KB
EXCHANGE_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=13.84 MB
KrpcDeferredRpcs: Total=0 Peak=1.69 MB
EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=40.00 KB
KrpcDeferredRpcs: Total=0 Peak=8.10 KB
EXCHANGE_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=26): Total=1021.05 KB Peak=1021.05 KB
CodeGen: Total=48.00 KB Peak=8.65 MB
Fragment b44c5962ad16140:4bd217e10000054e: Reservation=321.00 MB
OtherMemory=20.82 MB Total=341.82 MB Peak=352.83 MB
Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB
OtherMemory=0 Total=1.00 MB Peak=1.00 MB
HASH_JOIN_NODE (id=9): Reservation=320.00 MB OtherMemory=60.25 KB
Total=320.06 MB Peak=320.07 MB
Exprs: Total=22.12 KB Peak=22.12 KB
Hash Join Builder (join_node_id=9): Total=22.12 KB Peak=30.12 KB
Hash Join Builder (join_node_id=9) Exprs: Total=22.12 KB Peak=22.12
KB
EXCHANGE_NODE (id=18): Reservation=12.48 MB OtherMemory=0 Total=12.48
MB Peak=16.40 MB
KrpcDeferredRpcs: Total=0 Peak=4.41 MB
EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=4.35 MB
KrpcDeferredRpcs: Total=0 Peak=1.69 MB
KrpcDataStreamSender (dst_id=20): Total=8.26 MB Peak=19.26 MB
CodeGen: Total=8.02 KB Peak=1.34 MB
Fragment b44c5962ad16140:4bd217e100000048: Reservation=2.95 MB
OtherMemory=75.24 MB Total=78.19 MB Peak=86.44 MB
Runtime Filter Bank: Reservation=1.02 MB ReservationLimit=1.02 MB
OtherMemory=0 Total=1.02 MB Peak=1.02 MB
HASH_JOIN_NODE (id=8): Reservation=1.94 MB OtherMemory=79.25 KB
Total=2.01 MB Peak=2.01 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12
KB
HDFS_SCAN_NODE (id=0): Total=63.00 MB Peak=63.01 MB
EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=18): Total=12.09 MB Peak=20.43 MB
CodeGen: Total=8.66 KB Peak=1.50 MB
Fragment b44c5962ad16140:4bd217e1000002f3: Reservation=0 OtherMemory=0
Total=0 Peak=81.41 MB
HDFS_SCAN_NODE (id=1): Total=0 Peak=60.83 MB
KrpcDataStreamSender (dst_id=19): Total=0 Peak=19.08 MB
CodeGen: Total=0 Peak=335.00 KB
Fragment b44c5962ad16140:4bd217e1000007ef: Reservation=0 OtherMemory=0
Total=0 Peak=26.00 MB
HDFS_SCAN_NODE (id=5): Total=0 Peak=6.29 MB
KrpcDataStreamSender (dst_id=21): Total=0 Peak=19.66 MB
CodeGen: Total=0 Peak=241.50 KB
Untracked Memory: Total=98.71 GB
tcmalloc
------------------------------------------------
MALLOC: 108196856848 (103184.6 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 625752432 ( 596.8 MiB) Bytes in central cache freelist
MALLOC: + 25029120 ( 23.9 MiB) Bytes in transfer cache freelist
MALLOC: + 146667648 ( 139.9 MiB) Bytes in thread cache freelists
MALLOC: + 449097920 ( 428.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 109443403968 (104373.4 MiB) Actual memory used (physical + swap)
MALLOC: + 221184 ( 0.2 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 109443625152 (104373.6 MiB) Virtual address space used
MALLOC:
MALLOC: 6575796 Spans in use
MALLOC: 380 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
System
Physical Memory: 125.52 GB
Transparent Huge Pages Config:
enabled: always [madvise] never
defrag: always [madvise] never
khugepaged defrag: 1
Process and system memory metrics
Name Value Description
memory.anon-huge-page-bytes 640.00 MB Total bytes of anonymous
(a.k.a. transparent) huge pages used by this process.
memory.mapped-bytes 137.22 GB Total bytes of memory mappings in this
process (the virtual memory size).
memory.num-maps 4236 Total number of memory mappings in this process.
memory.rss 68.40 GB Resident set size (RSS) of this process,
including TCMalloc, buffer pool and Jvm.
memory.thp.defrag always [madvise] never The system-wide 'defrag'
setting for Transparent Huge Pages.
memory.thp.enabled always [madvise] never The system-wide 'enabled'
setting for Transparent Huge Pages.
memory.thp.khugepaged-defrag 1 The system-wide 'defrag' setting for
khugepaged.
memory.total-used 101.51 GB Total memory currently used by TCMalloc
and buffer pool.
Buffer pool memory metrics
Name Value Description
buffer-pool.clean-page-bytes 0 Total bytes of clean page memory cached
in the buffer pool.
buffer-pool.clean-pages 0 Total number of clean pages cached in the
buffer pool.
buffer-pool.clean-pages-limit 5.44 GB Limit on number of clean pages cached
in the buffer pool.
buffer-pool.free-buffer-bytes 0 Total bytes of free buffer memory
cached in the buffer pool.
buffer-pool.free-buffers 0 Total number of free buffers cached in
the buffer pool.
buffer-pool.limit 54.40 GB Maximum allowed bytes allocated by the
buffer pool.
buffer-pool.reserved 781.62 MB Total bytes of buffers reserved by
Impala subsystems
buffer-pool.system-allocated 710.53 MB Total buffer memory currently
allocated by the buffer pool.
buffer-pool.unused-reservation-bytes 71.09 MB Total bytes of buffer
reservations by Impala subsystems that are currently unused
JVM aggregate memory metrics
Name Value Description
jvm.total.committed-usage-bytes 1.73 GB Jvm total Committed Usage Bytes
jvm.total.current-usage-bytes 385.19 MB Jvm total Current Usage Bytes
jvm.total.init-usage-bytes 1.88 GB Jvm total Init Usage Bytes
jvm.total.max-usage-bytes 31.23 GB Jvm total Max Usage Bytes
jvm.total.peak-committed-usage-bytes 2.39 GB Jvm total Peak Committed Usage
Bytes
jvm.total.peak-current-usage-bytes 1.08 GB Jvm total Peak Current Usage
Bytes
jvm.total.peak-init-usage-bytes 1.88 GB Jvm total Peak Init Usage Bytes
jvm.total.peak-max-usage-bytes 31.32 GB Jvm total Peak Max Usage Bytes
JVM heap memory metrics
Name Value Description
jvm.heap.committed-usage-bytes 1.67 GB Jvm heap Committed Usage Bytes
jvm.heap.current-usage-bytes 322.88 MB Jvm heap Current Usage Bytes
jvm.heap.init-usage-bytes 1.96 GB Jvm heap Init Usage Bytes
jvm.heap.max-usage-bytes 26.67 GB Jvm heap Max Usage Bytes
jvm.heap.peak-committed-usage-bytes 0 Jvm heap Peak Committed Usage
Bytes
jvm.heap.peak-current-usage-bytes 0 Jvm heap Peak Current Usage
Bytes
jvm.heap.peak-init-usage-bytes 0 Jvm heap Peak Init Usage Bytes
jvm.heap.peak-max-usage-bytes 0 Jvm heap Peak Max Usage Bytes
JVM non-heap memory metrics
Name Value Description
jvm.non-heap.committed-usage-bytes 63.40 MB Jvm non-heap Committed
Usage Bytes
jvm.non-heap.current-usage-bytes 62.31 MB Jvm non-heap Current
Usage Bytes
jvm.non-heap.init-usage-bytes 2.44 MB Jvm non-heap Init Usage Bytes
jvm.non-heap.max-usage-bytes -1.00 B Jvm non-heap Max Usage Bytes
jvm.non-heap.peak-committed-usage-bytes 0 Jvm non-heap Peak Committed
Usage Bytes
jvm.non-heap.peak-current-usage-bytes 0 Jvm non-heap Peak Current Usage
Bytes
jvm.non-heap.peak-init-usage-bytes 0 Jvm non-heap Peak Init Usage
Bytes
jvm.non-heap.peak-max-usage-bytes 0 Jvm non-heap Peak Max Usage
Bytes
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
