[
https://issues.apache.org/jira/browse/IMPALA-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-6684:
----------------------------------
Assignee: Michael Ho
> Large payload going through exchange results in lots of untracked memory in
> the form of a string
> ------------------------------------------------------------------------------------------------
>
> Key: IMPALA-6684
> URL: https://issues.apache.org/jira/browse/IMPALA-6684
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Affects Versions: Impala 2.11.0
> Reporter: Mostafa Mokhtar
> Assignee: Michael Ho
> Priority: Major
> Attachments: Untracked memory profile.txt,
> heap_profile_growth_40GB.pdf
>
>
> While running a query that shuffles 1GB rows noticed that there is a large
> amount of untracked memory in the form of a string, this issue repros
> with/without KRPC.
> [~tgar] identified that the memory is from *tuple_data_* and that the while
> the payload is transient much of the memory untracked.
> [https://github.com/apache/impala/blob/master/be/src/runtime/row-batch.h#L88]
> {code:java}
> Process: Limit=200.00 GB Total=22.85 GB Peak=201.03 GB
> Buffer Pool: Free Buffers: Total=4.13 GB
> Buffer Pool: Clean Pages: Total=0
> Buffer Pool: Unused Reservation: Total=-2.00 GB
> Data Stream Service Queue: Limit=10.00 GB Total=0 Peak=1.00 GB
> Data Stream Manager Deferred RPCs: Total=0 Peak=27.70 KB
> TCMalloc Overhead: Total=206.90 MB
> Free Disk IO Buffers: Total=1.15 GB Peak=1.89 GB
> RequestPool=root.jenkins: Total=0 Peak=13.32 GB
> RequestPool=root.mmokhtar: Total=13.67 GB Peak=117.15 GB
> Query(ec4cab0940313fdd:d6196e4800000000): Reservation=72.00 MB
> ReservationLimit=160.00 GB OtherMemory=13.60 GB Total=13.67 GB Peak=13.67 GB
> Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB
> Peak=176.00 MB
> Fragment ec4cab0940313fdd:d6196e480000002b: Reservation=34.00 MB
> OtherMemory=4.00 GB Total=4.03 GB Peak=4.03 GB
> AGGREGATION_NODE (id=11): Total=24.00 KB Peak=24.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KB
> HASH_JOIN_NODE (id=10): Reservation=34.00 MB OtherMemory=34.25 KB
> Total=34.03 MB Peak=34.05 MB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
> Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB
> Peak=13.12 KB
> EXCHANGE_NODE (id=20): Reservation=4.00 GB OtherMemory=0 Total=4.00
> GB Peak=4.00 GB
> KrpcDeferredRpcs: Total=0 Peak=0
> EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=22): Total=5.12 KB Peak=5.12 KB
> CodeGen: Total=7.79 KB Peak=1.46 MB
> Fragment ec4cab0940313fdd:d6196e480000000f: Reservation=36.00 MB
> OtherMemory=9.60 GB Total=9.63 GB Peak=9.63 GB
> UNION_NODE (id=0): Total=2.09 GB Peak=2.09 GB
> Exprs: Total=2.07 GB Peak=2.07 GB
> AGGREGATION_NODE (id=13): Reservation=36.00 MB OtherMemory=5.44 GB
> Total=5.48 GB Peak=5.48 GB
> Exprs: Total=5.44 GB Peak=5.44 GB
> EXCHANGE_NODE (id=12): Reservation=0 OtherMemory=0 Total=0 Peak=57.54
> MB
> KrpcDeferredRpcs: Total=0 Peak=8.30 MB
> AGGREGATION_NODE (id=15): Total=17.12 KB Peak=17.12 KB
> Exprs: Total=17.12 KB Peak=17.12 KB
> EXCHANGE_NODE (id=14): Reservation=16.00 KB OtherMemory=0 Total=16.00
> KB Peak=16.00 KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=20): Total=1.03 GB Peak=1.03 GB
> CodeGen: Total=8.04 KB Peak=1.02 MB
> Fragment ec4cab0940313fdd:d6196e4800000007: Reservation=0 OtherMemory=0
> Total=0 Peak=6.36 GB
> AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0
> Peak=5.92 GB
> HDFS_SCAN_NODE (id=1): Total=0 Peak=535.01 MB
> KrpcDataStreamSender (dst_id=12): Total=0 Peak=84.57 MB
> CodeGen: Total=0 Peak=592.00 KB
> Fragment ec4cab0940313fdd:d6196e480000000e: Reservation=0 OtherMemory=0
> Total=0 Peak=2.34 MB
> AGGREGATION_NODE (id=4): Reservation=0 OtherMemory=0 Total=0
> Peak=2.02 MB
> HDFS_SCAN_NODE (id=3): Total=0 Peak=280.00 KB
> KrpcDataStreamSender (dst_id=14): Total=0 Peak=35.82 KB
> CodeGen: Total=0 Peak=622.00 KB
> Fragment ec4cab0940313fdd:d6196e4800000024: Reservation=0 OtherMemory=0
> Total=0 Peak=34.22 MB
> UNION_NODE (id=5): Total=0 Peak=0
> AGGREGATION_NODE (id=17): Reservation=0 OtherMemory=0 Total=0
> Peak=34.02 MB
> EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=64.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0
> Peak=1.96 MB
> EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00
> KB
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=21): Total=0 Peak=103.53 KB
> CodeGen: Total=0 Peak=491.50 KB
> Fragment ec4cab0940313fdd:d6196e4800000023: Reservation=0 OtherMemory=0
> Total=0 Peak=2.67 MB
> AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0
> Peak=2.02 MB
> HDFS_SCAN_NODE (id=8): Total=0 Peak=552.00 KB
> KrpcDataStreamSender (dst_id=18): Total=0 Peak=99.53 KB
> CodeGen: Total=0 Peak=504.50 KB
> Fragment ec4cab0940313fdd:d6196e480000001c: Reservation=0 OtherMemory=0
> Total=0 Peak=280.68 MB
> AGGREGATION_NODE (id=7): Reservation=0 OtherMemory=0 Total=0
> Peak=34.05 MB
> HDFS_SCAN_NODE (id=6): Total=0 Peak=246.54 MB
> KrpcDataStreamSender (dst_id=16): Total=0 Peak=127.53 KB
> CodeGen: Total=0 Peak=474.50 KB
> Untracked Memory: Total=5.70 GB
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
