[
https://issues.apache.org/jira/browse/IMPALA-12135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728498#comment-17728498
]
Riza Suminto commented on IMPALA-12135:
---------------------------------------
I suspect that F06:EXCHANGE SENDER is grabbing and monopolize memory faster
than other query operator in the plan (join build, aggregates), causing other
operator that do need to grow mem reservation to fail.
Thinking of running this test with BATCH_SIZE=64 (default is 1024) to let KRPC
sender do smaller serialization at a time. Will that work? How can I test?
> TestDataStreamSenderTpch.test_krpc_datastream_sender_shuffle fails with
> memory limit exceeded
> ----------------------------------------------------------------------------------------------
>
> Key: IMPALA-12135
> URL: https://issues.apache.org/jira/browse/IMPALA-12135
> Project: IMPALA
> Issue Type: Bug
> Reporter: Tamas Mate
> Assignee: Riza Suminto
> Priority: Major
>
> impala-asf-master-core-erasure-coding build is failing with memory limit
> exceeded
> {code:none}
> query_test/test_datastream_sender.py:36: in
> test_krpc_datastream_sender_shuffle
> self.run_test_case('datastream-sender', vector)
> common/impala_test_suite.py:718: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:656: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:992: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:214: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:369: in __execute_query
> self.wait_for_finished(handle) beeswax/impala_beeswax.py:390: in
> wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Memory limit exceeded: Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-ondemand-0ae4.vpc.cloudera.com:27002 by
> fragment 274c1835003503b5:b6e769a900000009 E Memory left in process limit:
> 8.29 GB E Memory left in query limit: -46.05 MB E
> Query(274c1835003503b5:b6e769a900000000): memory limit exceeded. Limit=1.00
> GB Reservation=19.50 MB ReservationLimit=819.20 MB OtherMemory=1.03 GB
> Total=1.04 GB Peak=1.04 GB
> E Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB
> Peak=75.69 MB
> E Fragment 274c1835003503b5:b6e769a900000007: Reservation=0 OtherMemory=0
> Total=0 Peak=6.68 MB
> E AGGREGATION_NODE (id=11): Reservation=0 OtherMemory=0 Total=0 Peak=2.03
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E HDFS_SCAN_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
> E KrpcDataStreamSender (dst_id=24): Total=0 Peak=14.25 KB
> E RowBatchSerialization: Total=0 Peak=31.00 B
> E Fragment 274c1835003503b5:b6e769a900000006: Reservation=0 OtherMemory=0
> Total=0 Peak=12.78 MB
> E AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=3.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=3.02 MB
> E HDFS_SCAN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=9.72 MB
> E KrpcDataStreamSender (dst_id=22): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E Fragment 274c1835003503b5:b6e769a900000008: Reservation=0 OtherMemory=0
> Total=0 Peak=4.95 MB
> E UNION_NODE (id=7): Total=0 Peak=0
> E AGGREGATION_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=4.77
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=136.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=2.08
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=27): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E CodeGen: Total=0 Peak=1.19 MB
> E CodeGen: Total=0 Peak=1.15 MB
> E CodeGen: Total=0 Peak=1.21 MB
> E Fragment 274c1835003503b5:b6e769a900000003: Reservation=0 OtherMemory=0
> Total=0 Peak=10.68 MB
> E AGGREGATION_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=8.63 MB
> E KrpcDataStreamSender (dst_id=18): Total=0 Peak=6.18 KB
> E RowBatchSerialization: Total=0 Peak=90.00 B
> E CodeGen: Total=0 Peak=1.31 MB
> E Fragment 274c1835003503b5:b6e769a900000004: Reservation=0 OtherMemory=0
> Total=0 Peak=4.13 MB
> E AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=2.10
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.09 MB
> E EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=20): Total=0 Peak=6.19 KB
> E RowBatchSerialization: Total=0 Peak=100.00 B
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000001: Reservation=0 OtherMemory=0
> Total=0 Peak=42.16 MB
> E AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=32.12
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.03 MB
> E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=12.90 MB
> E KrpcDataStreamSender (dst_id=14): Total=0 Peak=262.58 KB
> E RowBatchSerialization: Total=0 Peak=256.49 KB
> E CodeGen: Total=0 Peak=1.27 MB
> E Fragment 274c1835003503b5:b6e769a900000002: Reservation=0 OtherMemory=0
> Total=0 Peak=53.93 MB
> E AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=32.96
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.95 MB
> E AGGREGATION_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=37.53
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=16.37 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=16): Total=0 Peak=4.88 MB
> E RowBatchSerialization: Total=0 Peak=4.87 MB
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000005: Reservation=9.00 MB
> OtherMemory=794.49 MB Total=803.49 MB Peak=887.31 MB
> E UNION_NODE (id=0): Total=227.00 MB Peak=227.00 MB
> E Exprs: Total=226.14 MB Peak=226.14 MB
> E AGGREGATION_NODE (id=17): Reservation=9.00 MB OtherMemory=14.85 MB
> Total=23.85 MB Peak=24.84 MB
> E GroupingAggregator 0: Reservation=9.00 MB OtherMemory=14.85 MB
> Total=23.85 MB Peak=24.84 MB
> E Exprs: Total=14.84 MB Peak=15.83 MB
> E EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=11.02 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=21): Reservation=0 OtherMemory=17.12 KB Total=17.12
> KB Peak=17.12 KB
> E GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
> E Exprs: Total=17.12 KB Peak=17.12 KB
> E EXCHANGE_NODE (id=20): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB
> Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=26): Total=431.20 MB Peak=431.20 MB
> E RowBatchSerialization: Total=431.20 MB Peak=431.20 MB
> E CodeGen: Total=0 Peak=2.73 MB
> E Fragment 274c1835003503b5:b6e769a900000009: Reservation=8.50 MB
> OtherMemory=256.06 MB Total=264.56 MB Peak=264.56 MB
> E AGGREGATION_NODE (id=13): Total=4.00 KB Peak=4.00 KB
> E NonGroupingAggregator 0: Total=4.00 KB Peak=4.00 KB
> E Exprs: Total=4.00 KB Peak=4.00 KB
> E HASH_JOIN_NODE (id=12): Reservation=8.50 MB OtherMemory=34.25 KB
> Total=8.53 MB Peak=8.55 MB
> E Exprs: Total=13.12 KB Peak=13.12 KB
> E Hash Join Builder (join_node_id=12): Total=13.12 KB Peak=21.12 KB
> E Hash Join Builder (join_node_id=12) Exprs: Total=13.12 KB Peak=13.12
> KB
> E EXCHANGE_NODE (id=26): Reservation=256.01 MB OtherMemory=0 Total=256.01
> MB Peak=256.01 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E EXCHANGE_NODE (id=27): Reservation=0 OtherMemory=0 Total=0 Peak=272.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=28): Total=5.12 KB Peak=5.12 KB
> E RowBatchSerialization: Total=0 Peak=0
> E CodeGen: Total=0 Peak=1.87 MB
> E E Memory limit exceeded: Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-ondemand-0ae4.vpc.cloudera.com:27002 by
> fragment 274c1835003503b5:b6e769a900000009 E Memory left in process limit:
> 8.29 GB E Memory left in query limit: -46.05 MB E
> Query(274c1835003503b5:b6e769a900000000): memory limit exceeded. Limit=1.00
> GB Reservation=19.50 MB ReservationLimit=819.20 MB OtherMemory=1.03 GB
> Total=1.04 GB Peak=1.04 GB
> E Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB
> Peak=75.69 MB
> E Fragment 274c1835003503b5:b6e769a900000007: Reservation=0 OtherMemory=0
> Total=0 Peak=6.68 MB
> E AGGREGATION_NODE (id=11): Reservation=0 OtherMemory=0 Total=0 Peak=2.03
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E HDFS_SCAN_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
> E KrpcDataStreamSender (dst_id=24): Total=0 Peak=14.25 KB
> E RowBatchSerialization: Total=0 Peak=31.00 B
> E Fragment 274c1835003503b5:b6e769a900000006: Reservation=0 OtherMemory=0
> Total=0 Peak=12.78 MB
> E AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=3.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=3.02 MB
> E HDFS_SCAN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=9.72 MB
> E KrpcDataStreamSender (dst_id=22): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E Fragment 274c1835003503b5:b6e769a900000008: Reservation=0 OtherMemory=0
> Total=0 Peak=4.95 MB
> E UNION_NODE (id=7): Total=0 Peak=0
> E AGGREGATION_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=4.77
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=136.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=2.08
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=27): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E CodeGen: Total=0 Peak=1.19 MB
> E CodeGen: Total=0 Peak=1.15 MB
> E CodeGen: Total=0 Peak=1.21 MB
> E Fragment 274c1835003503b5:b6e769a900000003: Reservation=0 OtherMemory=0
> Total=0 Peak=10.68 MB
> E AGGREGATION_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=8.63 MB
> E KrpcDataStreamSender (dst_id=18): Total=0 Peak=6.18 KB
> E RowBatchSerialization: Total=0 Peak=90.00 B
> E CodeGen: Total=0 Peak=1.31 MB
> E Fragment 274c1835003503b5:b6e769a900000004: Reservation=0 OtherMemory=0
> Total=0 Peak=4.13 MB
> E AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=2.10
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.09 MB
> E EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=20): Total=0 Peak=6.19 KB
> E RowBatchSerialization: Total=0 Peak=100.00 B
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000001: Reservation=0 OtherMemory=0
> Total=0 Peak=42.16 MB
> E AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=32.12
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.03 MB
> E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=12.90 MB
> E KrpcDataStreamSender (dst_id=14): Total=0 Peak=262.58 KB
> E RowBatchSerialization: Total=0 Peak=256.49 KB
> E CodeGen: Total=0 Peak=1.27 MB
> E Fragment 274c1835003503b5:b6e769a900000002: Reservation=0 OtherMemory=0
> Total=0 Peak=53.93 MB
> E AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=32.96
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.95 MB
> E AGGREGATION_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=37.53
> MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=16.37 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=16): Total=0 Peak=4.88 MB
> E RowBatchSerialization: Total=0 Peak=4.87 MB
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000005: Reservation=9.00 MB
> OtherMemory=794.49 MB Total=803.49 MB Peak=887.31 MB
> E UNION_NODE (id=0): Total=227.00 MB Peak=227.00 MB
> E Exprs: Total=226.14 MB Peak=226.14 MB
> E AGGREGATION_NODE (id=17): Reservation=9.00 MB OtherMemory=14.85 MB
> Total=23.85 MB Peak=24.84 MB
> E GroupingAggregator 0: Reservation=9.00 MB OtherMemory=14.85 MB
> Total=23.85 MB Peak=24.84 MB
> E Exprs: Total=14.84 MB Peak=15.83 MB
> E EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=11.02 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=21): Reservation=0 OtherMemory=17.12 KB Total=17.12
> KB Peak=17.12 KB
> E GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
> E Exprs: Total=17.12 KB Peak=17.12 KB
> E EXCHANGE_NODE (id=20): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB
> Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=26): Total=431.20 MB Peak=431.20 MB
> E RowBatchSerialization: Total=431.20 MB Peak=431.20 MB
> E CodeGen: Total=0 Peak=2.73 MB
> E Fragment 274c1835003503b5:b6e769a900000009: Reservation=8.50 MB
> OtherMemory=256.06 MB Total=264.56 MB Peak=264.56 MB
> E AGGREGATION_NODE (id=13): Total=4.00 KB Peak=4.00 KB
> E NonGroupingAggregator 0: Total=4.00 KB Peak=4.00 KB
> E Exprs: Total=4.00 KB Peak=4.00 KB
> E HASH_JOIN_NODE (id=12): Reservation=8.50 MB OtherMemory=34.25 KB
> Total=8.53 MB Peak=8.55 MB
> E Exprs: Total=13.12 KB Peak=13.12 KB
> E Hash Join Builder (join_node_id=12): Total=13.12 KB Peak=21.12 KB
> E Hash Join Builder (join_node_id=12) Exprs: Total=13.12 KB Peak=13.12
> KB
> E EXCHANGE_NODE (id=26): Reservation=256.01 MB OtherMemory=0 Total=256.01
> MB Peak=256.01 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E EXCHANGE_NODE (id=27): Reservation=0 OtherMemory=0 Total=0 Peak=272.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=28): Total=5.12 KB Peak=5.12 KB
> E RowBatchSerialization: Total=0 Peak=0
> E CodeGen: Total=0 Peak=1.87 MB (1 of 4 similar)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
