[
https://issues.apache.org/jira/browse/IMPALA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244198#comment-17244198
]
Tim Armstrong commented on IMPALA-10251:
----------------------------------------
I'm not going to have time to investigate. I think we'd need to solve this by
either increasing the memory on the test machines, reducing test parallelism
for ASAN, or enabling some kind of admission control for tests (like we did on
the upstream dockerized tests).
> test_decimal_queries and test_tpcds_queries may run out of memory on ASAN
> builds
> --------------------------------------------------------------------------------
>
> Key: IMPALA-10251
> URL: https://issues.apache.org/jira/browse/IMPALA-10251
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Priority: Critical
> Labels: broken-build, flaky
>
> Encountered these test failures due to running out of memory:
> {code:java}
> query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> beeswax | exec_option: {'disable_codegen_rows_threshold': 0,
> 'disable_codegen': 'false', 'decimal_v2': 'true', 'batch_size': 1} |
> table_format:
> orc/def/block]query_test.test_tpcds_queries.TestTpcdsQuery.test_tpcds_q51a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> hs2 | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen':
> 'true', 'decimal_v2': 'true', 'batch_size': 1} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q18a[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q23_1[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q4[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] {code}
> The error is
> {code}
> query_test/test_decimal_queries.py:55: in test_queries
> self.run_test_case('QueryTest/decimal', vector)
> common/impala_test_suite.py:662: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:600: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:909: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Failed to get minimum memory reservation of 68.01 MB on
> daemon impala-ec2-centos74-r5-4xlarge-ondemand-109b.vpc.cloudera.com:27001
> for query 2b41e9326825c48d:15a4d93100000000 due to following error: Failed to
> increase reservation by 68.01 MB because it would exceed the applicable
> reservation limit for the "Process" ReservationTracker:
> reservation_limit=10.20 GB reservation=10.16 GB used_reservation=0
> child_reservations=10.16 GB
> E The top 5 queries that allocated memory under this tracker are:
> E Query(ca4c47a813876f70:d626ca0c00000000): Reservation=9.60 GB
> ReservationLimit=9.60 GB OtherMemory=115.84 MB Total=9.71 GB Peak=9.71 GB
> E Query(3f4e984e6b54204d:2597d87f00000000): Reservation=426.77 MB
> ReservationLimit=9.60 GB OtherMemory=113.19 MB Total=539.95 MB Peak=540.00 MB
> E Query(6c4aca5cba7141b7:cb0d9a4000000000): Limit=512.00 MB
> Reservation=68.03 MB ReservationLimit=409.60 MB OtherMemory=2.59 MB
> Total=70.62 MB Peak=71.93 MB
> E Query(e84c2ac37bac9309:7eddf8bb00000000): Reservation=14.94 MB
> ReservationLimit=9.60 GB OtherMemory=5.55 MB Total=20.48 MB Peak=39.46 MB
> E Query(eb44550f4e6aebbd:4b72c39600000000): Reservation=12.31 MB
> ReservationLimit=9.60 GB OtherMemory=5.85 MB Total=18.16 MB Peak=18.16 MB
> E
> E
> E
> E
> E
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
