[
https://issues.apache.org/jira/browse/IMPALA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-7361:
--------------------------------
Summary: test_heterogeneous_proc_mem_limit: Query aborted: Not enough
memory available on host (was: test_heterogeneous_proc_mem_limit: Query
aborted: )
> test_heterogeneous_proc_mem_limit: Query aborted: Not enough memory
> available on host
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-7361
> URL: https://issues.apache.org/jira/browse/IMPALA-7361
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: nithya
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: broken-build
>
> test_heterogeneous_proc_mem_limit fails with the following assertion error
>
> {code:java}
> AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
> available on host <hostname>:22001.Needed 2.00 GB but only 1.00 GB out of
> 3.00 GB was available. assert None + where None = <function search at
> 0x7f2b4a67c5f0>('Queued reason: Not enough memory available on host
> \\S+.Needed 2.00 GB but only 1.00 GB out of 2.00 GB was available.',
> 'ImpalaBeeswaxException:\n Query aborted:Admission for query exceeded timeout
> 200ms in pool default-pool. Queued reaso...<hostname>:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n') + where <function
> search at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n
> Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB
> out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
> {code}
>
> stack trace
> {code:java}
> *Stacktrace*
> custom_cluster/test_admission_controller.py:514: in
> test_heterogeneous_proc_mem_limit
> assert re.search("Queued reason: Not enough memory available on host
> \S+.Needed "
> E AssertionError: ImpalaBeeswaxException:
> E Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued reason: Not enough memory available on host
> <host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
> E
> E
> E assert None
> E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
> enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
> 2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission
> for query exceeded timeout 200ms in pool default-pool. Queued
> reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
> available.\n\n')
> E + where <function search at 0x7f2b4a67c5f0> = re.search
> E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued
> reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
> available.\n\n' = str(ImpalaBeeswaxException())
> *Standard Error*
> 08:55:51 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 08:55:52 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 08:55:53 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 08:55:54 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 08:55:55 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 08:55:58 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 08:55:59 MainThread: Getting num_known_live_backends from<hostname>:25000
> 08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
> 08:56:00 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25001
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25002
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> MainThread: Getting metric: statestore.live-backends from <hostname>:25010
> MainThread: Metric 'statestore.live-backends' has reached desired value: 4
> MainThread: Getting num_known_live_backends from <hostname>:25000
> MainThread: num_known_live_backends has reached value: 3
> MainThread: Getting num_known_live_backends from <hostname>:25001
> MainThread: num_known_live_backends has reached value: 3
> MainThread: Getting num_known_live_backends from <hostname>:25002
> MainThread: num_known_live_backends has reached value: 3
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
>
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `hs2_db` cascade;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=2G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=3G;
> SET batch_size=0;
> SET num_nodes=1;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=3G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> -- connecting to: <hostname>:21002
> SET mem_limit=1G;
> -- executing async: <hostname>:21002
> select sleep(1000);
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=2G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> -- closing connection to: i<hostname>:21002
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
