[
https://issues.apache.org/jira/browse/IMPALA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nithya updated IMPALA-7361:
---------------------------
Description:
test_heterogeneous_proc_mem_limit fails with the following assertion error
{code:java}
AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
available on host
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00
GB but only 1.00 GB out of 3.00 GB was available. assert None + where
None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not enough memory
available on host \\S+.Needed 2.00 GB but only 1.00 GB out of 2.00 GB was
available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n') + where <function search
at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n Query
aborted:Admission for query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
{code}
stack trace
{code:java}
*Stacktrace*
custom_cluster/test_admission_controller.py:514: in
test_heterogeneous_proc_mem_limit
assert re.search("Queued reason: Not enough memory available on host
\S+.Needed "
E AssertionError: ImpalaBeeswaxException:
E Query aborted:Admission for query exceeded timeout 200ms in pool
default-pool. Queued reason: Not enough memory available on host
<host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
E
E
E assert None
E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for
query exceeded timeout 200ms in pool default-pool. Queued
reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
available.\n\n')
E + where <function search at 0x7f2b4a67c5f0> = re.search
E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
available.\n\n' = str(ImpalaBeeswaxException())
*Standard Error*
08:55:51 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
08:55:52 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
08:55:53 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
08:55:54 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
08:55:55 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:55:58 MainThread: Getting num_known_live_backends from <hostname>:25000
08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
08:55:59 MainThread: Getting num_known_live_backends from<hostname>:25000
08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
08:56:00 MainThread: Getting num_known_live_backends from <hostname>:25000
08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25000
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25001
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25002
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from <hostname>:25010
MainThread: Metric 'statestore.live-backends' has reached desired value: 4
MainThread: Getting num_known_live_backends from <hostname>:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from <hostname>:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from <hostname>:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
use default;
SET sync_ddl=1;
-- executing against localhost:21000
drop database if exists `hs2_db` cascade;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=1;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- connecting to: <hostname>:21002
SET mem_limit=1G;
-- executing async: <hostname>:21002
select sleep(1000);
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- closing connection to: i<hostname>:21002
{code}
was:
test_heterogeneous_proc_mem_limit fails with the following assertion error
{code}
AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
available on host
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00
GB but only 1.00 GB out of 3.00 GB was available. assert None + where
None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not enough memory
available on host \\S+.Needed 2.00 GB but only 1.00 GB out of 2.00 GB was
available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n') + where <function search
at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n Query
aborted:Admission for query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
{code}
stack trace
{code}
*Stacktrace*
custom_cluster/test_admission_controller.py:514: in
test_heterogeneous_proc_mem_limit
assert re.search("Queued reason: Not enough memory available on host
\S+.Needed "
E AssertionError: ImpalaBeeswaxException:
E Query aborted:Admission for query exceeded timeout 200ms in pool
default-pool. Queued reason: Not enough memory available on host
<host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
E
E
E assert None
E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for
query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n')
E + where <function search at 0x7f2b4a67c5f0> = re.search
E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
*Standard Error*
08:55:51 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
08:55:52 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
08:55:53 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
08:55:54 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
08:55:55 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:55:58 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
08:55:59 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
08:56:00 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25001
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25002
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25010
MainThread: Metric 'statestore.live-backends' has reached desired value: 4
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
use default;
SET sync_ddl=1;
-- executing against localhost:21000
drop database if exists `hs2_db` cascade;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=1;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- connecting to:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
SET mem_limit=1G;
-- executing async:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
select sleep(1000);
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- closing connection to:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
{code}
> test_heterogeneous_proc_mem_limit - Assertion Failure
> -----------------------------------------------------
>
> Key: IMPALA-7361
> URL: https://issues.apache.org/jira/browse/IMPALA-7361
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: nithya
> Priority: Major
> Labels: broken-build
>
> test_heterogeneous_proc_mem_limit fails with the following assertion error
>
> {code:java}
> AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
> available on host
> impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed
> 2.00 GB but only 1.00 GB out of 3.00 GB was available. assert None +
> where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not enough
> memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of 2.00 GB
> was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n') + where <function
> search at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n
> Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n' =
> str(ImpalaBeeswaxException())
> {code}
>
> stack trace
> {code:java}
> *Stacktrace*
> custom_cluster/test_admission_controller.py:514: in
> test_heterogeneous_proc_mem_limit
> assert re.search("Queued reason: Not enough memory available on host
> \S+.Needed "
> E AssertionError: ImpalaBeeswaxException:
> E Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued reason: Not enough memory available on host
> <host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
> E
> E
> E assert None
> E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
> enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
> 2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission
> for query exceeded timeout 200ms in pool default-pool. Queued
> reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
> available.\n\n')
> E + where <function search at 0x7f2b4a67c5f0> = re.search
> E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued
> reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
> available.\n\n' = str(ImpalaBeeswaxException())
> *Standard Error*
> 08:55:51 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 08:55:52 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 08:55:53 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 08:55:54 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 08:55:55 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 08:55:58 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 08:55:59 MainThread: Getting num_known_live_backends from<hostname>:25000
> 08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
> 08:56:00 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25000
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25001
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25002
> 08:56:01 MainThread: num_known_live_backends has reached value: 3
> 08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> MainThread: Getting metric: statestore.live-backends from <hostname>:25010
> MainThread: Metric 'statestore.live-backends' has reached desired value: 4
> MainThread: Getting num_known_live_backends from <hostname>:25000
> MainThread: num_known_live_backends has reached value: 3
> MainThread: Getting num_known_live_backends from <hostname>:25001
> MainThread: num_known_live_backends has reached value: 3
> MainThread: Getting num_known_live_backends from <hostname>:25002
> MainThread: num_known_live_backends has reached value: 3
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> use default;
>
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `hs2_db` cascade;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=2G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=3G;
> SET batch_size=0;
> SET num_nodes=1;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=3G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> -- connecting to: <hostname>:21002
> SET mem_limit=1G;
> -- executing async: <hostname>:21002
> select sleep(1000);
>
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET mem_limit=2G;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> select * from functional.alltypesagg, (select 1) B limit 1;
>
> -- closing connection to: i<hostname>:21002
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
