nithya created IMPALA-7361:
------------------------------
Summary: test_heterogeneous_proc_mem_limit - Assertion Failure
Key: IMPALA-7361
URL: https://issues.apache.org/jira/browse/IMPALA-7361
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: nithya
test_heterogeneous_proc_mem_limit fails with the following assertion error
{code}
AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
available on host
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00
GB but only 1.00 GB out of 3.00 GB was available. assert None + where
None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not enough memory
available on host \\S+.Needed 2.00 GB but only 1.00 GB out of 2.00 GB was
available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n') + where <function search
at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n Query
aborted:Admission for query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
{code}
stack trace
{code}
*Stacktrace*
custom_cluster/test_admission_controller.py:514: in
test_heterogeneous_proc_mem_limit
assert re.search("Queued reason: Not enough memory available on host
\S+.Needed "
E AssertionError: ImpalaBeeswaxException:
E Query aborted:Admission for query exceeded timeout 200ms in pool
default-pool. Queued reason: Not enough memory available on host
<host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
E
E
E assert None
E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for
query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n')
E + where <function search at 0x7f2b4a67c5f0> = re.search
E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
*Standard Error*
08:55:51 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
08:55:52 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
08:55:53 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
08:55:54 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
08:55:55 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:55:58 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
08:55:59 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
08:56:00 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25001
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25002
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25010
MainThread: Metric 'statestore.live-backends' has reached desired value: 4
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
use default;
SET sync_ddl=1;
-- executing against localhost:21000
drop database if exists `hs2_db` cascade;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=1;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- connecting to:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
SET mem_limit=1G;
-- executing async:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
select sleep(1000);
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- closing connection to:
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:21002
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
