[
https://issues.apache.org/jira/browse/IMPALA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated IMPALA-7361:
--------------------------------
Description:
texttest_heterogeneous_proc_mem_limit custom cluster test fails with the
following assertion error
{preformatted}
Stacktrace
custom_cluster/test_admission_controller.py:514: in
test_heterogeneous_proc_mem_limit
assert re.search("Queued reason: Not enough memory available on host
\S+.Needed "
E AssertionError: ImpalaBeeswaxException:
E Query aborted:Admission for query exceeded timeout 200ms in pool
default-pool. Queued reason: Not enough memory available on host
impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00
GB but only 1.00 GB out of 3.00 GB was available.
E
E
E assert None
E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for
query exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n')
E + where <function search at 0x7f2b4a67c5f0> = re.search
E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n' = str(ImpalaBeeswaxException())
{preformatted}
was:
test_heterogeneous_proc_mem_limit fails with the following assertion error
{code:java}
AssertionError: ImpalaBeeswaxException: Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued reason: Not enough memory
available on host <hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00
GB was available. assert None + where None = <function search at
0x7f2b4a67c5f0>('Queued reason: Not enough memory available on host \\S+.Needed
2.00 GB but only 1.00 GB out of 2.00 GB was available.',
'ImpalaBeeswaxException:\n Query aborted:Admission for query exceeded timeout
200ms in pool default-pool. Queued reaso...<hostname>:22001.Needed 2.00 GB but
only 1.00 GB out of 3.00 GB was available.\n\n') + where <function search
at 0x7f2b4a67c5f0> = re.search + and 'ImpalaBeeswaxException:\n Query
aborted:Admission for query exceeded timeout 200ms in pool default-pool. Queued
reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
available.\n\n' = str(ImpalaBeeswaxException())
{code}
stack trace
{code:java}
*Stacktrace*
custom_cluster/test_admission_controller.py:514: in
test_heterogeneous_proc_mem_limit
assert re.search("Queued reason: Not enough memory available on host
\S+.Needed "
E AssertionError: ImpalaBeeswaxException:
E Query aborted:Admission for query exceeded timeout 200ms in pool
default-pool. Queued reason: Not enough memory available on host
<host>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was available.
E
E
E assert None
E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission for
query exceeded timeout 200ms in pool default-pool. Queued
reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
available.\n\n')
E + where <function search at 0x7f2b4a67c5f0> = re.search
E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
exceeded timeout 200ms in pool default-pool. Queued
reaso...<hostname>:22001.Needed 2.00 GB but only 1.00 GB out of 3.00 GB was
available.\n\n' = str(ImpalaBeeswaxException())
*Standard Error*
08:55:51 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
08:55:52 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
08:55:53 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
08:55:54 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
08:55:55 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
08:55:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:55:58 MainThread: Getting num_known_live_backends from <hostname>:25000
08:55:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0
08:55:59 MainThread: Getting num_known_live_backends from<hostname>:25000
08:55:59 MainThread: Waiting for num_known_live_backends=3. Current value: 1
08:56:00 MainThread: Getting num_known_live_backends from <hostname>:25000
08:56:00 MainThread: Waiting for num_known_live_backends=3. Current value: 2
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25000
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25001
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Getting num_known_live_backends from <hostname>:25002
08:56:01 MainThread: num_known_live_backends has reached value: 3
08:56:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from <hostname>:25010
MainThread: Metric 'statestore.live-backends' has reached desired value: 4
MainThread: Getting num_known_live_backends from <hostname>:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from <hostname>:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from <hostname>:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
use default;
SET sync_ddl=1;
-- executing against localhost:21000
drop database if exists `hs2_db` cascade;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=1;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=3G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- connecting to: <hostname>:21002
SET mem_limit=1G;
-- executing async: <hostname>:21002
select sleep(1000);
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET mem_limit=2G;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
select * from functional.alltypesagg, (select 1) B limit 1;
-- closing connection to: i<hostname>:21002
{code}
> test_heterogeneous_proc_mem_limit: Query aborted: Not enough memory
> available on host
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-7361
> URL: https://issues.apache.org/jira/browse/IMPALA-7361
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: nithya
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: broken-build
>
> texttest_heterogeneous_proc_mem_limit custom cluster test fails with the
> following assertion error
> {preformatted}
> Stacktrace
> custom_cluster/test_admission_controller.py:514: in
> test_heterogeneous_proc_mem_limit
> assert re.search("Queued reason: Not enough memory available on host
> \S+.Needed "
> E AssertionError: ImpalaBeeswaxException:
> E Query aborted:Admission for query exceeded timeout 200ms in pool
> default-pool. Queued reason: Not enough memory available on host
> impala-ec2-centos74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed
> 2.00 GB but only 1.00 GB out of 3.00 GB was available.
> E
> E
> E assert None
> E + where None = <function search at 0x7f2b4a67c5f0>('Queued reason: Not
> enough memory available on host \\S+.Needed 2.00 GB but only 1.00 GB out of
> 2.00 GB was available.', 'ImpalaBeeswaxException:\n Query aborted:Admission
> for query exceeded timeout 200ms in pool default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n')
> E + where <function search at 0x7f2b4a67c5f0> = re.search
> E + and 'ImpalaBeeswaxException:\n Query aborted:Admission for query
> exceeded timeout 200ms in pool default-pool. Queued
> reaso...os74-m5-4xlarge-ondemand-08d6.vpc.cloudera.com:22001.Needed 2.00 GB
> but only 1.00 GB out of 3.00 GB was available.\n\n' =
> str(ImpalaBeeswaxException())
> {preformatted}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
