[
https://issues.apache.org/jira/browse/IMPALA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963297#comment-16963297
]
Andrew Sherman commented on IMPALA-9076:
----------------------------------------
A similar failure occurred in
TestExecutorGroups.test_executor_group_starts_while_qeueud
{code:java}
Error Messageassert 'Waiting for executors to start' in 'Query
(id=0449909e43b1a06d:7713311500000000):\n DEBUG MODE WARNING: Query profile
created while running a DEBUG buil...0)\n - NumRowsFetchedFromCache: 0
(0)\n - RowMaterializationRate: 0\n - RowMaterializationTimer:
0.000ns\n'Stacktracecustom_cluster/test_executor_groups.py:139: in
test_executor_group_starts_while_qeueud
assert "Waiting for executors to start" in profile
E assert 'Waiting for executors to start' in 'Query
(id=0449909e43b1a06d:7713311500000000):\n DEBUG MODE WARNING: Query profile
created while running a DEBUG buil...0)\n - NumRowsFetchedFromCache: 0
(0)\n - RowMaterializationRate: 0\n - RowMaterializationTimer:
0.000ns\n'Standard Error-- 2019-10-30 02:07:59,705 INFO MainThread:
Starting cluster with command:
/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1
--log_dir=/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests
--log_level=1 --use_exclusive_coordinators '--impalad_args=
-executor_groups=coordinator ' --impalad_args=--default_query_options=
02:08:01 MainThread: Starting impala cluster without executors
02:08:01 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
02:08:01 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/statestored.INFO
02:08:02 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
02:08:03 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/impalad.INFO
02:08:06 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
02:08:07 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
02:08:07 MainThread: Getting num_known_live_backends from
shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com:25000
02:08:07 MainThread: Debug webpage not yet available: ('Connection aborted.',
error(111, 'Connection refused'))
02:08:09 MainThread: Debug webpage did not become available in expected time.
02:08:09 MainThread: Waiting for num_known_live_backends=1. Current value: None
02:08:10 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
02:08:10 MainThread: Getting num_known_live_backends from
shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com:25000
02:08:10 MainThread: Debug webpage not yet available: ('Connection aborted.',
error(111, 'Connection refused'))
02:08:12 MainThread: Debug webpage did not become available in expected time.
02:08:12 MainThread: Waiting for num_known_live_backends=1. Current value: None
02:08:13 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
02:08:13 MainThread: Getting num_known_live_backends from
shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com:25000
02:08:13 MainThread: num_known_live_backends has reached value: 1
02:08:13 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0
executors).
-- 2019-10-30 02:08:14,157 DEBUG MainThread: Found 1 impalad/1 statestored/1
catalogd process(es)
-- 2019-10-30 02:08:14,159 INFO MainThread: Getting metric:
statestore.live-backends from
shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com:25010
-- 2019-10-30 02:08:14,164 INFO MainThread: Starting new HTTP connection
(1): shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com
-- 2019-10-30 02:08:14,172 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 2
-- 2019-10-30 02:08:14,172 DEBUG MainThread: Getting num_known_live_backends
from shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com:25000
-- 2019-10-30 02:08:14,174 INFO MainThread: Starting new HTTP connection
(1): shared-centos64-ec2-m2-4xlarge-ondemand-0ed0.vpc.cloudera.com
-- 2019-10-30 02:08:14,178 INFO MainThread: num_known_live_backends has
reached value: 1
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_executor_group_starts_while_qeueud;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2019-10-30 02:08:14,500 INFO MainThread: Closing active operation
-- executing async: localhost:21000
select count(*) from functional.alltypestiny;
-- 2019-10-30 02:08:16,590 INFO MainThread: Started query
0449909e43b1a06d:7713311500000000
-- getting runtime profile operation:
<tests.common.impala_connection.OperationHandle object at 0x4d5fbd0>
{code}
> TestExecutorGroups.test_sequential_startup_wait appears to be flaky
> -------------------------------------------------------------------
>
> Key: IMPALA-9076
> URL: https://issues.apache.org/jira/browse/IMPALA-9076
> Project: IMPALA
> Issue Type: Bug
> Reporter: Thomas Tauber-Marshall
> Assignee: Lars Volker
> Priority: Blocker
> Labels: broken-build, flaky
>
> {noformat}
> Error Message
> assert 'Initial admission queue reason: Waiting for executors to start' in
> 'Query (id=d9484fef8ecacec7:5278c53400000000):\n DEBUG MODE WARNING: Query
> profile created while running a DEBUG buil...0)\n -
> NumRowsFetchedFromCache: 0 (0)\n - RowMaterializationRate: 0\n -
> RowMaterializationTimer: 0.000ns\n'
> Stacktrace
> custom_cluster/test_executor_groups.py:315: in test_sequential_startup_wait
> assert "Initial admission queue reason: Waiting for executors to start"
> in profile
> E assert 'Initial admission queue reason: Waiting for executors to start'
> in 'Query (id=d9484fef8ecacec7:5278c53400000000):\n DEBUG MODE WARNING:
> Query profile created while running a DEBUG buil...0)\n -
> NumRowsFetchedFromCache: 0 (0)\n - RowMaterializationRate: 0\n -
> RowMaterializationTimer: 0.000ns\n'
> Standard Error
> -- 2019-10-20 01:57:00,039 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1
> --log_dir=/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests
> --log_level=1 --use_exclusive_coordinators '--impalad_args=
> -executor_groups=coordinator ' --impalad_args=--default_query_options=
> 01:57:01 MainThread: Starting impala cluster without executors
> 01:57:01 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 01:57:01 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 01:57:03 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 01:57:04 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 01:57:08 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 01:57:08 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 01:57:08 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:08 MainThread: Debug webpage not yet available: ('Connection aborted.',
> error(111, 'Connection refused'))
> 01:57:10 MainThread: Debug webpage did not become available in expected time.
> 01:57:10 MainThread: Waiting for num_known_live_backends=1. Current value:
> None
> 01:57:11 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 01:57:11 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:11 MainThread: Debug webpage not yet available: ('Connection aborted.',
> error(111, 'Connection refused'))
> 01:57:13 MainThread: Debug webpage did not become available in expected time.
> 01:57:13 MainThread: Waiting for num_known_live_backends=1. Current value:
> None
> 01:57:14 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 01:57:14 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:14 MainThread: num_known_live_backends has reached value: 1
> 01:57:15 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0
> executors).
> -- 2019-10-20 01:57:15,401 DEBUG MainThread: Found 1 impalad/1
> statestored/1 catalogd process(es)
> -- 2019-10-20 01:57:15,402 INFO MainThread: Getting metric:
> statestore.live-backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25010
> -- 2019-10-20 01:57:15,409 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:15,421 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 2
> -- 2019-10-20 01:57:15,421 DEBUG MainThread: Getting
> num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> -- 2019-10-20 01:57:15,423 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:15,427 INFO MainThread: num_known_live_backends has
> reached value: 1
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_sequential_startup_wait;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2019-10-20 01:57:15,786 INFO MainThread: Closing active operation
> -- 2019-10-20 01:57:15,793 INFO MainThread: Adding 1 executors to group
> default-pool-group1 with minimum size 3
> -- 2019-10-20 01:57:15,794 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=0
> --log_dir=/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests
> --log_level=1 --add_executors --impalad_args=-admission_control_slots=0
> --impalad_args=-executor_groups=default-pool-group1:3
> --impalad_args=--default_query_options=
> 01:57:17 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 01:57:17 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 01:57:20 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:20 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:20 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:20 MainThread: Waiting for num_known_live_backends=2. Current value: 1
> 01:57:21 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:21 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:21 MainThread: Waiting for num_known_live_backends=2. Current value: 1
> 01:57:23 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:23 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:23 MainThread: Waiting for num_known_live_backends=2. Current value: 1
> 01:57:24 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:24 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:24 MainThread: Waiting for num_known_live_backends=2. Current value: 1
> 01:57:25 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:25 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> 01:57:25 MainThread: num_known_live_backends has reached value: 2
> 01:57:25 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 01:57:25 MainThread: Getting num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25001
> 01:57:25 MainThread: num_known_live_backends has reached value: 2
> 01:57:25 MainThread: Impala Cluster Running with 1 nodes (0 coordinators, 1
> executors).
> -- 2019-10-20 01:57:26,163 DEBUG MainThread: Found 2 impalad/1
> statestored/1 catalogd process(es)
> -- 2019-10-20 01:57:26,164 INFO MainThread: Getting metric:
> statestore.live-backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25010
> -- 2019-10-20 01:57:26,170 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:26,181 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 3
> -- 2019-10-20 01:57:26,181 DEBUG MainThread: Getting
> num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> -- 2019-10-20 01:57:26,183 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:26,189 INFO MainThread: num_known_live_backends has
> reached value: 2
> -- 2019-10-20 01:57:26,189 DEBUG MainThread: Getting
> num_known_live_backends from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25001
> -- 2019-10-20 01:57:26,191 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:26,196 INFO MainThread: num_known_live_backends has
> reached value: 2
> -- 2019-10-20 01:57:26,197 INFO MainThread: Getting metric:
> cluster-membership.backends.total from
> shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000
> -- 2019-10-20 01:57:26,199 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:26,214 INFO MainThread: Metric
> 'cluster-membership.backends.total' has reached desired value: 2
> -- 2019-10-20 01:57:26,216 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- 2019-10-20 01:57:26,230 INFO MainThread: Starting new HTTP connection
> (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com
> -- executing async: localhost:21000
> select sleep(4);
> -- 2019-10-20 01:57:26,663 INFO MainThread: Started query
> d9484fef8ecacec7:5278c53400000000
> -- getting runtime profile operation:
> <tests.common.impala_connection.OperationHandle object at 0x5164dd0>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
