Quanlong Huang created IMPALA-10889:
---------------------------------------
Summary:
TestExecutorGroups.test_admission_control_with_multiple_coords is flaky
Key: IMPALA-10889
URL: https://issues.apache.org/jira/browse/IMPALA-10889
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Quanlong Huang
Failed test:
custom_cluster.test_executor_groups.TestExecutorGroups.test_admission_control_with_multiple_coords
Error Message
{code}
AssertionError: Metric admission-controller.agg-num-running.default-pool did
not reach value 1 in 30s. Dumping debug webpages in JSON format... Dumped memz
JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json
Dumped metrics JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json
Dumped queries JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json
Dumped sessions JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json
Dumped threadz JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json
Dumped rpcz JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json Dumping
minidumps for impalads/catalogds... Dumped minidump for Impalad PID 23807
Dumped minidump for Impalad PID 23810 Dumped minidump for Impalad PID 24897
Dumped minidump for Impalad PID 24900 Dumped minidump for Catalogd PID
23746{code}
Stacktrace
{code}
custom_cluster/test_executor_groups.py:579: in
test_admission_control_with_multiple_coords
"admission-controller.agg-num-running.default-pool", 1, timeout=30)
common/impala_service.py:143: in wait_for_metric_value
self.__metric_timeout_assert(metric_name, expected_value, timeout)
common/impala_service.py:210: in __metric_timeout_assert
assert 0, assert_string
E AssertionError: Metric admission-controller.agg-num-running.default-pool
did not reach value 1 in 30s.
E Dumping debug webpages in JSON format...
E Dumped memz JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json
E Dumped metrics JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json
E Dumped queries JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json
E Dumped sessions JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json
E Dumped threadz JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json
E Dumped rpcz JSON to
$IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json
E Dumping minidumps for impalads/catalogds...
E Dumped minidump for Impalad PID 23807
E Dumped minidump for Impalad PID 23810
E Dumped minidump for Impalad PID 24897
E Dumped minidump for Impalad PID 24900
E Dumped minidump for Catalogd PID 23746
{code}
Standard Error
{code}
-- 2021-08-26 05:46:20,996 INFO MainThread: Starting cluster with command:
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1
--log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
--log_level=1 --use_exclusive_coordinators '--impalad_args=
-executor_groups=coordinator ' --impalad_args=--default_query_options=
05:46:21 MainThread: Starting impala cluster without executors
05:46:21 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
05:46:21 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
05:46:21 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
05:46:21 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
05:46:24 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:24 MainThread: 'backends'
05:46:24 MainThread: Waiting for num_known_live_backends=1. Current value: None
05:46:25 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
05:46:25 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:25 MainThread: Waiting for num_known_live_backends=1. Current value: 0
05:46:26 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
05:46:26 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:26 MainThread: Waiting for num_known_live_backends=1. Current value: 0
05:46:27 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
05:46:27 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:27 MainThread: num_known_live_backends has reached value: 1
05:46:28 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0
executors).
-- 2021-08-26 05:46:28,337 DEBUG MainThread: Found 1 impalad/1 statestored/1
catalogd process(es)
-- 2021-08-26 05:46:28,337 INFO MainThread: Getting metric:
statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
-- 2021-08-26 05:46:28,341 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 2
-- 2021-08-26 05:46:28,341 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:28,343 INFO MainThread: num_known_live_backends has
reached value: 1
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-08-26 05:46:28,489 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-08-26 05:46:28,509 INFO MainThread: Closing active operation
-- 2021-08-26 05:46:28,511 INFO MainThread: Adding a coordinator
-- 2021-08-26 05:46:28,511 INFO MainThread: Starting cluster with command:
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=2
--log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
--log_level=1 --use_exclusive_coordinators
--impalad_args=-executor_groups=coordinator
--impalad_args=-default_pool_mem_limit=100g
--impalad_args=--default_query_options=
05:46:28 MainThread: Starting impala cluster without executors
05:46:29 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
05:46:29 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
05:46:29 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
05:46:29 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
05:46:29 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:32 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:32 MainThread: 'backends'
05:46:32 MainThread: Waiting for num_known_live_backends=2. Current value: None
05:46:33 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:33 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:33 MainThread: Waiting for num_known_live_backends=2. Current value: 0
05:46:34 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:34 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:34 MainThread: Waiting for num_known_live_backends=2. Current value: 0
05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:35 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:35 MainThread: num_known_live_backends has reached value: 2
05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:35 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
05:46:35 MainThread: num_known_live_backends has reached value: 2
05:46:36 MainThread: Impala Cluster Running with 2 nodes (2 coordinators, 0
executors).
-- 2021-08-26 05:46:36,408 DEBUG MainThread: Found 2 impalad/1 statestored/1
catalogd process(es)
-- 2021-08-26 05:46:36,409 INFO MainThread: Getting metric:
statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
-- 2021-08-26 05:46:36,412 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 3
-- 2021-08-26 05:46:36,412 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:36,414 INFO MainThread: num_known_live_backends has
reached value: 2
-- 2021-08-26 05:46:36,414 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
-- 2021-08-26 05:46:36,416 INFO MainThread: num_known_live_backends has
reached value: 2
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
-- connecting to: localhost:21001
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
-- connecting to: localhost:21000
-- 2021-08-26 05:46:36,417 ERROR MainThread: Failed to open transport
(tries_left=3)
Traceback (most recent call last):
File
"/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
line 1099, in _execute
return func(request)
File
"/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py",
line 206, in CloseSession
return self.recv_CloseSession()
File
"/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py",
line 218, in recv_CloseSession
(fname, mtype, rseqid) = iprot.readMessageBegin()
File
"/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py",
line 134, in readMessageBegin
sz = self.readI32()
File
"/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py",
line 217, in readI32
buff = self.trans.readAll(4)
File
"/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py",
line 60, in readAll
chunk = self.read(sz - have)
File
"/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py",
line 162, in read
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
File
"/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
line 132, in read
message='TSocket read 0 bytes')
TTransportException: TSocket read 0 bytes
-- connecting to localhost:21050 with impyla
-- 2021-08-26 05:46:36,563 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-08-26 05:46:36,585 INFO MainThread: Closing active operation
-- 2021-08-26 05:46:36,586 INFO MainThread: Adding 2 executors to group
default-pool-group1 with minimum size 2
-- 2021-08-26 05:46:36,586 INFO MainThread: Starting cluster with command:
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=0
--log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
--log_level=1 --add_executors --impalad_args=-admission_control_slots=2
--impalad_args=-executor_groups=default-pool-group1:2
--impalad_args=-mem_limit=4g --impalad_args=--default_query_options=
05:46:37 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
05:46:37 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
05:46:37 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node3.INFO
05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
05:46:40 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
05:46:40 MainThread: num_known_live_backends has reached value: 4
05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
05:46:40 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
05:46:40 MainThread: num_known_live_backends has reached value: 4
05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
05:46:41 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002
05:46:41 MainThread: num_known_live_backends has reached value: 4
05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
05:46:41 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003
05:46:41 MainThread: num_known_live_backends has reached value: 4
05:46:41 MainThread: Impala Cluster Running with 2 nodes (0 coordinators, 2
executors).
-- 2021-08-26 05:46:41,307 DEBUG MainThread: Found 4 impalad/1 statestored/1
catalogd process(es)
-- 2021-08-26 05:46:41,307 INFO MainThread: Getting metric:
statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
-- 2021-08-26 05:46:41,310 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 5
-- 2021-08-26 05:46:41,310 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:41,312 INFO MainThread: num_known_live_backends has
reached value: 4
-- 2021-08-26 05:46:41,312 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
-- 2021-08-26 05:46:41,314 INFO MainThread: num_known_live_backends has
reached value: 4
-- 2021-08-26 05:46:41,314 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002
-- 2021-08-26 05:46:41,316 INFO MainThread: num_known_live_backends has
reached value: 4
-- 2021-08-26 05:46:41,316 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003
-- 2021-08-26 05:46:41,318 INFO MainThread: num_known_live_backends has
reached value: 4
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
SET mt_dop=2;
-- executing async: localhost:21001
select * from functional_parquet.alltypes where month < 3 and
id + random() < sleep(100);;
-- 2021-08-26 05:46:42,208 INFO MainThread: Started query
db4bd9b9ff7787ec:963015e700000000
-- 2021-08-26 05:46:42,208 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:42,213 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: None
-- 2021-08-26 05:46:42,213 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:43,214 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:43,218 INFO MainThread: Metric
'admission-controller.agg-num-running.default-pool' has reached desired value: 1
-- executing async: localhost:21000
select count(*) from functional.alltypes where month + random() < 3;
-- 2021-08-26 05:46:43,673 INFO MainThread: Started query
de46a56beb40a1c9:6f12b84c00000000
-- 2021-08-26 05:46:43,673 INFO MainThread: Getting metric:
admission-controller.local-num-queued.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:43,677 INFO MainThread: Metric
'admission-controller.local-num-queued.default-pool' has reached desired value:
1
-- getting runtime profile operation:
<tests.common.impala_connection.OperationHandle object at 0x7f3651f12710>
-- closing query for operation handle:
<tests.common.impala_connection.OperationHandle object at 0x7f3651f12710>
-- closing query for operation handle:
<tests.common.impala_connection.OperationHandle object at 0x7f3651f3ae50>
-- 2021-08-26 05:46:43,680 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:43,684 INFO MainThread: Metric
'admission-controller.agg-num-running.default-pool' has reached desired value: 0
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
SET
client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
SET mem_limit=4g;
-- executing async: localhost:21001
select * from functional_parquet.alltypes where month < 3 and
id + random() < sleep(100);;
-- 2021-08-26 05:46:43,701 INFO MainThread: Started query
1841bd63d1384bff:7296726f00000000
-- 2021-08-26 05:46:43,701 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:43,705 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:43,706 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:44,707 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:44,711 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:44,711 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:45,712 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:45,717 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:45,717 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:46,717 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:46,722 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:46,722 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:47,723 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:47,727 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:47,727 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:48,728 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:48,733 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:48,733 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:49,734 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:49,739 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:49,739 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:50,740 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:50,744 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:50,744 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:51,745 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:51,749 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:51,749 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:52,750 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:52,754 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:52,754 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:53,755 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:53,760 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:53,760 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:54,760 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:54,765 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:54,765 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:55,766 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:55,770 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:55,770 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:56,771 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:56,776 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:56,776 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:57,776 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:57,780 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:57,781 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:58,781 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:58,785 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:58,785 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:46:59,786 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:46:59,790 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:46:59,790 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:00,791 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:00,796 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:00,796 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:01,797 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:01,801 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:01,801 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:02,802 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:02,807 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:02,807 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:03,808 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:03,813 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:03,813 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:04,814 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:04,818 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:04,818 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:05,819 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:05,823 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:05,823 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:06,824 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:06,828 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:06,828 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:07,829 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:07,834 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:07,834 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:08,835 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:08,839 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:08,839 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:09,840 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:09,845 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:09,845 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:10,845 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:10,850 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:10,850 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:11,851 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:11,855 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:11,855 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:12,857 INFO MainThread: Getting metric:
admission-controller.agg-num-running.default-pool from
impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
-- 2021-08-26 05:47:12,861 INFO MainThread: Waiting for metric value
'admission-controller.agg-num-running.default-pool'=1. Current value: 0
-- 2021-08-26 05:47:12,861 INFO MainThread: Sleeping 1s before next retry.
-- 2021-08-26 05:47:13,862 INFO MainThread: Metric
admission-controller.agg-num-running.default-pool did not reach value 1 in 30s.
Failing...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
