[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217170#comment-17217170
]
Tim Armstrong commented on IMPALA-9884:
---------------------------------------
Grabbed the coordinator log
[^impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-011116.23933.gz]
. Here's the JUnit XML output too
{noformat}
<testcase
classname="custom_cluster.test_admission_controller.TestAdmissionControllerStress"
file="custom_cluster/test_admission_controller.py" line="1829"
name="test_mem_limit[num_queries: 50 | protocol: beeswax | table_format:
text/none | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 5000, 'disable_codegen':
False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0} | submission_delay_ms: 150 |
round_robin_submission: False]" time="614.131304741"><failure
message="AssertionError: Queued queries should not run until others are made to
finish
assert 1 == 0">custom_cluster/test_admission_controller.py:1856: in
test_mem_limit
{'request_pool': self.pool_name, 'mem_limit':
query_mem_limit})
custom_cluster/test_admission_controller.py:1712: in run_admission_test
assert metric_deltas['dequeued'] == 0,\
E AssertionError: Queued queries should not run until others are made to
finish
E assert 1 == 0</failure><system-err>01:11:15 MainThread: Found 0 impalad/0
statestored/0 catalogd process(es)
01:11:15 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/statestored.INFO
01:11:16 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
01:11:16 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad.INFO
01:11:16 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
01:11:16 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
01:11:19 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:19 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:19 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25000
01:11:19 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:11:20 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:20 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25000
01:11:20 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:11:21 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:21 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25000
01:11:21 MainThread: num_known_live_backends has reached value: 3
01:11:21 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:21 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25001
01:11:21 MainThread: num_known_live_backends has reached value: 3
01:11:22 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:11:22 MainThread: Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25002
01:11:22 MainThread: num_known_live_backends has reached value: 3
01:11:22 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es)
INFO:impala_service:Getting metric: statestore.live-backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25010
INFO:impala_service:Metric 'statestore.live-backends' has reached
desired value: 4
DEBUG:impala_service:Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25000
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25001
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:25002
INFO:impala_service:num_known_live_backends has reached value: 3
</system-err></testcase>
{noformat}
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -----------------------------------------------------------------
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Vihang Karajgaonkar
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build, flaky
> Attachments:
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-011116.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
