[ https://issues.apache.org/jira/browse/IMPALA-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564486#comment-16564486 ]
David Knupp commented on IMPALA-7379: ------------------------------------- Assigning to [~kwho] because it's an RPC timeout issue. Please reassign if this is not correct. Thanks. > test_random_rpc_timeout failed on exhaustive build: Debug webpage did not > become available in expected time > ----------------------------------------------------------------------------------------------------------- > > Key: IMPALA-7379 > URL: https://issues.apache.org/jira/browse/IMPALA-7379 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 3.1.0 > Reporter: David Knupp > Assignee: Michael Ho > Priority: Major > > *Stacktrace* > {noformat} > custom_cluster/test_rpc_timeout.py:131: in test_random_rpc_timeout > self.execute_query_verify_metrics(self.TEST_QUERY, 10) > custom_cluster/test_rpc_timeout.py:51: in execute_query_verify_metrics > v.wait_for_metric("impala-server.num-fragments-in-flight", 0) > verifiers/metric_verifier.py:62: in wait_for_metric > self.impalad_service.wait_for_metric_value(metric_name, expected_value, > timeout) > common/impala_service.py:132: in wait_for_metric_value > json.dumps(self.read_debug_webpage('memz?json')), > common/impala_service.py:63: in read_debug_webpage > return self.open_debug_webpage(page_name, timeout=timeout, > interval=interval).read() > common/impala_service.py:60: in open_debug_webpage > assert 0, 'Debug webpage did not become available in expected time.' > E AssertionError: Debug webpage did not become available in expected time. > {noformat} > *Standard Error* > {noformat} > 10:27:58 MainThread: Starting State Store logging to > /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO > 10:27:58 MainThread: Starting Catalog Service logging to > /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO > 10:27:59 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO > 10:28:00 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO > 10:28:01 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO > 10:28:04 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 10:28:04 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > 10:28:04 MainThread: Waiting for num_known_live_backends=3. Current value: 1 > 10:28:05 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > 10:28:05 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 10:28:06 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > 10:28:06 MainThread: num_known_live_backends has reached value: 3 > 10:28:06 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25001 > 10:28:06 MainThread: num_known_live_backends has reached value: 3 > 10:28:06 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25002 > 10:28:06 MainThread: num_known_live_backends has reached value: 3 > 10:28:06 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 > executors). > MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > MainThread: Getting metric: statestore.live-backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25010 > MainThread: Metric 'statestore.live-backends' has reached desired value: 4 > MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: num_known_live_backends has reached value: 3 > MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25001 > MainThread: num_known_live_backends has reached value: 3 > MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25002 > MainThread: num_known_live_backends has reached value: 3 > -- connecting to: localhost:21000 > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > -- executing against localhost:21000 > select count(c2.string_col) from functional.alltypestiny join > functional.alltypessmall c2; > MainThread: no process found with pid 19511 > MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Getting metric: impala-server.num-fragments-in-flight from > impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000 > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage did not become available in expected time. > MainThread: Waiting for metric value > 'impala-server.num-fragments-in-flight'=0. Current value: None > MainThread: Sleeping 1s before next retry. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > MainThread: Debug webpage not yet available. > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org