[
https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498280#comment-16498280
]
Thomas Tauber-Marshall commented on IMPALA-6642:
------------------------------------------------
[~tarasbob] picked you at random. Got some time to look into this? If not, let
me know. Seems its been happening in builds more frequently this week, so it
would be good to get it fixed asap. Thanks
> start-impala-cluster.py failing in some customer cluster tests
> --------------------------------------------------------------
>
> Key: IMPALA-6642
> URL: https://issues.apache.org/jira/browse/IMPALA-6642
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.12.0
> Reporter: Thomas Tauber-Marshall
> Priority: Blocker
> Labels: broken-build
>
> Seen in recent builds, both on the 2.x branch. Not quite the same test/error,
> but they seem similar enough to probably be related:
> {noformat}
> 17:01:18 _________ ERROR at setup of
> TestAdmissionController.test_require_user __________
> 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
> 17:01:18 self._start_impala_cluster(cluster_args)
> 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 17:01:18 check_call(cmd + options, close_fds=True)
> 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 17:01:18 raise CalledProcessError(retcode, cmd)
> 17:01:18 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="-vmodule admission-controller=3
> -fair_scheduler_allocation_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
> -llama_site_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
> -disable_admission_control=false -require_username" ',
> '--state_store_args="-statestore_heartbeat_frequency_ms=100
> -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit
> status 1
> 17:01:18 ---------------------------- Captured stdout setup
> -----------------------------
> 17:01:18 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 17:01:18 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
> 17:01:18
> 17:01:18 ---------------------------- Captured stderr setup
> -----------------------------
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> {noformat}
> {noformat}
> 16:42:41 _______ ERROR at setup of
> TestAuthorization.test_access_runtime_profile ________
> 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
> 16:42:41 self._start_impala_cluster(cluster_args)
> 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 16:42:41 check_call(cmd + options, close_fds=True)
> 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 16:42:41 raise CalledProcessError(retcode, cmd)
> 16:42:41 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="--server_name=server1
> --authorization_policy_file=/test-warehouse/authz-policy.ini
> --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status
> 1
> 16:42:41 ---------------------------- Captured stdout setup
> -----------------------------
> 16:42:41 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 16:42:41 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 16:42:41 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 16:42:41 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 16:42:41 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 16:42:41 Error starting cluster: num_known_live_backends did not reach
> expected value in time
> 16:42:41 ---------------------------- Captured stderr setup
> -----------------------------
> 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Debug webpage not yet available.
> 16:42:41 MainThread: Debug webpage not yet available.
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 1
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> ...
> 16:42:41 MainThread: Getting num_known_live_backends from
> ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
