[ https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Tauber-Marshall reassigned IMPALA-6642: ---------------------------------------------- Assignee: Taras Bobrovytsky > start-impala-cluster.py failing in some customer cluster tests > -------------------------------------------------------------- > > Key: IMPALA-6642 > URL: https://issues.apache.org/jira/browse/IMPALA-6642 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 2.12.0 > Reporter: Thomas Tauber-Marshall > Assignee: Taras Bobrovytsky > Priority: Blocker > Labels: broken-build > > Seen in recent builds, both on the 2.x branch. Not quite the same test/error, > but they seem similar enough to probably be related: > {noformat} > 17:01:18 _________ ERROR at setup of > TestAdmissionController.test_require_user __________ > 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method > 17:01:18 self._start_impala_cluster(cluster_args) > 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster > 17:01:18 check_call(cmd + options, close_fds=True) > 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call > 17:01:18 raise CalledProcessError(retcode, cmd) > 17:01:18 E CalledProcessError: Command > '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py', > '--cluster_size=3', '--num_coordinators=3', > '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests', > '--log_level=1', '--impalad_args="-vmodule admission-controller=3 > -fair_scheduler_allocation_path > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml > -llama_site_path > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml > -disable_admission_control=false -require_username" ', > '--state_store_args="-statestore_heartbeat_frequency_ms=100 > -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit > status 1 > 17:01:18 ---------------------------- Captured stdout setup > ----------------------------- > 17:01:18 Starting State Store logging to > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO > 17:01:18 Starting Catalog Service logging to > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO > 17:01:18 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO > 17:01:18 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO > 17:01:18 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO > 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found > 17:01:18 > 17:01:18 ---------------------------- Captured stderr setup > ----------------------------- > 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) > 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) > {noformat} > {noformat} > 16:42:41 _______ ERROR at setup of > TestAuthorization.test_access_runtime_profile ________ > 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method > 16:42:41 self._start_impala_cluster(cluster_args) > 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster > 16:42:41 check_call(cmd + options, close_fds=True) > 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call > 16:42:41 raise CalledProcessError(retcode, cmd) > 16:42:41 E CalledProcessError: Command > '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py', > '--cluster_size=3', '--num_coordinators=3', > '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests', > '--log_level=1', '--impalad_args="--server_name=server1 > --authorization_policy_file=/test-warehouse/authz-policy.ini > --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status > 1 > 16:42:41 ---------------------------- Captured stdout setup > ----------------------------- > 16:42:41 Starting State Store logging to > /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO > 16:42:41 Starting Catalog Service logging to > /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO > 16:42:41 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO > 16:42:41 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO > 16:42:41 Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO > 16:42:41 Error starting cluster: num_known_live_backends did not reach > expected value in time > 16:42:41 ---------------------------- Captured stderr setup > ----------------------------- > 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Debug webpage not yet available. > 16:42:41 MainThread: Debug webpage not yet available. > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 1 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > ... > 16:42:41 MainThread: Getting num_known_live_backends from > ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000 > 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org