Thomas Tauber-Marshall created IMPALA-6642:
----------------------------------------------

             Summary: start-impala-cluster.py failing in some customer cluster 
tests
                 Key: IMPALA-6642
                 URL: https://issues.apache.org/jira/browse/IMPALA-6642
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 2.12.0
            Reporter: Thomas Tauber-Marshall


Seen in recent builds, both on the 2.x branch. Not quite the same test/error, 
but they seem similar enough to probably be related:
{noformat}
17:01:18 _________ ERROR at setup of TestAdmissionController.test_require_user 
__________
17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
17:01:18     self._start_impala_cluster(cluster_args)
17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
17:01:18     check_call(cmd + options, close_fds=True)
17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
17:01:18     raise CalledProcessError(retcode, cmd)
17:01:18 E   CalledProcessError: Command 
'['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
 '--cluster_size=3', '--num_coordinators=3', 
'--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
 '--log_level=1', '--impalad_args="-vmodule admission-controller=3 
-fair_scheduler_allocation_path 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
 -llama_site_path 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
 -disable_admission_control=false -require_username" ', 
'--state_store_args="-statestore_heartbeat_frequency_ms=100 
-statestore_priority_update_frequency_ms=100" ']' returned non-zero exit status 
1
17:01:18 ---------------------------- Captured stdout setup 
-----------------------------
17:01:18 Starting State Store logging to 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
17:01:18 Starting Catalog Service logging to 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
17:01:18 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
17:01:18 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
17:01:18 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
17:01:18 
17:01:18 ---------------------------- Captured stderr setup 
-----------------------------
17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
{noformat}

{noformat}
16:42:41 _______ ERROR at setup of 
TestAuthorization.test_access_runtime_profile ________
16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
16:42:41     self._start_impala_cluster(cluster_args)
16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
16:42:41     check_call(cmd + options, close_fds=True)
16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
16:42:41     raise CalledProcessError(retcode, cmd)
16:42:41 E   CalledProcessError: Command 
'['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py',
 '--cluster_size=3', '--num_coordinators=3', 
'--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests',
 '--log_level=1', '--impalad_args="--server_name=server1      
--authorization_policy_file=/test-warehouse/authz-policy.ini      
--authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status 1
16:42:41 ---------------------------- Captured stdout setup 
-----------------------------
16:42:41 Starting State Store logging to 
/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
16:42:41 Starting Catalog Service logging to 
/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
16:42:41 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
16:42:41 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
16:42:41 Starting Impala Daemon logging to 
/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
16:42:41 Error starting cluster: num_known_live_backends did not reach expected 
value in time
16:42:41 ---------------------------- Captured stderr setup 
-----------------------------
16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Debug webpage not yet available.
16:42:41 MainThread: Debug webpage not yet available.
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 1
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
...
16:42:41 MainThread: Getting num_known_live_backends from 
ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to