Quanlong Huang created IMPALA-13421:
---------------------------------------
Summary: HiveMetastore failed to launch in ozone build
Key: IMPALA-13421
URL: https://issues.apache.org/jira/browse/IMPALA-13421
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Michael Smith
Hive Metastore failed to start in an ozone build.
{noformat}
12:33:04 Starting ozone (Web UI - http://localhost:9874)
12:33:09 The kms cluster is running
12:33:37 The ozone cluster is running
12:33:37 Creating Ozone volume/bucket
12:33:37 Ozone encryption enabled for impala/test-warehouse
12:33:37 24/10/03 12:33:39 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
12:33:39 24/10/03 12:33:40 INFO rpc.RpcClient: Creating Volume: impala, with
jenkins as owner and space quota set to -1 bytes, counts quota set to -1
12:33:40 VOLUME_ALREADY_EXISTS Volume already exists
12:33:40 24/10/03 12:33:41 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
12:33:41 24/10/03 12:33:41 INFO rpc.RpcClient: Creating Bucket:
impala/test-warehouse, with server-side default bucket layout, jenkins as
owner, Versioning false, Storage Type set to DISK and Encryption set to true,
Replication Type set to server-side default replication type, Namespace Quota
set to -1, Space Quota set to -1
12:33:41 BUCKET_ALREADY_EXISTS Bucket already exist
12:33:42 Stopping kudu
12:33:42 Starting kudu (Web UI - http://localhost:8051)
12:33:49 The kudu cluster is running
12:34:04 --> Starting Hive Metastore Service
12:34:04 No handlers could be found for logger "thrift.transport.TSocket"
12:34:04 Waiting for the Metastore at localhost:9083...
12:34:48 Waiting for the Metastore at localhost:9083...
12:34:48 Waiting for the Metastore at localhost:9083...
...
12:39:05 Waiting for the Metastore at localhost:9083...
12:39:05 Metastore service failed to start within 300.0 seconds.
12:39:05 ERROR in
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/testdata/bin/run-hive-server.sh
at line 166: ${CLUSTER_BIN}/wait-for-metastore.py
--transport=${METASTORE_TRANSPORT}
12:39:05 Generated:
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.run-hive-server.20241003_19_39_05.xml
12:39:05 ERROR in
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/testdata/bin/run-all.sh
at line 91: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-hive-server.log
12:39:05 Generated:
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.run-all.20241003_19_39_05.xml
12:39:05 + echo 'buildall.sh ' -format '-snapshot_file
/data/jenkins/workspace/impala-asf-master-core-ozone/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-1339-SNAPSHOT.tar.gz'
'-metastore_snapshot_file
/data/jenkins/workspace/impala-asf-master-core-ozone/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-1339.txt
failed.'
12:39:05 buildall.sh -format -snapshot_file
/data/jenkins/workspace/impala-asf-master-core-ozone/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-1339-SNAPSHOT.tar.gz
-metastore_snapshot_file
/data/jenkins/workspace/impala-asf-master-core-ozone/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-1339.txt
failed{noformat}
Looking into HMS logs, it failed to connect to the Ozone Manager:
{noformat}
2024-10-03T12:34:08,927 INFO [main] metastore.HiveMetaStore: Starting hive
metastore on port 9083
2024-10-03T12:34:08,935 INFO [main] metastore.AuthFactory: Using
authentication NOSASL with kerberos authentication disabled
2024-10-03T12:34:11,908 INFO [main] retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
impala-ec2-centos79-m6i-4xlarge-xldisk-0058.vpc.cloudera.com/127.0.0.1 to
localhost:9862 failed on connection exception: java.net.ConnectException:
Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
$Proxy31.submitRequest over nodeId=null,nodeAddress=localhost:9862 after 1
failover attempts. Trying to failover after sleeping for 4000ms.
...
2024-10-03T12:38:41,926 INFO [main] retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
impala-ec2-centos79-m6i-4xlarge-xldisk-0058.vpc.cloudera.com/127.0.0.1 to
localhost:9862 failed on connection exception: java.net.ConnectException:
Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
$Proxy31.submitRequest over nodeId=null,nodeAddress=localhost:9862 after 16
failover attempts. Trying to failover after sleeping for 34000ms.
2024-10-03T12:39:05,886 INFO [shutdown-hook-0] metastore.HiveMetaStore:
Shutting down hive metastore.{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
