Client can't connect to metastore server
----------------------------------------
Key: HCATALOG-366
URL: https://issues.apache.org/jira/browse/HCATALOG-366
Project: HCatalog
Issue Type: Bug
Components: client
Affects Versions: 0.4, 0.5, 0.4.1
Reporter: David Capwell
Seems that after a while the client will start to see the following exception:
12/04/09 19:20:23 INFO hive.metastore: Trying to connect to metastore with URI
thrift://hcatserver.exampl.com:9080
12/04/09 19:20:23 WARN hive.metastore: Failed to connect to the MetaStore
Server...
12/04/09 19:20:23 INFO hive.metastore: Waiting 1 seconds before next connection
attempt.
12/04/09 19:20:24 WARN hive.metastore: Failed to connect to the MetaStore
Server...
12/04/09 19:20:24 INFO hive.metastore: Waiting 1 seconds before next connection
attempt.
12/04/09 19:20:25 WARN hive.metastore: Failed to connect to the MetaStore
Server...
12/04/09 19:20:25 INFO hive.metastore: Waiting 1 seconds before next connection
attempt.
12/04/09 19:20:26 WARN hive.metastore: Failed to connect to the MetaStore
Server...
12/04/09 19:20:26 INFO hive.metastore: Waiting 1 seconds before next connection
attempt.
12/04/09 19:20:27 WARN hive.metastore: Failed to connect to the MetaStore
Server...
12/04/09 19:20:27 WARN hive.metastore: [Ljava.lang.StackTraceElement;@92668c
12/04/09 19:20:27 WARN hive.metastore: Unable to connect to metastore with URI
thrift://hcatserver.example.com:9080
java.io.IOException: MetaException(message:Could not connect to meta store
using any of the URIs provided)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:42)
at org.apache.hcatalog.test.StoreNumbers.main(StoreNumbers.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hcatalog.test.HCatTestDriver.main(HCatTestDriver.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: MetaException(message:Could not connect to meta store using any of
the URIs provided)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:205)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:157
at
org.apache.hcatalog.mapreduce.InitializeInput.createHiveMetaClient(InitializeInput.java:58)
at
org.apache.hcatalog.mapreduce.InitializeInput.getSerializedHcatKeyJobInfo(InitializeInput.java:85)
at
org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:73)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:40)
... 13 more
On the server side I see the following exception:
2012-04-09 19:20:00,344 ERROR server.TThreadPoolServer
(TThreadPoolServer.java:run(182)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
Peer indicated failure: GSS initiate failed
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:557)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:555)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1064)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:555)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated
failure: GSS initiate failed
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
... 10 more
This happens both on 0.4 and trunk. After a few seconds it seems that
everything stabilizes again and the metastore can reply again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
