Hive Connection to Metastore is lost frequently

[Aaryan Reddy]

Hello hive community,

I am working on HDP 2.5.3 distribution that comes with packaged Apache Hive
1.2.1 . Every so often the hive connection to the backend metastore is
broken throwing the below message.

2018-02-16 00:00:06,679 ERROR [pool-7-thread-76]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(209))
> - Retrying HMSHandler after 2000 ms (attempt 5 of 10) with error:
> javax.jdo.JDOException: Exception thrown when executing query : SELECT
> 'org.apache.hadoop.hive.metastore.model.MNotificationNextId' AS
> `NUCLEUS_TYPE`,`A0`.`NEXT_EVENT_ID`,`A0`.`NNI_ID` FROM
> `NOTIFICATION_SEQUENCE` `A0`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:677)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:388)



Only option that I see is to restart the metastore/Hive service . Also we
observed from couple of days there below error in metastore log , although
even after restart of service i still see them .

2018-02-16 00:00:04,912 ERROR [pool-7-thread-169]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - Error occurred during processing of
> message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException: Peer indicated failure:
> GSS initiate failed
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)



The mysql version that is used for metastore is 5.6.22 & the jdbc jar
version used to connect the metastore to the backed Db is "
mysql-connector-java-5.1.25.jar"

Any help here is appreciated.

Thanks ..

Re: Hive Connection to Metastore is lost frequently

[Aaryan Reddy]

folks any suggestion here ?

On Tue, Feb 27, 2018 at 11:31 AM, Aaryan Reddy <aaryanreddy...@gmail.com>
wrote:

> Hello hive community,
>
> I am working on HDP 2.5.3 distribution that comes with packaged Apache
> Hive 1.2.1 . Every so often the hive connection to the backend metastore is
> broken throwing the below message.
>
> 2018-02-16 00:00:06,679 ERROR [pool-7-thread-76]:
>> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(209))
>> - Retrying HMSHandler after 2000 ms (attempt 5 of 10) with error:
>> javax.jdo.JDOException: Exception thrown when executing query : SELECT
>> 'org.apache.hadoop.hive.metastore.model.MNotificationNextId' AS
>> `NUCLEUS_TYPE`,`A0`.`NEXT_EVENT_ID`,`A0`.`NNI_ID` FROM
>> `NOTIFICATION_SEQUENCE` `A0`
>> at org.datanucleus.api.jdo.NucleusJDOHelper.
>> getJDOExceptionForNucleusException(NucleusJDOHelper.java:677)
>> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:388)
>
>
>
> Only option that I see is to restart the metastore/Hive service . Also we
> observed from couple of days there below error in metastore log , although
> even after restart of service i still see them .
>
> 2018-02-16 00:00:04,912 ERROR [pool-7-thread-169]:
>> server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error
>> occurred during processing of message.
>> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
>> Peer indicated failure: GSS initiate failed
>> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(
>> TSaslServerTransport.java:219)
>
>
>
> The mysql version that is used for metastore is 5.6.22 & the jdbc jar
> version used to connect the metastore to the backed Db is "
> mysql-connector-java-5.1.25.jar"
>
> Any help here is appreciated.
>
> Thanks ..
>