[
https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067498#comment-16067498
]
Alexander Kolbasov edited comment on SENTRY-1630 at 6/29/17 12:20 AM:
----------------------------------------------------------------------
[~vamsee] found the root cause of this problem. The {{FullUpdateInitializer}}
uses the same connection to HMS for multiple threads fetching information from
HMS and this messes up the connection.
We need to have a connection pool for Hive connections instead.
was (Author: akolb):
[~vamsee] found the root cause of this problem. The FUllUpdateInitializer uses
the same connection to HMS for multiple threads fetching information from HMS
and this messes up the connection.
We need to have a connection pool for Hive connections instead.
> out of sequence error in HMSFollower
> ------------------------------------
>
> Key: SENTRY-1630
> URL: https://issues.apache.org/jira/browse/SENTRY-1630
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Priority: Blocker
> Fix For: sentry-ha-redesign
>
>
> When HMSFollower exits in an abnormal way, for all the following connection
> would encounter out of sequence and SocketTimeoutException: Read timed out.
> Looking at HIVE-6893, it seems to relate to leakage connection problem.
> {noformat}2017-02-15 19:03:42,822 ERROR
> org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete
> successfully after 0 tries. Exception got:
> org.apache.thrift.TApplicationException: get_database failed: out of sequence
> response
> 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer:
> Task did not complete successfully after 0 tries. Exception got:
> MetaException(message:Got exception:
> org.apache.thrift.transport.TTransportException null)
> 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to
> metastore, current connections: 0
> 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower:
> Exception occurred persisting Hive full snapshot into DB
> java.lang.RuntimeException: org.apache.thrift.TApplicationException:
> get_database failed: out of sequence response
> at
> org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324)
> at
> org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343)
> at
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.TApplicationException: get_database failed: out
> of sequence response
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213)
> at
> org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256)
> at
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110)
> at
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150)
> at
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower:
> Making a kerberos connection to HMS{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
