[
https://issues.apache.org/jira/browse/SENTRY-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851131#comment-15851131
]
Hadoop QA commented on SENTRY-1621:
-----------------------------------
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12850766/SENTRY-1621.01-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/2302/console
This message is automatically generated.
> HMSFollower to retry connecting to HMS upon connection loss
> -----------------------------------------------------------
>
> Key: SENTRY-1621
> URL: https://issues.apache.org/jira/browse/SENTRY-1621
> Project: Sentry
> Issue Type: Sub-task
> Components: Hive Binding
> Reporter: Vamsee Yarlagadda
> Assignee: Vamsee Yarlagadda
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1621.01-sentry-ha-redesign.patch
>
>
> Looking at the existing logic of HMSFollower, upon the connection loss to HMS
> the client never attempts to reconnect to HMS but rather just errors out with
> "Broken pipe" indefinitely.
> {code}
> 2017-02-02 20:00:03,314 ERROR org.apache.sentry.service.thrift.HMSFollower:
> ThriftException occured fetching Notification entries, will try
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId(ThriftHiveMetastore.java:4233)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId(ThriftHiveMetastore.java:4221)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId(HiveMetaStoreClient.java:2030)
> at
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:232)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2017-02-02 20:00:03,317 ERROR org.apache.sentry.service.thrift.HMSFollower:
> Encountered Socket Exception during fetching Notification entries, will
> reconnect to HMS
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Broken pipe
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> at
> org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501)
> at
> org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.flush(TFilterTransport.java:77)
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_current_notificationEventId(ThriftHiveMetastore.java:4227)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId(ThriftHiveMetastore.java:4220)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId(HiveMetaStoreClient.java:2030)
> at
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:227)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ... 16 more
> {code}
> To improve this, we can extend HMSFollower to scan for SocketException errors
> and if received, it can instead reconnect to HMS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
