-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56271/
-----------------------------------------------------------
Review request for sentry, Alexander Kolbasov, Hao Hao, kalyan kumar
kalvagadda, and Vadim Spector.
Repository: sentry
Description
-------
Looking at the existing logic of HMSFollower, upon the connection loss to HMS
the client never attempts to reconnect to HMS but rather just errors out with
"Broken pipe" indefinitely.
```bash
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
...
...
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
...
...
```
To improve this, we can extend HMSFollower to scan for SocketException errors
and if received, it can instead reconnect to HMS.
Diffs
-----
sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/FullUpdateInitializer.java
a1f970b1e6f14eedca057eb84df72de955d6f0c1
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java
1f05ba9cb6483251821f2965808dfd0711e70c63
Diff: https://reviews.apache.org/r/56271/diff/
Testing
-------
Ran the scenario of:
1. Start with Sentry + HMS connected.
2. Stop the HMS and verify the broken pipe errors reported on Sentry side and
it's attempts to establish a new connection.
3. Start the HMS and verify the subsequent attempt of HMS client connection
succeeds and processes transactions.
```bash
########################################## FIRST STEP
##################################################
2017-02-02 19:59:43,724 INFO org.apache.sentry.service.thrift.HMSFollower:
Making a kerberos connection to HMS
2017-02-02 19:59:43,729 INFO org.apache.sentry.service.thrift.HMSFollower:
Using kerberos principal:
sentry/[email protected]
2017-02-02 19:59:43,729 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new
Context
2017-02-02 19:59:43,753 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer
thread started
2017-02-02 19:59:43,760 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 19:59:43,864 INFO hive.metastore: Opened a connection to metastore,
current connections: 1
2017-02-02 19:59:43,865 INFO hive.metastore: Connected to metastore.
2017-02-02 19:59:43,865 INFO org.apache.sentry.service.thrift.HMSFollower:
Secure connection established with HMS
2017-02-02 19:59:43,865 INFO org.apache.sentry.service.thrift.HMSFollower:
HMSFollower of Sentry successfully connected to HMS
2017-02-02 19:59:43,978 INFO org.apache.sentry.service.thrift.HMSFollower:
Before fetching hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 19:59:45,005 INFO org.apache.sentry.service.thrift.HMSFollower: ####
Hive full update initialization complete !!
2017-02-02 19:59:45,008 INFO org.apache.sentry.service.thrift.HMSFollower:
After fetching hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 19:59:45,009 INFO org.apache.sentry.service.thrift.HMSFollower:
Successfully fetched hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 19:59:45,096 INFO org.apache.sentry.service.thrift.HMSFollower:
CurrentEventID = 0. Processing 0 events
....
########################################## SECOND STEP
##################################################
2017-02-02 20:00:03,314 ERROR org.apache.sentry.service.thrift.HMSFollower:
ThriftException occured fetching Notification entries, will try
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
...
...
2017-02-02 20:00:03,317 ERROR org.apache.sentry.service.thrift.HMSFollower:
Encountered Socket Exception during fetching Notification entries, will
reconnect to HMS
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
...
...
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
...
...
2017-02-02 20:00:03,323 INFO hive.metastore: Closed a connection to metastore,
current connections: 0
2017-02-02 20:00:03,435 INFO org.apache.sentry.service.thrift.HMSFollower:
Making a kerberos connection to HMS
2017-02-02 20:00:03,435 INFO org.apache.sentry.service.thrift.HMSFollower:
Using kerberos principal:
sentry/[email protected]
2017-02-02 20:00:03,435 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new
Context
2017-02-02 20:00:03,443 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 20:00:03,449 WARN hive.metastore: Failed to connect to the MetaStore
Server...
2017-02-02 20:00:03,449 INFO hive.metastore: Waiting 1 seconds before next
connection attempt.
2017-02-02 20:00:03,449 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer
thread started
2017-02-02 20:00:03,759 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer
thread finished
2017-02-02 20:00:04,449 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 20:00:04,451 WARN hive.metastore: Failed to connect to the MetaStore
Server...
2017-02-02 20:00:04,451 INFO hive.metastore: Waiting 1 seconds before next
connection attempt.
2017-02-02 20:00:05,452 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 20:00:05,453 WARN hive.metastore: Failed to connect to the MetaStore
Server...
2017-02-02 20:00:05,453 INFO hive.metastore: Waiting 1 seconds before next
connection attempt.
2017-02-02 20:00:06,456 ERROR org.apache.sentry.service.thrift.HMSFollower:
Failed to setup secure connection to HMS.
2017-02-02 20:00:06,456 ERROR org.apache.sentry.service.thrift.HMSFollower:
HMSFollower cannot connect to HMS!!
java.security.PrivilegedActionException: MetaException(message:Could not
connect to meta store using any of the URIs provided. Most recent failure:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
...
...
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
...
...
Caused by: MetaException(message:Could not connect to meta store using any of
the URIs provided. Most recent failure:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
...
...
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
...
...
########################################## THIRD STEP
##################################################
2017-02-02 20:01:29,266 INFO org.apache.sentry.service.thrift.HMSFollower:
Making a kerberos connection to HMS
2017-02-02 20:01:29,266 INFO org.apache.sentry.service.thrift.HMSFollower:
Using kerberos principal:
sentry/[email protected]
2017-02-02 20:01:29,266 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new
Context
2017-02-02 20:01:29,277 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 20:01:29,279 WARN hive.metastore: Failed to connect to the MetaStore
Server...
2017-02-02 20:01:29,279 INFO hive.metastore: Waiting 1 seconds before next
connection attempt.
2017-02-02 20:01:29,279 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer
thread started
2017-02-02 20:01:30,202 INFO
org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer
thread finished
2017-02-02 20:01:30,279 INFO hive.metastore: Trying to connect to metastore
with URI thrift://vamsee-sentry-1.vpc.cloudera.com:9083
2017-02-02 20:01:30,311 INFO hive.metastore: Opened a connection to metastore,
current connections: 1
2017-02-02 20:01:30,311 INFO hive.metastore: Connected to metastore.
2017-02-02 20:01:30,311 INFO org.apache.sentry.service.thrift.HMSFollower:
Secure connection established with HMS
2017-02-02 20:01:30,311 INFO org.apache.sentry.service.thrift.HMSFollower:
HMSFollower of Sentry successfully connected to HMS
2017-02-02 20:01:30,437 INFO org.apache.sentry.service.thrift.HMSFollower:
Before fetching hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 20:01:31,451 INFO org.apache.sentry.service.thrift.HMSFollower: ####
Hive full update initialization complete !!
2017-02-02 20:01:31,454 INFO org.apache.sentry.service.thrift.HMSFollower:
After fetching hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 20:01:31,454 INFO org.apache.sentry.service.thrift.HMSFollower:
Successfully fetched hive full snapshot, Current NotificationID =
CurrentNotificationEventId(eventId:0).
2017-02-02 20:01:31,516 INFO org.apache.sentry.service.thrift.HMSFollower:
CurrentEventID = 0. Processing 0 events
```
Thanks,
Vamsee Yarlagadda
