> On Feb. 1, 2017, 11:17 p.m., Alexander Kolbasov wrote: > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java, > > line 182 > > <https://reviews.apache.org/r/56191/diff/1/?file=1621825#file1621825line182> > > > > I trhink we should sleep here for a while before retrying. > > > > Also we shouldn't continue in this case - we should either retry > > forever or fail completely.
It looks the HMS client itself has a retry logic defined in it. ```bash 2017-02-01 18:44:36,102 INFO org.apache.sentry.service.thrift.HMSFollower: Established kerberos context, will now connect to HMS 2017-02-01 18:44:49,646 INFO hive.metastore: Trying to connect to metastore with URI thrift://nightly-1.gce.cloudera.com:9083 2017-02-01 18:44:51,604 WARN hive.metastore: Failed to connect to the MetaStore Server... 2017-02-01 18:44:51,605 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 2017-02-01 18:44:52,605 INFO hive.metastore: Trying to connect to metastore with URI thrift://nightly-1.gce.cloudera.com:9083 2017-02-01 18:44:52,623 WARN hive.metastore: Failed to connect to the MetaStore Server... 2017-02-01 18:44:52,623 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 2017-02-01 18:44:53,624 INFO hive.metastore: Trying to connect to metastore with URI thrift://nightly-1.gce.cloudera.com:9083 2017-02-01 18:44:53,644 WARN hive.metastore: Failed to connect to the MetaStore Server... 2017-02-01 18:44:53,644 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 2017-02-01 18:44:54,644 INFO hive.metastore: Trying to connect to metastore with URI thrift://nightly-1.gce.cloudera.com:9083 2017-02-01 18:44:54,661 WARN hive.metastore: Failed to connect to the MetaStore Server... 2017-02-01 18:44:54,661 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 2017-02-01 18:44:55,661 INFO hive.metastore: Trying to connect to metastore with URI thrift://nightly-1.gce.cloudera.com:9083 2017-02-01 18:44:55,674 WARN hive.metastore: Failed to connect to the MetaStore Server... 2017-02-01 18:44:55,674 INFO hive.metastore: Waiting 1 seconds before next connection attempt. ``` Although, we can still preserve the retry logic in our HMSFollower to back off considerably if connection setup fails using the HMS client to avoid too much network traffic. Modified code to give up after the re-tries so that the subsequent re-run of HMSFollower instantiates new connection as per the thread execution logic defined in SentryService.java (https://github.com/apache/sentry/blob/d5176b2ea2e80d51f49d9e13075d794436fbe504/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java#L161-L162) - Vamsee ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56191/#review163899 ----------------------------------------------------------- On Feb. 1, 2017, 10:30 p.m., Vamsee Yarlagadda wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56191/ > ----------------------------------------------------------- > > (Updated Feb. 1, 2017, 10:30 p.m.) > > > Review request for sentry, Alexander Kolbasov, Lei Xu, Hao Hao, kalyan kumar > kalvagadda, and Vadim Spector. > > > Repository: sentry > > > Description > ------- > > SENTRY-1619: Fix the secure HMS connection code in HMSFollower > -- HMSFollower has bugs in implementing logic that's preventing secure HMS > connection from taking place. > > 1. Need to use the right principal and keytab to use for the connection. > 2. Fix the while blocks to allow the connection setup to happen. > > > Diffs > ----- > > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java > 749c2ce8f89fe5960af5a4b48ff45a38091350f4 > > Diff: https://reviews.apache.org/r/56191/diff/ > > > Testing > ------- > > Sentry server log: > ```bash > 2017-02-01 14:02:48,537 INFO org.apache.sentry.service.thrift.HMSFollower: > HMSFollower is being initialized > 2017-02-01 14:02:50,545 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 2017-02-01 14:02:51,911 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:02:57,186 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:02:57,335 INFO org.apache.sentry.service.thrift.SentryService: > Attempting to start... > 2017-02-01 14:02:57,345 INFO > org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new > Context > 2017-02-01 14:02:57,403 INFO org.apache.sentry.service.thrift.SentryService: > ProcessorFactory being used: > org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessorFactory > 2017-02-01 14:02:57,711 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 2017-02-01 14:02:58,969 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:02:59,035 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:02:59,533 INFO org.apache.sentry.hdfs.SentryPlugin: Sentry HDFS > plugin initialized !! > 2017-02-01 14:02:59,561 INFO org.apache.sentry.service.thrift.SentryService: > ProcessorFactory being used: > org.apache.sentry.provider.db.generic.service.thrift.SentryGenericPolicyProcessorFactory > 2017-02-01 14:02:59,699 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 2017-02-01 14:03:00,776 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:03:00,854 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections > < 1. Setting to 20 > 2017-02-01 14:03:00,917 INFO org.apache.sentry.service.thrift.SentryService: > ProcessorFactory being used: > org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory > 2017-02-01 14:03:00,917 INFO > org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory: Calling > registerProcessor from SentryHDFSServiceProcessorFactory > 2017-02-01 14:03:00,945 INFO org.apache.sentry.service.thrift.SentryService: > Serving on nightly-1.gce.cloudera.com/172.31.112.33:8038 > 2017-02-01 14:03:01,133 INFO org.eclipse.jetty.server.Server: > jetty-7.6.16.v20140903 > 2017-02-01 14:03:01,159 INFO org.eclipse.jetty.server.handler.ContextHandler: > started o.e.j.s.h.ContextHandler{/,null} > 2017-02-01 14:03:01,180 INFO org.eclipse.jetty.server.handler.ContextHandler: > started o.e.j.s.ServletContextHandler{/,null} > 2017-02-01 14:03:01,211 INFO org.eclipse.jetty.server.AbstractConnector: > Started [email protected]:29000 > 2017-02-01 14:03:01,510 WARN org.apache.sentry.hdfs.SentryPlugin: Recieved > Authz Path FULL update [6].. > > 2017-02-01 14:04:23,514 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS > 2017-02-01 14:04:46,255 INFO org.apache.sentry.service.thrift.HMSFollower: > Using kerberos principal: sentry/[email protected] > 2017-02-01 14:05:15,016 INFO > org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new > Context > 2017-02-01 14:05:20,783 INFO org.apache.sentry.service.thrift.HMSFollower: > Established kerberos context, will now connect to HMS > 2017-02-01 14:05:35,212 INFO hive.metastore: Trying to connect to metastore > with URI thrift://nightly-1.gce.cloudera.com:9083 > 2017-02-01 14:05:37,210 INFO hive.metastore: Opened a connection to > metastore, current connections: 1 > 2017-02-01 14:05:37,212 INFO hive.metastore: Connected to metastore. > > ``` > > > Thanks, > > Vamsee Yarlagadda > >
