[
https://issues.apache.org/jira/browse/SENTRY-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sravya Tirukkovalur updated SENTRY-1264:
----------------------------------------
Description:
Seems like we are opening a connection to Sentry from HMS once per request when
client connection pool is not used. Some times this can lead to false errors
for reply attacks if requests are too close to each other. Seems like we may
have to do a retry in the client?
HMS log:
{noformat}
2016-05-01 20:06:03,832 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hive/xx@xxx (auth:KERBEROS)
cause:sentry.org.apache.thrift.transport.TTransportException: Peer indicated
failure: GSS initiate failed
2016-05-01 20:06:03,832 ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler:
MetaException(message:Failed to connect to Sentry service null)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.getSentryServiceClient(SentryMetastorePostEventListener.java:259)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryPrivileges(SentryMetastorePostEventListener.java:302)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryTablePrivilege(SentryMetastorePostEventListener.java:287)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.onDropTable(SentryMetastorePostEventListener.java:129)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1529)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1676)
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
at com.sun.proxy.$Proxy5.drop_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8923)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8907)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:681)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:676)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:676)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Sentry log:
{noformat}
2016-05-01 20:06:03,841 ERROR
sentry.org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
Failure unspecified at GSS-API level (Mechanism level: Request is a replay
(34))]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
at
sentry.org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
at
sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
at
sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at
sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
... 8 more
Caused by: KrbException: Request is a replay (34)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
at
sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
... 11 more
2016-05-01 20:06:03,842 ERROR
sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during
processing of message.
java.lang.RuntimeException:
sentry.org.apache.thrift.transport.TTransportException: GSS initiate failed
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: sentry.org.apache.thrift.transport.TTransportException: GSS initiate
failed
at
sentry.org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at
sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at
sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
{noformat}
was:
Seems like we are opening a connection to Sentry from HMS once per request when
client connection pool is not used. Some times this can lead to false errors
for reply attacks if requests are too close to each other. Seems like would be
best to reuse the connection? Theoritically, there should be no reason HMS
maintains a single connection to Sentry.
HMS log:
{noformat}
2016-05-01 20:06:03,832 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hive/xx@xxx (auth:KERBEROS)
cause:sentry.org.apache.thrift.transport.TTransportException: Peer indicated
failure: GSS initiate failed
2016-05-01 20:06:03,832 ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler:
MetaException(message:Failed to connect to Sentry service null)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.getSentryServiceClient(SentryMetastorePostEventListener.java:259)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryPrivileges(SentryMetastorePostEventListener.java:302)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryTablePrivilege(SentryMetastorePostEventListener.java:287)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.onDropTable(SentryMetastorePostEventListener.java:129)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1529)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1676)
at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
at com.sun.proxy.$Proxy5.drop_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8923)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8907)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:681)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:676)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:676)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Sentry log:
{noformat}
2016-05-01 20:06:03,841 ERROR
sentry.org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
Failure unspecified at GSS-API level (Mechanism level: Request is a replay
(34))]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
at
sentry.org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
at
sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
at
sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at
sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
... 8 more
Caused by: KrbException: Request is a replay (34)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
at
sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
... 11 more
2016-05-01 20:06:03,842 ERROR
sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during
processing of message.
java.lang.RuntimeException:
sentry.org.apache.thrift.transport.TTransportException: GSS initiate failed
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: sentry.org.apache.thrift.transport.TTransportException: GSS initiate
failed
at
sentry.org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at
sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at
sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
{noformat}
> Avoid false alerts of replay attacks from Sentry Clients
> --------------------------------------------------------
>
> Key: SENTRY-1264
> URL: https://issues.apache.org/jira/browse/SENTRY-1264
> Project: Sentry
> Issue Type: Improvement
> Reporter: Sravya Tirukkovalur
>
> Seems like we are opening a connection to Sentry from HMS once per request
> when client connection pool is not used. Some times this can lead to false
> errors for reply attacks if requests are too close to each other. Seems like
> we may have to do a retry in the client?
> HMS log:
> {noformat}
> 2016-05-01 20:06:03,832 WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hive/xx@xxx (auth:KERBEROS)
> cause:sentry.org.apache.thrift.transport.TTransportException: Peer indicated
> failure: GSS initiate failed
> 2016-05-01 20:06:03,832 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler:
> MetaException(message:Failed to connect to Sentry service null)
> at
> org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.getSentryServiceClient(SentryMetastorePostEventListener.java:259)
> at
> org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryPrivileges(SentryMetastorePostEventListener.java:302)
> at
> org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryTablePrivilege(SentryMetastorePostEventListener.java:287)
> at
> org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.onDropTable(SentryMetastorePostEventListener.java:129)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1529)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1676)
> at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy5.drop_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8923)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8907)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:681)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:676)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:676)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Sentry log:
> {noformat}
> 2016-05-01 20:06:03,841 ERROR
> sentry.org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> sentry.org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at
> sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at
> sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 8 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
> at
> sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 11 more
> 2016-05-01 20:06:03,842 ERROR
> sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during
> processing of message.
> java.lang.RuntimeException:
> sentry.org.apache.thrift.transport.TTransportException: GSS initiate failed
> at
> sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sentry.org.apache.thrift.transport.TTransportException: GSS
> initiate failed
> at
> sentry.org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
> at
> sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
