Kishor Patil created STORM-1596:
-----------------------------------
Summary: Multiple Subject sharing Kerberos TGT - causes services
to fail
Key: STORM-1596
URL: https://issues.apache.org/jira/browse/STORM-1596
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.10.0, 1.0.0, 0.10.1, 2.0.0
Reporter: Kishor Patil
Assignee: Kishor Patil
Priority: Critical
With multiple threads accessing same {{Subject}}, it can cause
{{ServiceTicket}} in use be by one thread be destroyed by another thread.
Running BasicDRPCTopology with high parallelism in secure cluster would
reproduce the issue.
Here is sample log from such a scenarios:
{code}
2016-01-20 15:52:26.904 o.a.t.t.TSaslTransport [ERROR] SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
~[?:1.8.0_40]
at
org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
~[storm-core-0.10.1.y.jar:0.10.1.y]
at
org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:195)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:191)
[storm-core-0.10.1.y.jar:0.10.1.y]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_40]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_40]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:190)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:54)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:109)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.drpc.DRPCInvocationsClient.reconnectClient(DRPCInvocationsClient.java:57)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.drpc.ReturnResults.reconnectClient(ReturnResults.java:113)
[storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.drpc.ReturnResults.execute(ReturnResults.java:103)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.daemon.executor$fn__6377$tuple_action_fn__6379.invoke(executor.clj:689)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__6301.invoke(executor.clj:448)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.disruptor$clojure_handler$reify__6018.onEvent(disruptor.clj:40)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:437)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:416)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-0.10.1.y.jar:0.10.1.y]
at
backtype.storm.daemon.executor$fn__6377$fn__6390$fn__6441.invoke(executor.clj:801)
[storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.util$async_loop$fn__742.invoke(util.clj:482)
[storm-core-0.10.1.y.jar:0.10.1.y]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism
level: The ticket isn't for us (35) - BAD TGS SERVER NAME)
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.8.0_40]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
~[?:1.8.0_40]
... 23 more
Caused by: sun.security.krb5.KrbException: The ticket isn't for us (35) - BAD
TGS SERVER NAME
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
~[?:1.8.0_40]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
~[?:1.8.0_40]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
~[?:1.8.0_40]
at
sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
~[?:1.8.0_40]
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.8.0_40]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
~[?:1.8.0_40]
... 23 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) ~[?:1.8.0_40]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65) ~[?:1.8.0_40]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
~[?:1.8.0_40]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
~[?:1.8.0_40]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
~[?:1.8.0_40]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
~[?:1.8.0_40]
at
sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
~[?:1.8.0_40]
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.8.0_40]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.8.0_40]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
~[?:1.8.0_40]
... 23 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
