Dmitry Lysnichenko created AMBARI-17258:
-------------------------------------------
Summary: Storm service check failed after Ambari upgrade
Key: AMBARI-17258
URL: https://issues.apache.org/jira/browse/AMBARI-17258
Project: Ambari
Issue Type: Bug
Reporter: Dmitry Lysnichenko
Assignee: Dmitry Lysnichenko
Attachments: AMBARI-17258.patch
Marked this as blocker since the issue affected many EU End to End test runs
that involved running a service check as part of pre-check condition for EU
ambari-server-2.4.0.0-665.x86_64
ambari-server --hash
*Steps*
# Deploy HDP-2.3.2.0 cluster with Ambari 2.1.2 (secure, non-HA cluster)
# Upgrade Ambari to 2.4.0.0-665
# Stop and Start all services
# Run service check for Storm
*Result*
Failure as below:
{code}
stderr: /var/lib/ambari-agent/data/errors-729.txt
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py",
line 79, in <module>
ServiceCheck().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 257, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/service_check.py",
line 70, in service_check
user=params.storm_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line
155, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
293, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'storm jar
/tmp/wordCount.jar storm.starter.WordCountTopology
WordCountid16ac396c_date241516' returned 1. 951 [main] INFO b.s.u.Utils -
Using defaults.yaml from resources
1088 [main] INFO b.s.u.Utils - Using storm.yaml from resources
1184 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
1240 [main] INFO b.s.u.Utils - Using storm.yaml from resources
1285 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for
MD5-digest: -7463484230184273671:-7128399429695564111
1287 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
1292 [main] WARN b.s.u.NimbusClient - Using deprecated config nimbus.host for
backward compatibility. Please update your storm.yaml so it only has config
nimbus.seeds
1352 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The
baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]
1615 [main] INFO o.a.s.z.Login - successfully logged in.
1650 [main] ERROR o.a.t.t.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
~[?:1.7.0_67]
at
org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.KrbException: Server not found in Kerberos
database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
1666 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Client failed to open
SaslClientTransport to interact with a server during session initiation:
org.apache.thrift7.transport.TTransportException: GSS initiate failed
org.apache.thrift7.transport.TTransportException: GSS initiate failed
at
org.apache.thrift7.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:316)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
1675 [main] WARN b.s.u.NimbusClient - Ignoring exception while trying to get
leader nimbus info from localhost. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: SASL authentication not
complete
at org.apache.thrift7.transport.TSaslTransport.write(TSaslTransport.java:474)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:62)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.send_getClusterInfo(Nimbus.java:553)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:546)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
Exception in thread "main" java.lang.RuntimeException: Could not find leader
nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus
hosts for config nimbus.seeds
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
stdout: /var/lib/ambari-agent/data/output-729.txt
2016-06-15 06:24:45,500 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
2016-06-15 06:24:45,503 - checked_call['hostid'] {}
2016-06-15 06:24:45,516 - checked_call returned (0, '16ac396c')
2016-06-15 06:24:45,517 - File['/tmp/wordCount.jar'] {'owner': 'storm',
'content': StaticFile('wordCount.jar')}
2016-06-15 06:24:45,526 - Execute['storm jar /tmp/wordCount.jar
storm.starter.WordCountTopology WordCountid16ac396c_date241516'] {'logoutput':
True, 'path': ['/usr/hdp/current/storm-client/bin'], 'user': 'storm'}
951 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
1088 [main] INFO b.s.u.Utils - Using storm.yaml from resources
1184 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
1240 [main] INFO b.s.u.Utils - Using storm.yaml from resources
1285 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for
MD5-digest: -7463484230184273671:-7128399429695564111
1287 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
1292 [main] WARN b.s.u.NimbusClient - Using deprecated config nimbus.host for
backward compatibility. Please update your storm.yaml so it only has config
nimbus.seeds
1352 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The
baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]
1615 [main] INFO o.a.s.z.Login - successfully logged in.
1650 [main] ERROR o.a.t.t.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
~[?:1.7.0_67]
at
org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism
level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.KrbException: Server not found in Kerberos
database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66) ~[?:1.7.0_67]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) ~[?:1.7.0_67]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) ~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
~[?:1.7.0_67]
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
~[?:1.7.0_67]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
~[?:1.7.0_67]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
~[?:1.7.0_67]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[?:1.7.0_67]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[?:1.7.0_67]
... 18 more
1666 [main] ERROR b.s.s.a.k.KerberosSaslTransportPlugin - Client failed to open
SaslClientTransport to interact with a server during session initiation:
org.apache.thrift7.transport.TTransportException: GSS initiate failed
org.apache.thrift7.transport.TTransportException: GSS initiate failed
at
org.apache.thrift7.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:316)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:145)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:141)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_67]
at
backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:140)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:104)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:95)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:67)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
1675 [main] WARN b.s.u.NimbusClient - Ignoring exception while trying to get
leader nimbus info from localhost. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: SASL authentication not
complete
at org.apache.thrift7.transport.TSaslTransport.write(TSaslTransport.java:474)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
org.apache.thrift7.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:62)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.send_getClusterInfo(Nimbus.java:553)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:546)
~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at
backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
[wordCount.jar:?]
Exception in thread "main" java.lang.RuntimeException: Could not find leader
nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus
hosts for config nimbus.seeds
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:225)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
Command failed after 1 tries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
