Keep in mind that the authentication path for DelegationToken
(mapreduce) and KerberosToken are completely different.
Since most mapreduce jobs have multiple mappers (or reducers), I expect
we would have run into the case that the same DelegationToken was used
multiple times. It would still be good to narrow down the scope of the
problem.
Xu (Simon) Chen wrote:
Thanks Josh...
I tested this in scala REPL, and called DataStoreFinder.getDataStore()
multiple times, each time it seems to be reusing the same
KerberosToken object, and it works fine each time.
So my problem only happens when the token is used in accumulo's mapred
package. Weird..
-Simon
On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser<josh.el...@gmail.com> wrote:
Simon,
Can you reproduce this in plain-jane Java code? I don't know enough about
spark/scala, much less what Geomesa is actually do, to know what the issue
is.
Also, which token are you referring to: A KerberosToken or a
DelegationToken? Either of them should be usable as many times as you'd like
(given the underlying credentials are still available for KT or the DT token
hasn't yet expired).
Xu (Simon) Chen wrote:
Folks,
I am working on geomesa+accumulo+spark integration. For some reason, I
found that the same token cannot be used to authenticate twice.
The workflow is that geomesa would try to create a hadoop rdd, during
which it tries to create an AccumuloDataStore:
https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
During this process, a ZooKeeperInstance is created:
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
I modified geomesa such that it would use kerberos to authenticate
here. This step works fine.
But next, geomesa calls ConfigurationBase.setConnectorInfo:
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
This is using the same token and the same zookeeper URI, for some
reason it is stuck on spark-shell, and the following is outputted on
tserver side:
2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
at
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
at
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 11 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 17 more
Any idea why?
Thanks.
-Simon
