On 6/7/17 3:54 PM, James Srinivasan wrote:
[snip]
Fortunately I found this:
https://github.com/apache/hive/blob/master/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloTableInputFormat.java
Is it a good example of Accumulo + MapReduce that I can copy?
That one is definitely over-kill. There's a bit of reflection in there to
work around older versions of Accumulo. However, it should be an example of
something that does work with Kerberos authentication.
Also, take note that Hive uses the InputFormat regardless of the execution
engine (local, MapReduce, Tez, etc). There are some comments to that effect
in the code. You can likely simplify those methods/blocks as well :)
Think those are two things I'll need to handle at some point anyways.
I think I'm setting all the AccumuloInputFormat statics correctly, and
see the DelegationToken in my job's and context's credentials.
However, my custom InputFormat's createRecordReader function needs to
connect to Accumulo to get some config. Am I right in thinking I need
to convert the Hadoop wrapped token (kind=ACCUMULO_AUTH_TOKEN) into an
Accumulo DelegationToken to create my connector? If so, how do I do
that?
Yes, you need to deserialize the AuthenticationToken from the
InputSplit. You can look back into the AccumuloInputFormat
implementation to see how this is done:
https://github.com/apache/accumulo/blob/f81a8ec7410e789d11941351d5899b8894c6a322/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java#L515-L518
calls
https://github.com/apache/accumulo/blob/f81a8ec7410e789d11941351d5899b8894c6a322/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java#L240-L243
calls
https://github.com/apache/accumulo/blob/f81a8ec7410e789d11941351d5899b8894c6a322/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L485-L500
This pulls the "DelegationTokenStub" out of the InputFormat and creates
a real Accumulo AuthenticationToken (which you can use with a Connector
per-usual).
