[
https://issues.apache.org/jira/browse/HIVE-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086299#comment-16086299
]
Josh Elser commented on HIVE-16973:
-----------------------------------
On master, MapReduce seems to be working correctly, but Tez appears to be
dropping the Accumulo delegation token that is added to the JobConf's
credentials.
{noformat}
2017-07-13T15:37:31,733 DEBUG [a763f46e-cea1-4dab-a716-5b75ee664436 main]
accumulo.AccumuloStorageHandler: All job tokens: [Kind: ACCUMULO_AUTH_TOKEN,
Service: AccumuloDelegationToken-3fee0d70-4411-4933-8c99-e5367e3bd658, Ident:
00 00 00 4b 18 12 6a 65 6c 73 65 72 40 45 58 41 4d 50 4c 45 2e 43 4f 4d 15 08
16 a2 bd a6 d7 a7 57 16 a2 cd 8a 98 ac 57 18 24 33 66 65 65 30 64 37 30 2d 34
34 31 31 2d 34 39 33 33 2d 38 63 39 39 2d 65 35 33 36 37 65 33 62 64 36 35 38
00]
2017-07-13T15:37:35,567 INFO [main] hdfs.DFSClient: Created
HDFS_DELEGATION_TOKEN token 111 for jelser on 10.200.31.19:8020
2017-07-13T15:37:35,568 INFO [a763f46e-cea1-4dab-a716-5b75ee664436 main]
security.TokenCache: Got dt for hdfs://hw10447.local:8020; Kind:
HDFS_DELEGATION_TOKEN, Service: 10.200.31.19:8020, Ident:
(HDFS_DELEGATION_TOKEN token 111 for jelser)
2017-07-13T15:37:39,979 DEBUG [a763f46e-cea1-4dab-a716-5b75ee664436 main]
security.SecurityUtil: Acquired token Kind: YARN_CLIENT_TOKEN, Service:
10.200.31.19:54340, Ident: (appAttemptId { application_id { id: 10
cluster_timestamp: 1499973213648 } attemptId: 1 } clientName:
"[email protected]")
2017-07-13T15:37:40,445 DEBUG [a763f46e-cea1-4dab-a716-5b75ee664436 main]
client.TezClientUtils: Credentials: #sessionTokens=2,
Services=10.200.31.19:8020(HDFS_DELEGATION_TOKEN),application_1499973213648_0010(tez.job)
2017-07-13T15:37:40,501 DEBUG [a763f46e-cea1-4dab-a716-5b75ee664436 main]
api.DAG: Credentials: #dagTokens=2,
Services=10.200.31.19:8020(HDFS_DELEGATION_TOKEN),application_1499973213648_0010(tez.job)
{noformat}
Then, in the tez dag log:
{noformat}
2017-07-13 15:37:41,061 [INFO] [IPC Server handler 0 on 54340]
|impl.VertexImpl|: Default container context for vertex_1499973213648_0010_1_00
[Map 1]=LocalResources: [...], environment: [...], credentials(token kinds):
[HDFS_DELEGATION_TOKEN,tez.job,], javaOpts: -server
-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -server
-Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC
-XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
-Dtez.root.logger=DEBUG,CLA
-Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
-Dlog4j.configuration=tez-container-log4j.properties
-Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex:
vertex_1499973213648_0010_1_00 [Map 1], Default Resources=<memory:2048,
vCores:1>
{noformat}
And when the {{getSplits}} method is called on my InputFormat
{noformat}
2017-07-13 15:37:42,339 [INFO] [InputInitializer {Map 1} #0]
|mr.HiveAccumuloTableInputFormat|: Job credential tokens: [Kind:
HDFS_DELEGATION_TOKEN, Service: 10.200.31.19:8020, Ident:
(HDFS_DELEGATION_TOKEN token 111 for jelser), Kind: tez.job, Service:
application_1499973213648_0010, Ident: 1e 61 70 70 6c 69 63 61 74 69 6f 6e 5f
31 34 39 39 39 37 33 32 31 33 36 34 38 5f 30 30 31 30]
{noformat}
This reads to me like the token that was added to the JobConf's credentials
section was dropped at some point (or just never actually added by the
HadoopShims code). I'll try to dig into this..
To be clear, this only happens on master with hive.execution.engine=tez.
branch-2 MR, branch-2 tez, master with MR is fine.
FYI [~sushanth]
> Fetching of Delegation tokens (Kerberos) for AccumuloStorageHandler fails in
> HS2
> --------------------------------------------------------------------------------
>
> Key: HIVE-16973
> URL: https://issues.apache.org/jira/browse/HIVE-16973
> Project: Hive
> Issue Type: Bug
> Components: Accumulo Storage Handler
> Reporter: Josh Elser
> Assignee: Josh Elser
> Attachments: HIVE-16973.001.patch, HIVE-16973.002.branch-2.patch,
> HIVE-16973.003.branch-2.patch, HIVE-16973.004-branch-2.patch,
> HIVE-16973.004.branch-2.patch, HIVE-16973.004-master.patch
>
>
> Had a report from a user that Kerberos+AccumuloStorageHandler+HS2 was broken.
> Looking into it, it seems like the bit-rot got pretty bad. You'll see
> something like the following:
> {noformat}
> Caused by: java.io.IOException: Failed to unwrap AuthenticationToken
> at
> org.apache.hadoop.hive.accumulo.HiveAccumuloHelper.unwrapAuthenticationToken(HiveAccumuloHelper.java:312)
>
> at
> org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableInputFormat.getSplits(HiveAccumuloTableInputFormat.java:122)
>
> {noformat}
> It appears that some of the code-paths changed since when I first did my
> testing (or I just did poor testing) and the delegation token was never being
> fetched/serialized. There also are some issues with fetching the delegation
> token from Accumulo properly which were addressed in ACCUMULO-4665
> I believe it would also be best to just update the dependency to use Accumulo
> 1.7 (drop 1.6 support) as it's lacking in this regard. These changes would
> otherwise get much more complicated with reflection -- Accumulo has moved on
> past 1.6, so let's do the same in Hive.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
