GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/19140
[SPARK-21890] Credentials not being passed to add the tokens
I observed this while running a oozie job trying to connect to hbase via
spark.
It look like the creds are not being passed in
thehttps://github.com/apache/spark/blob/branch-2.2/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala#L53
for 2.2 release.
More Info as to why it fails on secure grid:
Oozie client gets the necessary tokens the application needs before
launching. It passes those tokens along to the oozie launcher job (MR job)
which will then actually call the Spark client to launch the spark app and pass
the tokens along.
The oozie launcher job cannot get anymore tokens because all it has is
tokens ( you can't get tokens with tokens, you need tgt or keytab).
The error here is because the launcher job runs the Spark Client to submit
the spark job but the spark client doesn't see that it already has the hdfs
tokens so it tries to get more, which ends with the exception.
There was a change with SPARK-19021 to generalize the hdfs credentials
provider that changed it so we don't pass the existing credentials into the
call to get tokens so it doesn't realize it already has the necessary tokens.
https://issues.apache.org/jira/browse/SPARK-21890
Modified to pass creds to get delegation tokens
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/redsanket/spark SPARK-21890-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19140
----
commit ebbcb7f887557e09eacc56dbd7bb88da445b8fa2
Author: Sanket Chintapalli <[email protected]>
Date: 2017-09-01T20:32:20Z
Credentials not being passed to gather the tokens
commit 0cfca504e3ee30c1cb62ae5976a7784292418f45
Author: Sanket Chintapalli <[email protected]>
Date: 2017-09-05T19:43:50Z
Feel fetchCreds is appropriate naming convention
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
