This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 827d371 [SPARK-25689][FOLLOW-UP][CORE] Get proxy user's delegation
tokens
827d371 is described below
commit 827d3718770e2c65821a1f28b889743ba872b7cf
Author: Yuming Wang <yumw...@ebay.com>
AuthorDate: Mon Mar 4 13:21:24 2019 -0800
[SPARK-25689][FOLLOW-UP][CORE] Get proxy user's delegation tokens
## What changes were proposed in this pull request?
This pr makes it get proxy user's delegation token, otherwise throws
`AccessControlException`([full
log](https://issues.apache.org/jira/browse/SPARK-25689?focusedCommentId=16780609&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16780609)):
```java
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
...
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:95)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:185)
```
How to reproduce this issue:
```shell
$ ssh user_admspark-getaway-host1
$ export HADOOP_PROXY_USER=user_a
$ spark-sql --master yarn
```
## How was this patch tested?
Test on our production environment.
Closes #23922 from wangyum/SPARK-25689.
Authored-by: Yuming Wang <yumw...@ebay.com>
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
---
.../apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
index 6e8aa47..dc0f21c 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
@@ -408,7 +408,7 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
val ugi = UserGroupInformation.getCurrentUser()
val tokens = if (dtm.renewalEnabled) {
dtm.start()
- } else if (ugi.hasKerberosCredentials()) {
+ } else if (ugi.hasKerberosCredentials() ||
SparkHadoopUtil.get.isProxyUser(ugi)) {
val creds = ugi.getCredentials()
dtm.obtainDelegationTokens(creds)
SparkHadoopUtil.get.serialize(creds)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
