[
https://issues.apache.org/jira/browse/SPARK-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110730#comment-14110730
]
Thomas Graves edited comment on SPARK-2541 at 8/26/14 2:11 PM:
---------------------------------------------------------------
So I haven't had time to full investigate this to make sure it works in all the
various cluster setups. You can try this patch which is basically what the
code did in the 0.9 release:
What it does it if you are already the current user it doesn't do the doAs,
which will then let you access hfds using your current credentials. If you
are running the standalone cluster as one user and trying to access HDFS as
another this may not work for you.
Patch didn't come through, I'll attach it separately.
was (Author: tgraves):
So I haven't had time to full investigate this to make sure it works in all the
various cluster setups. You can try this patch which is basically what the
code did in the 0.9 release:
What it does it if you are already the current user it doesn't do the doAs,
which will then let you access hfds using your current credentials. If you
are running the standalone cluster as one user and trying to access HDFS as
another this may not work for you.
---
core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
index 148115d..e9b6891 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
@@ -44,8 +44,10 @@ class SparkHadoopUtil extends Logging {
* do a FileSystem.closeAllForUGI in order to avoid leaking Filesystems
*/
def runAsSparkUser(func: () => Unit) {
+ val currentUser = Option(System.getProperty("user.name")).
+ getOrElse(SparkContext.SPARK_UNKNOWN_USER)
val user =
Option(System.getenv("SPARK_USER")).getOrElse(SparkContext.SPARK_UNKNOWN_USER)
- if (user != SparkContext.SPARK_UNKNOWN_USER) {
+ if (user != SparkContext.SPARK_UNKNOWN_USER && currentUser != user) {
logDebug("running as user: " + user)
val ugi = UserGroupInformation.createRemoteUser(user)
transferCredentials(UserGroupInformation.getCurrentUser(), ugi)
--
> Standalone mode can't access secure HDFS anymore
> ------------------------------------------------
>
> Key: SPARK-2541
> URL: https://issues.apache.org/jira/browse/SPARK-2541
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.0.0, 1.0.1
> Reporter: Thomas Graves
>
> In spark 0.9.x you could access secure HDFS from Standalone deploy, that
> doesn't work in 1.X anymore.
> It looks like the issues is in SparkHadoopUtil.runAsSparkUser. Previously it
> wouldn't do the doAs if the currentUser == user. Not sure how it affects
> when the daemons run as a super user but SPARK_USER is set to someone else.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
