[ https://issues.apache.org/jira/browse/SPARK-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-7791: -------------------------------- Labels: bulk-closed (was: ) > Set user for executors in standalone-mode > ----------------------------------------- > > Key: SPARK-7791 > URL: https://issues.apache.org/jira/browse/SPARK-7791 > Project: Spark > Issue Type: Wish > Components: Spark Core > Reporter: Tomasz Früboes > Priority: Major > Labels: bulk-closed > > I'm opening this following a discussion in > https://www.mail-archive.com/user@spark.apache.org/msg28633.html > Our setup was following. Spark (1.3.1, prebuilt for hadoop 2.6, also 2.4) > was installed in the standalone mode and started manually from the root > account. Everything worked properly apart of operations such us > rdd.saveAsPickleFile(ofile) > which end with exception: > py4j.protocol.Py4JJavaError: An error occurred while calling o27.save. > : java.io.IOException: Failed to rename > DeprecatedRawLocalFileStatus{path=file:/mnt/lustre/bigdata/med_home/tmp/test19EE/namesAndAges.parquet2/_temporary/0/task_201505191540_0009_r_000001/part-r-00002.parquet; > isDirectory=false; length=534; replication=1; blocksize=33554432; > modification_time=1432042832000; access_time=0; owner=; group=; > permission=rw-rw-rw-; isSymlink=false} to > file:/mnt/lustre/bigdata/med_home/tmp/test19EE/namesAndAges.parquet2/part-r-00002.parquet > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346) > (files created in _temporary were owned by user root). It would be great if > spark could set the user for the executor also in standalone mode. Setting > SPARK_USER has no effect here. > BTW it may be a good idea to add some warning (e.g. during spark startup) > that running from root account is not very healthy idea. E.g. mapping this > function > def test(x): > f = open('/etc/testTMF.txt', 'w') > return 0 > on a rdd creates a file in /etc/ (surprisingly calls like f.Write("text") end > with an exception) > Thanks, > Tomasz -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org