Thomas Graves created SPARK-6508:
------------------------------------
Summary: error handling issue running python in yarn cluster mode
Key: SPARK-6508
URL: https://issues.apache.org/jira/browse/SPARK-6508
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.3.0
Reporter: Thomas Graves
I was running python in yarn cluster mode and didn't have the SPARK_HOME
envirnoment variables set. The client reported a failure of:
ava.io.FileNotFoundException: File does not exist:
hdfs://axonitered-nn1.red.ygrid.yahoo.com:8020/user/tgraves/.sparkStaging/application_1425530846697_59578/pi.py
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1201)
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1193)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
But when you look at the application master log:
Log Contents:
Traceback (most recent call last):
File "pi.py", line 29, in <module>
sc = SparkContext(appName="PythonPi")
File
"/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py",
line 108, in __init__
File
"/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py",
line 222, in _ensure_initialized
File
"/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/java_gateway.py",
line 32, in launch_gateway
File "/usr/lib64/python2.6/UserDict.py", line 22, in __getitem__
raise KeyError(key)
KeyError: 'SPARK_HOME'
But the application master thought it succeeded and removed the staging
directory when it shouldn't have:
15/03/24 14:50:07 INFO yarn.ApplicationMaster: Waiting for spark context
initialization ...
15/03/24 14:50:08 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED,
exitCode: 0, (reason: Shutdown hook called before final status was reported.)
15/03/24 14:50:08 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster
with SUCCEEDED (diag message: Shutdown hook called before final status was
reported.)
15/03/24 14:50:08 INFO yarn.ApplicationMaster: Deleting staging directory
.sparkStaging/application_1425530846697_59578
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
