[
https://issues.apache.org/jira/browse/SPARK-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652443#comment-14652443
]
Marcelo Vanzin commented on SPARK-6508:
---------------------------------------
I believe both errors here will be fixed by SPARK-7736 (proper error handling)
and SPARK-6506 (SPARK_HOME issue, fixed long ago). Is there anything else?
> error handling issue running python in yarn cluster mode
> ---------------------------------------------------------
>
> Key: SPARK-6508
> URL: https://issues.apache.org/jira/browse/SPARK-6508
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Thomas Graves
>
> I was running python in yarn cluster mode and didn't have the SPARK_HOME
> envirnoment variables set. The client reported a failure of:
> ava.io.FileNotFoundException: File does not exist:
> hdfs://axonitered-nn1.red.ygrid.yahoo.com:8020/user/tgraves/.sparkStaging/application_1425530846697_59578/pi.py
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1201)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1193)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> But when you look at the application master log:
> Log Contents:
> Traceback (most recent call last):
> File "pi.py", line 29, in <module>
> sc = SparkContext(appName="PythonPi")
> File
> "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py",
> line 108, in __init__
> File
> "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py",
> line 222, in _ensure_initialized
> File
> "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/java_gateway.py",
> line 32, in launch_gateway
> File "/usr/lib64/python2.6/UserDict.py", line 22, in __getitem__
> raise KeyError(key)
> KeyError: 'SPARK_HOME'
> But the application master thought it succeeded and removed the staging
> directory when it shouldn't have:
> 15/03/24 14:50:07 INFO yarn.ApplicationMaster: Waiting for spark context
> initialization ...
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED,
> exitCode: 0, (reason: Shutdown hook called before final status was reported.)
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Unregistering
> ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before
> final status was reported.)
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Deleting staging directory
> .sparkStaging/application_1425530846697_59578
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
