[
https://issues.apache.org/jira/browse/SPARK-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weizhong closed SPARK-5663.
---------------------------
Resolution: Won't Fix
> Delete appStagingDir on local file system
> -----------------------------------------
>
> Key: SPARK-5663
> URL: https://issues.apache.org/jira/browse/SPARK-5663
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Reporter: Weizhong
> Priority: Minor
>
> As we know, in yarn mode Client will create appStagingDir on file system, and
> AppMaster will delete this appStagingDir when it exit. If file system is HDFS
> then it will work OK.
> As we know, to run Spark on Tachyon will create a core-site.xml on
> SPARK_HOME/conf, so when load core-site.xml will read
> SPARK_HOME/conf/core-site.xml, and in SPARK_HOME/conf/core-site.xml don't set
> fs.defaultFS, so we will get local file system. So in yarn mode Client will
> create appStagingDir on local file system, and if Client and AppMaster are
> not in the same node, then the appStagingDir will not be deleted.
> To solve this issue, we can do:
> 1. add fs.defaultFS setting to SPARK_HOME/conf/core-site.xml so that when get
> file system will return HDFS
> 2. or cleanup appStagingDir while Client exit or stop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
