[
https://issues.apache.org/jira/browse/SPARK-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-2508.
------------------------------
Resolution: Won't Fix
Target Version/s: 1.0.0, 0.9.1, 0.9.0 (was: 0.9.0, 0.9.1, 1.0.0)
No response on the PR; auto-closed.
> fail to remove staging dir when app quits in a yarn-cluster
> -----------------------------------------------------------
>
> Key: SPARK-2508
> URL: https://issues.apache.org/jira/browse/SPARK-2508
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 0.9.0, 0.9.1, 1.0.0
> Environment: RedHat 5.3
> 2.0.0-cdh4.6.0
> java version "1.6.0_45"
> Reporter: James Z.M. Gao
> Priority: Minor
>
> Hi,
> In a yarn-cluster, the driver is actually running as 'yarn' user in some
> container by a nodemanager. When
> submitting jobs from other users, e.g. user1, the stagingDir is created in
> submitting stage, which is actually a progress run by user1. So the full path
> of stagingDir on the hdfs should look like '/user/user1/.staging/...'.
> And in cleaning up stage, stagingDir could be removed by the driver. At this
> time, we need give stagingDir a full path, so
> the driver could remove the correct directory ('/user/user1/.staging/...'),
> not '/user/yarn/.staging/...'.
> Here is a simple PR: https://github.com/apache/spark/pull/154
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
