[
https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feiwang updated SPARK-29037:
----------------------------
Description:
For a stage, whose tasks commit output, a task saves output to a staging dir
firstly, when all tasks of this stage success, all task output under staging
dir will be moved to destination dir.
However, when we kill an application, which is committing tasks' output, parts
of tasks' results will be kept in staging dir, which would not be cleared
gracefully.
Then we rerun this application and the new application will reuse this staging
dir.
And when the task commit stage of new application success, all task output
under this staging dir, which contains parts of old application's task output ,
would be moved to destination dir and the result is duplicated.
More common case, I think it is confused that several application running with
same root path simultaneously, they will have same staging dir for same jobId.
was:
Case:
A spark application was be killed due to long-running.
Then we re-run this application, we find that spark gives duplicated result.
> [Core] Spark gives duplicate result when an application was killed and rerun
> ----------------------------------------------------------------------------
>
> Key: SPARK-29037
> URL: https://issues.apache.org/jira/browse/SPARK-29037
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.1
> Reporter: feiwang
> Priority: Major
>
> For a stage, whose tasks commit output, a task saves output to a staging dir
> firstly, when all tasks of this stage success, all task output under staging
> dir will be moved to destination dir.
> However, when we kill an application, which is committing tasks' output,
> parts of tasks' results will be kept in staging dir, which would not be
> cleared gracefully.
> Then we rerun this application and the new application will reuse this
> staging dir.
> And when the task commit stage of new application success, all task output
> under this staging dir, which contains parts of old application's task output
> , would be moved to destination dir and the result is duplicated.
> More common case, I think it is confused that several application running
> with same root path simultaneously, they will have same staging dir for same
> jobId.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
