GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/21577
[WIP] [SPARK-24552][core] Correctly identify tasks in output commit
coordinator.
When an output stage is retried, it's possible that tasks from the previous
attempt are still running. In that case, there would be a new task for the
same partition in the new attempt, and the coordinator would allow both
tasks to commit their output since it did not keep track of stage attempts.
The change adds more information to the stage state tracked by the
coordinator,
so that only one task if allowed to commit the output in the above case.
This also removes some code added in SPARK-18113 that allowed for duplicate
commit requests; with the RPC code used in Spark 2, that situation cannot
happen, so there is no need to handle it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-24552
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21577.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21577
----
commit 09e5d158e5dda6af7d83e9714dad6a64c21adf17
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-06-15T23:21:58Z
[SPARK-24552][core] Correctly identify tasks in output commit coordinator.
When an output stage is retried, it's possible that tasks from the previous
attempt are still running. In that case, there would be a new task for the
same partition in the new attempt, and the coordinator would allow both
tasks to commit their output since it did not keep track of stage attempts.
The change adds more information to the stage state tracked by the
coordinator,
so that only one task if allowed to commit the output in the above case.
This also removes some code added in SPARK-18113 that allowed for duplicate
commit requests; with the RPC code used in Spark 2, that situation cannot
happen, so there is no need to handle it.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
