Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19848
WiP:
[a_zero_rename_committer.pdf](https://github.com/steveloughran/zero-rename-committer/files/1604894/a_zero_rename_committer.pdf)
I would really like some early review of the spark-side of that commit
algorithm, in the overall coverage & UML, and the deeper bits. I'm currently
unsure how the OutputCommitCoordinator gets told of failures of Executors/task
attempts, and I worry it has an expectation "you can reattempt to commit any
task where the committed attempt fails during task commit". I don't think that
guaranteed with FileOutputCommitter v2, which is a fact which the committer
itself doesn't declare to the callers. Could fix that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
