Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/19848#discussion_r157063770
--- Diff:
core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala ---
@@ -70,7 +70,8 @@ object SparkHadoopMapRedUtil extends Logging {
if (shouldCoordinateWithDriver) {
val outputCommitCoordinator = SparkEnv.get.outputCommitCoordinator
val taskAttemptNumber = TaskContext.get().attemptNumber()
- val canCommit = outputCommitCoordinator.canCommit(jobId, splitId,
taskAttemptNumber)
+ val stageId = TaskContext.get().stageId()
+ val canCommit = outputCommitCoordinator.canCommit(stageId,
splitId, taskAttemptNumber)
--- End diff --
Ever thought of returning why the commit was refused, e.g: unknown stage ID
vs other task attempt committed, vs you ar considered failed?
Not that the task committer should behave differently, but it might be nice
to pass that info back for logging alone
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
