turboFei commented on issue #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task URL: https://github.com/apache/spark/pull/26339#issuecomment-613226194 > IIUC, only one task attempt should commit at a time. If so, will there still be conflicts with task speculation when the first task commit successfully ? Got, It seems that we need judge whether the commit operation performed in SparkHadoopMapRedUtil.commitTask. Then we need revert the performed commit status if some exception happened when renaming dynacmicStagingTaskFiles.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
