turboFei edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite be unique URL: https://github.com/apache/spark/pull/26086#issuecomment-547703757 @viirya > This is exactly the cases of using addedAbsPathFiles. Only paths in allAbsPathFiles will be in filesToMove. For dynamicPartitionOverwrite, it can be partitionPaths or addedAbsPathFiles used. It is not no risk to cause duplicate result, for this fix. Thanks for your reply. For this case you mentioned, we can check the code below. We can see that only after SparkHadoopMapRedUtil.commitTask completed, this task would send a TaskCommitMessage contains addedAbsPathFiles. https://github.com/apache/spark/blob/8682bb11ae72c89b84d4bb26b835736f5a788280/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L241-L247 And when committing job, the addedAbsPathFiles was obtained from the task commit messages. So, I think there is no risk for the case of using addedAbsPathFiles. https://github.com/apache/spark/blob/8682bb11ae72c89b84d4bb26b835736f5a788280/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L169-L172
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
