viirya edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite be unique URL: https://github.com/apache/spark/pull/26086#issuecomment-547684414 @turboFei > dynamicPartitionOverwrite would keep a filesToMove, so there is no risk to cause duplicate result. This is exactly the cases of using addedAbsPathFiles. Only paths in allAbsPathFiles will be in filesToMove. For dynamicPartitionOverwrite, it can be partitionPaths or addedAbsPathFiles used. It is not no risk to cause duplicate result, for this fix. If addedAbsPathFiles is used, because UUID is added to the filename so the filenames are unique, there is no duplicate result or file already existing exception, without this fix. That said this fix is doing the almost same thing as #24142. The comments there can also apply there. This is not a fix for that issue too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
