advancedxy commented on issue #25795: [WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed URL: https://github.com/apache/spark/pull/25795#issuecomment-533207078 > How does it work for `dynamicPartitionOverwrite=true`? The output path is staging dir which is unique. Can we detect 2 jobs that are writing to the same table with `dynamicPartitionOverwrite=true`? Looks like there is no easy way to detect concurrent writes with `dynamicPartitionOverwrite=true`. Maybe we should notice user about that limitation. User won't get duplicated result in this case, but the result could be messed(some part replaced by other part, while some part remains) when writing concurrently since no transaction is involved.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
