steveloughran commented on issue #25795: [WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed URL: https://github.com/apache/spark/pull/25795#issuecomment-533636905 Is there any way to block on the commit process to ensure that exactly one can be committing at the same time, eg. some lease file with a timestamp inside whose (unexpired) presence is a sign someone else is committing? for all filesystems where `create(path, overwrite=false)` is atomic you can be confident when you create the file that you have the permission to commit. For others, you either need to fail, spin, or decide when it is so out of date that you could delete it and go for the lock yourself. (I'm ignoring inconsistent storage without CRUD consistency or atomic creates, obviously)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
