gaborgsomogyi commented on issue #23764: [SPARK-26825][SS] Fix temp checkpoint creation in cluster mode when default filesystem is not local. URL: https://github.com/apache/spark/pull/23764#issuecomment-473844390 > My concern is that this change could break other scenarios that currently work. For example, I don't think stateful operations will work properly with a local FS checkpoint; executors won't be able to see the same checkpoint data. @jose-torres Thank you for your time and good point. Not considered this scenario and agree that it may happen when executor becoming bad. I've double checked the sinks where temporary checkpoints are created and none of them guarantee fault tolerance. This change still fulfills this. On the other hand if you still has concerns we can change the approach to create the temp checkpoint on default FS but then I would document that the directory configured by `java.io.tmpdir` system variable should be writable on the default filesystem. Please share your opinion. @HeartSaVioR I agree with you and I would write out a warning when checkpoint (not just temp) is created on local FS (a good example is what Jose mentioned).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
