HeartSaVioR edited a comment on issue #23782: [SPARK-26875][SS] Add an option on FileStreamSource to include modified files URL: https://github.com/apache/spark/pull/23782#issuecomment-463866912 Maybe @gaborgsomogyi is considering more than what it is as of now. Currently #22952 is implemented as synchronous approach, but @gaborgsomogyi had his voice to make archiving/deletion being async in #22952 (we talked about dealing with it in next TODO), and it's based on the assumption that we never process the file again even same file is added as new. When the option is turned on, it breaks assumption and things will get changed very differently - including race condition what @gaborgsomogyi is stated. When we are hit by race condition It would produce unintended result (maybe worse scenario would be archive/delete a new file before processing which end users expect to get them processed). (EDIT: Looks like things will be same even without asynchronous approach.) Race condition still occurs between Spark and FileSystem even without this feature (just in #22952), but that's not Spark can control and Spark just should make its best to archive/delete faster before others overwrite/delete them. It doesn't still break query semantic or output even we are hit by race condition.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
