HeartSaVioR edited a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-466301469 Commit 9e45876 pretty much covers it: it removes obsolete file entries from metadata as well as SeenFilesMap. New file which has same path can be picked up as new source after metadata is compacted, so it will eventually happen, not immediately. As I commented earlier, I'll put this commit in separate branch (will leave link here), and rebase this branch to before this commit. Just after build finishes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
