HeartSaVioR commented on issue #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log URL: https://github.com/apache/spark/pull/24128#issuecomment-474149223 This is a commit based on master for the last alternative (exclude old output files in metadata): https://github.com/HeartSaVioR/spark/commit/8564ab44a45db0cbf5d09dcc3b3c993f440b264e I guess this is the least intrusive approach and simple. One thing to note is that excluding output files are reflected when compacting batches so it depends on compaction interval as well as batch interval. We can safely document it as "eventually happens". @jose-torres What do you think about this approach?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
