HeartSaVioR opened a new pull request #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log URL: https://github.com/apache/spark/pull/24128 ## What changes were proposed in this pull request? This patch proposes to provide a new option to disable metadata log in FileStreamSink, which is only leveraged when Spark reads the output directory. The metadata log greatly helps to easily achieve exactly-once but given the output path is open to arbitrary readers, there's no way to compact the metadata log, which ends up growing the metadata file as query runs for long time, especially for compacted batch. (There're some reports from end users which include their workarounds: [SPARK-24295](https://issues.apache.org/jira/browse/SPARK-24295)) This patch takes a different commit protocol which doesn't leverage metadata if disable metadata option is true, as well as leverages no-op metadata log to ensure the code is not much diverged for each mode. ## How was this patch tested? Added unit tests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
