jose-torres commented on issue #24128: [SPARK-27188][SS] FileStreamSink: provide a new option to disable metadata log URL: https://github.com/apache/spark/pull/24128#issuecomment-474149209 I'm not sure I agree with the premise. Adding a feature, even behind a flag, represents a judgment by the Spark community that the feature works well and an ongoing obligation for Spark committers to maintain and fix bugs in the feature. We shouldn't implement ad-hoc changes with unclear behavior and semantics just to fix an immediate symptom some end-users face. A retention policy generally makes sense to me, although I don't personally know enough about how Hadoop-compatible filesystems deal with timestamps to effectively review such a change.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
