HeartSaVioR edited a comment on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata URL: https://github.com/apache/spark/pull/23840#issuecomment-467160556 FileStreamSink only reads the last batch of metadata to determine which batch sink wrote successfully. The huge metadata on FileStreamSink is actually not for FileStreamSink (it just effectively takes `the last successful batch ID`: it's for File(Stream)Source from another query. Now I think the ideal way to address is incorporating last successful batch ID to query checkpoint and only writing sink metadata when option is enabled (though maybe we should make it default to true since it breaks backward compatibility).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
