alfredo-gimenez edited a comment on issue #23840: [WIP][DISCUSSION_NEEDED][SPARK-24295][SS] Add option to retain only last batch in file stream sink metadata URL: https://github.com/apache/spark/pull/23840#issuecomment-467142031 Taking a look at this, as it could fix a longstanding critical performance bug in our ingestion pipeline. Couple comments: 1. When retaining only the last metadata log, we should probably also disable log file compaction, since it will just copy the last log. 2. Can this be used in conjunction with checkpoints? If checkpoints are present, they depend on the metadata log contents. If they are guaranteed to only use the last metadata log, this is fine, otherwise this case will have to be handled separately.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
