HeartSaVioR commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-465861957 While I think it is safest way to only let CompactibleFileStreamLog to maintain logs, we have alternative options here: 1) Only purge batches which are safe to delete and don't throw exception Suppose it has batch 0, 1, 2, 3, 4 which batch 2 is compacted. If 4 is given as parameter, purge will try to remove 0, 1, 2, 3 which removing batch 2 (latest compaction batch) would break the internal state. Instead of this, this method could be overridden to remove only batch 0 and 1 and silently ignore removing 2 and 3. 2) Throw exception (maybe IllegalArgumentException, or IllegalStateException?) only when batches to purge contain the latest compaction batch This would selectively throw exception - when it can break internal state of CompactibleFileStreamLog. Please let me know if alternative would make more sense. Thanks in advance!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
