[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-26949: ---------------------------------- Target Version/s: 3.0.0 > Prevent "purge" to remove needed batch files in CompactibleFileStreamLog > ------------------------------------------------------------------------ > > Key: SPARK-26949 > URL: https://issues.apache.org/jira/browse/SPARK-26949 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.0.0 > Reporter: Jungtaek Lim > Assignee: Jungtaek Lim > Priority: Minor > > I've seen couple of trials (in opened PRs, even I've also tried) which calls > purge() in CompactibleFileStreamLog, but after looking at the codebase of > CompactibleFileStreamLog, I've realized that purging latest compaction batch > would break internal of CompactibleFileStreamLog and it throws > IllegalStateException. > Given that CompactibleFileStreamLog maintains the batches and purges > according to its configuration, it would be safer to just rely on > CompactibleFileStreamLog to purge and prevent calling `purge` outside of > CompactibleFileStreamLog. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org