dragonknightk commented on issue #5130: URL: https://github.com/apache/paimon/issues/5130#issuecomment-6905735921
Hi @Aitozi, I'd like to work on this. Here is my plan: - In batch mode, after packing, `AppendCompactCoordinator` puts the files left over in a partition into one extra task when there are at least 2 of them, even if they are fewer than `compaction.min.file-num`. A single file is not compacted, since rewriting it alone does not reduce the file count. - The coordinator already knows the mode from its `isStreaming` argument, which the Flink compact source, the Flink multi-table compact scan and the Spark compact procedure pass, so no caller or option changes. - Streaming is unchanged: leftover files still wait for `COMPACT_AGE`, because new files can still arrive there. In tests, a batch compaction now covers all files, including the leftovers, while a streaming coordinator still keeps them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
