dragonknightk commented on issue #5130:
URL: https://github.com/apache/paimon/issues/5130#issuecomment-6905735921

   Hi @Aitozi, I'd like to work on this. Here is my plan:
   
   - In batch mode, after packing, `AppendCompactCoordinator` puts the files 
left over in a partition into one extra task when there are at least 2 of them, 
even if they are fewer than `compaction.min.file-num`. A single file is not 
compacted, since rewriting it alone does not reduce the file count.
   - The coordinator already knows the mode from its `isStreaming` argument, 
which the Flink compact source, the Flink multi-table compact scan and the 
Spark compact procedure pass, so no caller or option changes.
   - Streaming is unchanged: leftover files still wait for `COMPACT_AGE`, 
because new files can still arrive there.
   
   In tests, a batch compaction now covers all files, including the leftovers, 
while a streaming coordinator still keeps them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to