zhoulii opened a new issue, #11200:
URL: https://github.com/apache/paimon/issues/11200

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Deletion-vector materialization currently ends a rewrite task at each gap 
between selected row-ID ranges. When small selected ranges are separated by 
untouched ranges, this can produce many small output files and tasks.
   
   Materialization already filters deleted rows and assigns new row IDs. 
Packing complete selected ranges together can reduce output file and task 
counts while leaving independent unselected ranges unchanged.
   
   ### Solution
   
   - Remove the gap-based task boundary when packing selected components from 
the same partition and planning batch.
   - Preserve the existing size-based packing and keep each overlapping-file 
component intact; `target-file-size` remains a soft target.
   - Keep candidate selection, overlapping-file closure, and ordinary 
compaction unchanged.
   - Add regressions for merged output across an untouched range and size-based 
task separation.
   
   ### Anything else?
   
   Builds on the explicit Data Evolution deletion-vector materialization 
introduced in [#9177](https://github.com/apache/paimon/pull/9177).
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to