zhoulii opened a new issue, #11271:
URL: https://github.com/apache/paimon/issues/11271

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Data files are rewritten in parallel, but DV index rewriting runs serially 
during commit preparation: on the Spark driver or in a single Flink operator. 
Bitmap reads, row-position remapping, and replacement index writes can become a 
bottleneck for large compaction batches.
   
   ### Solution
   
   - Separate metadata planning from bitmap rewriting and execute shared core 
tasks on Flink workers and Spark executors.
   - Pack work within each partition using 
`deletion-vector.index-file.target-size` as a soft target.
   - Keep all contributors to each output anchor together, and give each old 
index file one owner to preserve retained entries and avoid duplicate deletion.
   - Cover shared index files, commit retries, and task failure cleanup.
   
   ### Anything else?
   
   Applies to ordinary Data Evolution compaction and DV materialization. 
Metadata planning and snapshot commits remain centralized.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to