zhoulii opened a new issue, #11220:
URL: https://github.com/apache/paimon/issues/11220

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Data Evolution compaction initializes a deletion-vector maintainer for each 
affected partition by scanning the full index manifest and then filtering by 
partition. Multi-partition batches repeatedly decode the same metadata and 
allocate objects, even when the manifest is cached.
   
   On a table with hundreds of partitions, querying `$table_indexes` with 
`index_type = 'DELETION_VECTORS'` returned 34,395 index files (`COUNT(*)`) 
containing 2,161,645 DV entries (`SUM(row_count)`). Repeating the full scan for 
each affected partition adds avoidable work to commit preparation.
   
   
   ### Solution
   
   - Collect the partitions requiring DV maintenance and scan the snapshot's 
index manifest once per commit-preparation attempt, filtering to the relevant 
DV index entries as they are read.
   - Keep temporary grouping structures local to maintainer initialization and 
reuse metadata references during bitmap processing, without introducing a 
persistent full-table cache.
   - Add a multi-partition regression for the scan count and verify that peak 
heap usage does not increase.
   
   ### Anything else?
   
   This is a core optimization shared by Flink and Spark. DV rewriting, index 
file formats, and commit semantics remain unchanged;
   
   A local synthetic benchmark against `57c0e3fd48` used 120 partitions, 2,400 
DV index files, and 300,000 DV entries, with all partitions selected and a warm 
manifest cache. Each implementation ran in a separate JDK 11 JVM with Serial GC 
and a 128 MiB initial / 1,536 MiB maximum heap.
   
   | Metric | Before | After |
   | --- | ---: | ---: |
   | Metadata preparation time | 7.05 s | 1.67 s |
   | Main-thread allocated bytes | 9,776 MiB | 2,393 MiB |
   | Sampled peak JVM heap usage | 609 MiB | 329 MiB |
   
   These are single-run results for index scanning and maintainer 
initialization, excluding bitmap I/O and data compaction. Peak heap was sampled 
every 1 ms; the results do not represent end-to-end job speedup.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to