zhoulii opened a new issue, #11220: URL: https://github.com/apache/paimon/issues/11220
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Data Evolution compaction initializes a deletion-vector maintainer for each affected partition by scanning the full index manifest and then filtering by partition. Multi-partition batches repeatedly decode the same metadata and allocate objects, even when the manifest is cached. On a table with hundreds of partitions, querying `$table_indexes` with `index_type = 'DELETION_VECTORS'` returned 34,395 index files (`COUNT(*)`) containing 2,161,645 DV entries (`SUM(row_count)`). Repeating the full scan for each affected partition adds avoidable work to commit preparation. ### Solution - Collect the partitions requiring DV maintenance and scan the snapshot's index manifest once per commit-preparation attempt, filtering to the relevant DV index entries as they are read. - Keep temporary grouping structures local to maintainer initialization and reuse metadata references during bitmap processing, without introducing a persistent full-table cache. - Add a multi-partition regression for the scan count and verify that peak heap usage does not increase. ### Anything else? This is a core optimization shared by Flink and Spark. DV rewriting, index file formats, and commit semantics remain unchanged; A local synthetic benchmark against `57c0e3fd48` used 120 partitions, 2,400 DV index files, and 300,000 DV entries, with all partitions selected and a warm manifest cache. Each implementation ran in a separate JDK 11 JVM with Serial GC and a 128 MiB initial / 1,536 MiB maximum heap. | Metric | Before | After | | --- | ---: | ---: | | Metadata preparation time | 7.05 s | 1.67 s | | Main-thread allocated bytes | 9,776 MiB | 2,393 MiB | | Sampled peak JVM heap usage | 609 MiB | 329 MiB | These are single-run results for index scanning and maintainer initialization, excluding bitmap I/O and data compaction. Peak heap was sampled every 1 ms; the results do not represent end-to-end job speedup. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
