JingsongLi opened a new pull request, #9177: URL: https://github.com/apache/paimon/pull/9177
## Summary - plan data-evolution compaction candidates from projected manifest metadata instead of retaining full file entries for the whole scan - select exact row-id ranges from first row ID, row count, file size, and file type, then read all files for each selected batch in one scan - move the 100,000-file batching policy into `DataEvolutionCompactRangePlanner` - keep the legacy rewrite-row-ids option for compatibility, but reject `true` and require deletion-vector materialization to run separately ## Why `DataEvolutionCompactCoordinator` previously loaded full `ManifestEntry` and `DataFileMeta` objects for every live file before deciding which files needed compaction. Large data-evolution tables could therefore consume several gigabytes of heap and fail with OOM even when only a small subset of files were compact candidates. The new two-phase planning flow first scans only the projected fields needed to identify exact candidate row-id ranges. Full file metadata is then loaded only for selected ranges, in bounded batches. Candidate metadata uses primitive arrays with 32 bytes of payload per live file, substantially reducing the planning footprint. ## Impact - reduces coordinator memory usage for tables with large file counts - preserves compaction decisions for normal, blob, and vector files - keeps compatibility for manifests without row-id bounds - preserves stable row IDs during ordinary data-evolution compaction - makes `data-evolution.compaction.rewrite-row-ids=true` fail fast with guidance to use a separate operation ## Validation - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=DataEvolutionCompactCoordinatorTest,CompactCandidateRangeCollectorTest,DataEvolutionCompactRangePlannerTest,DataEvolutionDeletionVectorTest,FullTextSearchBuilderTest,VectorSearchBuilderTest test` (116 tests) - `mvn -pl paimon-flink/paimon-flink-common,paimon-spark/paimon-spark-common -am -Pfast-build -Pflink1 -Pspark3 -DskipTests -DfailIfNoTests=false compile` - `mvn -pl paimon-api,paimon-core,paimon-flink/paimon-flink-common,paimon-spark/paimon-spark-common -Pflink1 -Pspark3 -DskipTests spotless:check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
