leaves12138 opened a new pull request, #8896: URL: https://github.com/apache/paimon/pull/8896
## What changed - add a dedicated low-memory conflict-check path for global index-only commits - scan projected `BinaryManifestEntry` fields directly instead of materializing all data files as `SimpleFileEntry` - prune manifest files with partition statistics and manifest row-id bounds - use reusable byte-array partition lookups, compact deleted-file identifiers, and primitive row-range storage - preserve existing data manifests for index-only commits instead of opportunistically sorting or merging them - retain the original conflict path for data-file changes, deletion vectors, commit pre-callbacks, and `rowIdCheckFromSnapshot` ## Why A global index-only commit previously read every live data entry in the affected partitions into Java objects and retained that base-file list through snapshot preparation. It also invoked data-manifest sorting even though no data file changed. On the large test table, conflict detection materialized 4,105,298 `SimpleFileEntry` instances and the subsequent manifest sort rewrote 32 data manifests. These phases overlapped in memory. The new path still validates that every global-index row-id range is covered by current live data files. It performs the check with projected binary manifest entries, applies DELETE entries by full file identity, and retains only primitive ranges intersecting the requested index ranges. ## Measured impact Snapshot 3496 metadata, Linux x86, `-Xmx24g`: | Metric | Pre-change (`bbcac771ff`) | This change, rebased on `903aa197a1` | | --- | ---: | ---: | | Commit elapsed time | 64.127 s | 3.102 s | | Peak Java heap | 16.384 GiB | 0.899 GiB | | Peak process RSS | 19.160 GiB | 1.492 GiB | The index commit produced the next snapshot successfully, and the tests assert that its data-manifest list is unchanged. ## Validation - `mvn -pl paimon-core -Dtest="ConflictDetectionTest,FileStoreCommitTest#testGlobalIndexCommitChecksExistingRowIds+testGlobalIndexCommitChecksAdjacentDataRanges+testGlobalIndexCommitFailsForMissingRowIds+testGlobalIndexCommitDoesNotUseDeletedDataFileRowIds+testGlobalIndexCommitChecksPartitionAndBucket" test` - Linux x86 package build on dev2 - the five global-index commit scenarios above on dev2 - snapshot 3496 benchmark on dev2 No full regression suite was run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
