JingsongLi opened a new pull request, #8909: URL: https://github.com/apache/paimon/pull/8909
## What changed - add projected binary manifest scans and compact, collision-safe deleted-entry tracking - retain conflict-detection base files as compact binary-backed `SimpleFileEntry` instances - merge retry changes with maps sized by changed identities instead of all base files - avoid full-base identifier sets when discarding duplicate commit entries - share compact entry data with deletion-vector wrappers - bound conflict exception formatting to the first 50 entries ## Why Commit conflict detection retains all files from changed partitions. Materializing every manifest entry into POJOs, then building additional full-base lists, identifier maps, and exception strings can create a large heap amplification and lead to OOM for tables with many files. The compact path keeps the projected manifest row in independently owned binary memory and materializes strings, lists, and key rows only when accessed. The merge path now grows its identity maps with the delta instead of the base snapshot. In a JOL object-graph comparison using 10,000 equivalent entries, compact binary-backed base entries retained about 4.04 MB versus an estimated 6.44 MB for the previous representation, a reduction of roughly 37%. This excludes the additional savings from removing base-sized identity maps and list copies. ## Validation - `mvn -pl paimon-core -DskipTests compile` - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=BinaryManifestEntryTest,FileEntryTest,ManifestFileTest,ConflictDetectionTest,FileStoreCommitTest test` - 121 related tests passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
