leaves12138 opened a new pull request, #10150: URL: https://github.com/apache/paimon/pull/10150
### Purpose Reuse completed data-evolution compaction outputs when a concurrent row-ID reassignment changes only their metadata. This covers row-ID-preserving data/BLOB compaction and global-index compaction without rewriting their physical output files. The change is confined to commit preparation; it does not change compaction tasks, task serializers, or persisted data/index formats. ### Implementation - For eligible `COMPACT` commits with `commit.last-safe-snapshot`, inspect subsequent snapshots in order, similarly to `StrictModeChecker`. - Recognize snapshot-local reassign markers, validate and read their persisted plans, and rebase both ADD and DELETE entries. Update data-file `firstRowId` and global-index RowID ranges while preserving physical file names and contents. - Accept only complete, contiguous, length-preserving range mappings. Leave unaffected ranges unchanged and reject partial/non-contiguous mappings. - Carry the rebased changes forward across optimistic-commit retries without mutating the original worker results. - Resolve an uncertain successful commit before validating source files that the successful commit already removed. - Allow strict-mode checking to skip only the reassign snapshots whose plans have been validated and applied. Keep ordinary conflict checks and verify current index inputs. - Preserve the existing paths for materialized deletions, physical `_ROW_ID` columns, and other unsupported increments. Missing required history or missing/corrupt plans fail closed; the boundary snapshot itself need not still exist. ### Tests Executed in an isolated Linux environment: - 295 targeted tests passed, including `CompactReassignReuseTest`, `DataEvolutionRowIdReassignerTest`, `DataEvolutionNormalCompactTaskTest`, `DataEvolutionCompactCoordinatorTest`, `ConflictDetectionTest`, `FileStoreCommitTest`, and `DataEvolutionDeletionVectorTest`. - New checked-in tests cover data/BLOB and BTree metadata rebasing, repeated reassignments, reassignments during commit retries, lost successful commit responses, ordinary overwrite/input-replacement conflicts, missing/corrupt plans, legacy option fallback, expired boundary snapshots, and unsafe range rejection. - Additional isolated diagnostics checked 240,000 ranges against a per-row oracle, a BTree replacement across two reassignments, and tasks planned after an earlier reassign but using an older last-safe boundary. - Checkstyle/Spotless validation passed. Kept as draft while larger retained-metadata validation is in progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
