steFaiz opened a new pull request, #8773:
URL: https://github.com/apache/paimon/pull/8773
### Purpose
The main purpose is to ensure data evolution DV works with row-id
reassignment.
I added some tests and find an another issue: Concurrent row-id reassignment
and DataEvolution Compaction:
A compaction may be planned using an old RowID range. If RowID
reassignment commits before that compaction,
committing the stale compaction could restore the old RowIDs. For example,
reassignment moves [0, 9] to [10,
19], but a stale compaction may move them back to [0, 9].
So I added row-id existence check for compact commit:
1. Collecting existing data ranges with `MERGE Adjacent`(while non-compact
commit will not merge)
2. Each generated compaction files should be contained by old merge row
ranges (by part and bucket)
### Tests
1. ConflictDetectionTest
2. DataEvolutionDeletionTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
