JustinBinber commented on code in PR #11174:
URL: https://github.com/apache/paimon/pull/11174#discussion_r4600977861
##########
paimon-core/src/main/java/org/apache/paimon/operation/commit/DataEvolutionConflictDetection.java:
##########
@@ -494,7 +508,10 @@ private Optional<RuntimeException>
checkForRowIdFromSnapshot(
DataFileMeta file = entry.file();
if (file.firstRowId() != null
&& file.nonNullRowIdRange().from < checkNextRowId
- && conflictChecker.conflictsWith(file)) {
+ && ((conflictChecker != null &&
conflictChecker.conflictsWith(file))
+ || deletedRowRanges.intersects(
Review Comment:
Confirmed, including the update-before-reassignment ordering. I reproduced
both silent-loss cases on `29ecc748d8`; the identical no-reassignment control
correctly rejects the stale commit.
Fixed in
[`05237fa1dc`](https://github.com/apache/paimon/pull/11174/commits/05237fa1dc60e093d61af5ee128fd16a6ede98b6).
Before checking historical updates, the DML conflict check now compares each
actually changed DV anchor's physical identity (partition, bucket, file name)
and RowID range against the preparation snapshot. A missing anchor or changed
mapping rejects the commit with the existing conflict error, requiring
replanning. This catches metadata-only reassignment even though it publishes an
empty data-file delta, and avoids mixing old history with newly assigned
ranges. I did not remove the `nextRowId` cutoff.
The existing content comparison still excludes unchanged vectors moved
during index repacking. The new baseline scan is skipped for uncontended
commits and for commits without changed DV anchors; the materialize-DV strategy
remains unchanged. This scan may still traverse the relevant manifests, so I am
not claiming its cost is strictly proportional to the number of anchors.
Added regressions for both `prepare -> reassign -> UPDATE -> stale commit`
and `prepare -> UPDATE -> reassign -> stale commit`, plus a positive control
that allows reassignment and an update in an unrelated partition. Both negative
cases preserve the winning `id=8` row and the published snapshot. The existing
shared-DV relocation/disjoint-update case also passes.
Local validation: all 16 actual procedure/conflict ScalaTest cases pass on
JDK8 / Spark3.5.9, along with the separate real-operation probes and scoped
Spotless check. The first full-suite attempt reached the explicit 192 MiB test
Metaspace cap; the complete validation passes at 384 MiB, with the failed
attempt retained. These are direct ScalaTest/runtime results, not a normal
local Maven reactor claim. [Automatic upstream CI for this
commit](https://github.com/apache/paimon/actions/runs/44103650874) is queued; I
am following it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]