JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4521057496
##########
paimon-python/pypaimon/execution/python/write/commit/row_id_conflict.py:
##########
@@ -35,10 +35,9 @@ def check_row_id_conflicts(table, messages,
latest_snapshot_id, check_from_snaps
baseline = manager.get_snapshot_by_id(check_from_snapshot)
if latest is None or baseline is None:
raise ValueError('Row-ID conflict check snapshots must exist in the
current table')
- current_schema = table.schema_manager.latest()
- if (baseline.schema_id != latest.schema_id
- or current_schema is not None and current_schema.id !=
table.table_schema.id):
- raise ValueError('Row-ID conflict checks require an unchanged, current
table schema')
+ # Logical conflicts compare field IDs resolved in each file's historical
+ # schema, just like ordinary commits. Different schema versions alone do
Review Comment:
**[P1] Native REST snapshot retries still lose updates to recreated fields**
I reproduced this on `c36d679c` using a real REST catalog fixture and the
Rust committer. I staged `age = 99` for an existing row, then dropped and
re-added `age` and committed a competing append with `age = 12` as the first
native publication request arrived.
The append won the snapshot CAS. A single native commit call sent two
publication requests, first with schema 0 and then with schema 1. The second
request succeeded, but reading `age` returned `[NULL, 12]`.
The new per-attempt validation covers the Python committer. Rust
independently [loads the latest schema ID when building each
snapshot](https://github.com/apache/paimon-rust/blob/e8939e5b419b380fd5da0d7be1092594907cee5e/crates/paimon/src/table/table_commit.rs#L1657-L1665),
after the [Ray coordinator's field
check](https://github.com/apache/paimon/blob/c36d679c33c08c75828b2b9928ec7d9eea856580/paimon-python/pypaimon/ray/row_id_conflict_rewriter.py#L111-L118)
has already run. Please validate the staged field IDs against the schema used
by each native publication attempt, including after a failed REST CAS.
##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -547,6 +548,12 @@ def _try_commit_once(self, retry_result:
Optional[RetryResult], commit_kind: str
# Automatic rollback can delete snapshots and tags already
used by readers.
raise conflict_exception
+ # Capture existing-row updates before assigning row IDs to new appends.
+ update_files = [
+ entry.file for entry in commit_entries
+ if entry.kind == 0 and entry.file.first_row_id is not None
+ ] if commit_kind == 'APPEND' and
self.table.options.data_evolution_enabled() else []
Review Comment:
**[P1] Validate staged updates when a mixed update/delete commits as
OVERWRITE**
A Ray self-merge can stage an UPDATE and a DELETE on different rows. With
deletion vectors enabled, the DELETE makes the combined commit an `OVERWRITE`,
so this condition leaves `update_files` empty.
I reproduced this on `c36d679c` with real staged files and a real snapshot
retry. I staged `age = 99` for row ID 0 and deleted row ID 1. During the first
publication, I dropped and re-added `age` and committed a competing append. The
first `OVERWRITE` attempt under schema 0 failed the CAS. The retry under schema
1 succeeded. The delete took effect, but the remaining rows read back as IDs
`[1, 3, 4]` with ages `[NULL, NULL, 10]`.
Please select existing-row update files for validation even when deletion
vectors make the commit an `OVERWRITE`, and add a mixed update/delete case to
the inner CAS retry test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]