JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4520854281
##########
paimon-python/pypaimon/execution/python/write/commit/row_id_conflict.py:
##########
@@ -35,10 +35,9 @@ def check_row_id_conflicts(table, messages,
latest_snapshot_id, check_from_snaps
baseline = manager.get_snapshot_by_id(check_from_snapshot)
if latest is None or baseline is None:
raise ValueError('Row-ID conflict check snapshots must exist in the
current table')
- current_schema = table.schema_manager.latest()
- if (baseline.schema_id != latest.schema_id
- or current_schema is not None and current_schema.id !=
table.table_schema.id):
- raise ValueError('Row-ID conflict checks require an unchanged, current
table schema')
+ # Logical conflicts compare field IDs resolved in each file's historical
+ # schema, just like ordinary commits. Different schema versions alone do
Review Comment:
**[P1] The inner snapshot retry can still publish an update to a dropped
field**
I reproduced a remaining case on `57a908dc` with real filesystem commits. I
staged `age = 99` for the first row in a two-row table. After the Ray field-ID
check passed, I paused the first snapshot publication, dropped and recreated
`age` under the same name, and let another writer append a third row with `age
= 10`.
The append published snapshot 2 under schema 1. The original update's
attempt to publish snapshot 2 under schema 0 then lost the snapshot CAS.
`FileStoreCommit` retried internally and published snapshot 3 under schema 1.
The update returned successfully, but reading `age` yielded `[NULL, NULL, 10]`.
The new check runs once before `commit.commit()`. The [inner retry
loop](https://github.com/apache/paimon/blob/57a908dc399f52ef4307995809a1f54eb4c2b374/paimon-python/pypaimon/execution/python/write/file_store_commit.py#L355-L395)
can take a new [snapshot
schema](https://github.com/apache/paimon/blob/57a908dc399f52ef4307995809a1f54eb4c2b374/paimon-python/pypaimon/execution/python/write/file_store_commit.py#L614-L619)
without revalidating staged field IDs. Please validate the staged IDs against
the schema used to build the snapshot on every publication attempt, including
retries after a failed CAS. The regression test should force that inner retry.
##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -357,17 +403,26 @@ def _rewrite_group(
num_partitions=num_partitions,
ray_remote_args=ray_remote_args,
base_snapshot_id=snapshot_id,
+ estimated_size_bytes=estimated_size_bytes,
+ estimated_num_rows=estimated_num_rows,
)
def _validate_no_logical_conflict(
table,
update_messages: List[CommitMessage],
latest_snapshot,
- base_snapshot_id: int,
+ base_snapshot,
) -> None:
"""Reject non-compaction changes before advancing the rewrite baseline."""
- check_row_id_conflicts(table, update_messages, latest_snapshot.id,
base_snapshot_id)
+ # The native checker requires identical snapshot schema IDs. Choose the
+ # Python read-only check for this unsupported case before dispatch, while
+ # leaving the actual commit backend unchanged. Native check failures must
+ # still propagate without retrying the check through another backend.
+ if (table.options.native_commit_enabled()
+ and base_snapshot.schema_id != latest_snapshot.schema_id):
+ table = table.copy({CoreOptions.COMMIT_NATIVE_ENABLED.key(): 'false'})
Review Comment:
**[P2] Resolve nested write paths in the cross-schema conflict check**
When this branch selects the Python checker, a Java partial update in the
intervening snapshot history can carry a dotted write column such as `nest.a`.
I reproduced the failure in the checker with a staged Ray update to `age` and a
concurrent write to `nest.a` over the same row-ID range. The checker raises
`Column 'nest.a' not found in schema 1`, so recovery aborts even though the two
writes touch different fields.
Please resolve dotted write paths to field IDs in each file's historical
schema and cover a Java partial write in the recovery history. The [Java
test](https://github.com/apache/paimon/blob/57a908dc399f52ef4307995809a1f54eb4c2b374/paimon-core/src/test/java/org/apache/paimon/table/NestedSubfieldDataEvolutionTableTest.java#L128-L134)
shows how such an update is written. The [Python
lookup](https://github.com/apache/paimon/blob/57a908dc399f52ef4307995809a1f54eb4c2b374/paimon-python/pypaimon/execution/python/write/commit/conflict_detection.py#L111-L134)
currently indexes only top-level names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]