JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4519855743
##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -247,6 +234,17 @@ def _rewrite_updates(
if not _ranges_are_still_covered(current_files, candidates):
return None
+ # An append can advance the snapshot without changing update file ranges.
+ # Ordinary commits retain their own validation; only actual rewrites need
+ # this extra check before advancing the conflict baseline.
+ table = _with_latest_schema(table)
+ _validate_no_logical_conflict(
Review Comment:
**[P2] Handle compatible schema changes with native REST commits**
For a supported REST table with `commit.native.enabled=true`, this check
dispatches to Rust. If a nullable column is added after the update is staged
and compaction then publishes a snapshot with the new schema, the base and
latest snapshot schema IDs differ. The [Rust
guard](https://github.com/apache/paimon-rust/blob/0be46006820483370b7ab7a36547b5e29fc5e278/crates/paimon/src/table/table_commit.rs#L350-L356)
still rejects that pair even after the table schema is refreshed, so recovery
fails before rewriting. Please align the native conflict check with the Python
behavior and add coverage with native commit enabled. The new schema recovery
tests explicitly disable native commit.
##########
paimon-python/pypaimon/ray/update_by_row_id.py:
##########
@@ -155,25 +156,18 @@ def _project_cast(batch: pa.Table) -> pa.Table:
raise # _reraise_inner always raises; keeps msgs/num_updated defined
for linters
if msgs:
- _commit_update_messages(table, msgs)
+ _commit_update_messages(
+ table, msgs, num_partitions=num_partitions,
ray_remote_args=ray_remote_args)
return {"num_updated": num_updated}
-def _commit_update_messages(table, commit_messages) -> None:
- messages = list(commit_messages)
- table_commit = None
+def _commit_update_messages(
+ table, commit_messages, *, num_partitions=None, ray_remote_args=None)
-> None:
+ from pypaimon.ray.row_id_conflict_rewriter import
commit_updates_with_compaction_retry
+
try:
- table_commit = table.new_batch_write_builder().new_commit()
- table_commit.commit(messages)
+ commit_updates_with_compaction_retry(
Review Comment:
**[P2] Limit the recovery precheck to affected partitions**
This new caller enters the coordinator's [unfiltered
`plan_all()`](https://github.com/apache/paimon/blob/22d13beab624ebb07d9f220ec4fd553625f444b6/paimon-python/pypaimon/ray/row_id_conflict_rewriter.py#L192-L208)
whenever another commit advances the snapshot. An append to an unrelated
partition therefore triggers an extra full-table metadata plan on the driver
before the coordinator discovers there is nothing to rewrite. The existing
commit path prunes manifests by the changed partitions. With ongoing ingestion,
this extra plan can recur for each `process_row_id_ranges` batch. Please
restrict the precheck to partitions in the staged messages or defer planning
until a row-ID range conflict is reported.
##########
paimon-python/pypaimon/ray/update_by_row_id.py:
##########
@@ -155,25 +156,18 @@ def _project_cast(batch: pa.Table) -> pa.Table:
raise # _reraise_inner always raises; keeps msgs/num_updated defined
for linters
if msgs:
- _commit_update_messages(table, msgs)
+ _commit_update_messages(
Review Comment:
**[P2] Preserve the explicit recovery opt-out**
The [existing option
documentation](https://github.com/apache/paimon/blob/4b8e50df3178f15ccafe6c76cf2b543229b8f4e1/docs/docs/pypaimon/data-evolution.md#L395-L399)
says that setting `data-evolution.row-id-conflict-rewrite.max-size` to `0 B`
disables automatic rewriting. This entry point now invokes the Ray rewriter
even with that setting. I ran the range-batch compaction test with `0 B` and
confirmed that it still launches distributed recovery and commits. Please
preserve the explicit zero opt-out for `update_by_row_id` while letting Ray
recovery bypass positive local size limits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]