JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4519855743


##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -247,6 +234,17 @@ def _rewrite_updates(
     if not _ranges_are_still_covered(current_files, candidates):
         return None
 
+    # An append can advance the snapshot without changing update file ranges.
+    # Ordinary commits retain their own validation; only actual rewrites need
+    # this extra check before advancing the conflict baseline.
+    table = _with_latest_schema(table)
+    _validate_no_logical_conflict(

Review Comment:
   **[P2] Handle compatible schema changes with native REST commits**
   
   For a supported REST table with `commit.native.enabled=true`, this check 
dispatches to Rust. If a nullable column is added after the update is staged 
and compaction then publishes a snapshot with the new schema, the base and 
latest snapshot schema IDs differ. The [Rust 
guard](https://github.com/apache/paimon-rust/blob/0be46006820483370b7ab7a36547b5e29fc5e278/crates/paimon/src/table/table_commit.rs#L350-L356)
 still rejects that pair even after the table schema is refreshed, so recovery 
fails before rewriting. Please align the native conflict check with the Python 
behavior and add coverage with native commit enabled. The new schema recovery 
tests explicitly disable native commit.



##########
paimon-python/pypaimon/ray/update_by_row_id.py:
##########
@@ -155,25 +156,18 @@ def _project_cast(batch: pa.Table) -> pa.Table:
         raise  # _reraise_inner always raises; keeps msgs/num_updated defined 
for linters
 
     if msgs:
-        _commit_update_messages(table, msgs)
+        _commit_update_messages(
+            table, msgs, num_partitions=num_partitions, 
ray_remote_args=ray_remote_args)
     return {"num_updated": num_updated}
 
 
-def _commit_update_messages(table, commit_messages) -> None:
-    messages = list(commit_messages)
-    table_commit = None
+def _commit_update_messages(
+        table, commit_messages, *, num_partitions=None, ray_remote_args=None) 
-> None:
+    from pypaimon.ray.row_id_conflict_rewriter import 
commit_updates_with_compaction_retry
+
     try:
-        table_commit = table.new_batch_write_builder().new_commit()
-        table_commit.commit(messages)
+        commit_updates_with_compaction_retry(

Review Comment:
   **[P2] Limit the recovery precheck to affected partitions**
   
   This new caller enters the coordinator's [unfiltered 
`plan_all()`](https://github.com/apache/paimon/blob/22d13beab624ebb07d9f220ec4fd553625f444b6/paimon-python/pypaimon/ray/row_id_conflict_rewriter.py#L192-L208)
 whenever another commit advances the snapshot. An append to an unrelated 
partition therefore triggers an extra full-table metadata plan on the driver 
before the coordinator discovers there is nothing to rewrite. The existing 
commit path prunes manifests by the changed partitions. With ongoing ingestion, 
this extra plan can recur for each `process_row_id_ranges` batch. Please 
restrict the precheck to partitions in the staged messages or defer planning 
until a row-ID range conflict is reported.



##########
paimon-python/pypaimon/ray/update_by_row_id.py:
##########
@@ -155,25 +156,18 @@ def _project_cast(batch: pa.Table) -> pa.Table:
         raise  # _reraise_inner always raises; keeps msgs/num_updated defined 
for linters
 
     if msgs:
-        _commit_update_messages(table, msgs)
+        _commit_update_messages(

Review Comment:
   **[P2] Preserve the explicit recovery opt-out**
   
   The [existing option 
documentation](https://github.com/apache/paimon/blob/4b8e50df3178f15ccafe6c76cf2b543229b8f4e1/docs/docs/pypaimon/data-evolution.md#L395-L399)
 says that setting `data-evolution.row-id-conflict-rewrite.max-size` to `0 B` 
disables automatic rewriting. This entry point now invokes the Ray rewriter 
even with that setting. I ran the range-batch compaction test with `0 B` and 
confirmed that it still launches distributed recovery and commits. Please 
preserve the explicit zero opt-out for `update_by_row_id` while letting Ray 
recovery bypass positive local size limits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to