JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4523203898


##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -357,17 +399,26 @@ def _rewrite_group(
         num_partitions=num_partitions,
         ray_remote_args=ray_remote_args,
         base_snapshot_id=snapshot_id,
+        estimated_size_bytes=estimated_size_bytes,

Review Comment:
   **[P2] Keep the rewrite's routing plan scoped to affected partitions**
   
   On a three-partition FileIO table I staged an update only in `a` and 
compacted `a`. The new precheck planned only `a`, but this call then built 
another `plan_all()` over `a`, `b`, and `c` via `distributed_update_apply` and 
put the entire routing plan into Ray. The base's local rewrite used the 
affected partition's commit entries. A localized update on a large partitioned 
table now incurs whole-table planning and serialization on each rebase 
group/retry. Could this call reuse a routing plan restricted to the candidate 
partitions and complete file groups?



##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -86,20 +92,29 @@ def commit_self_merge_with_compaction_retry(
         if result is not None:
             current_updates = result.update_messages
             logger.info(
-                "Rewrote %d stale self-merge file(s) against snapshot %d "
+                "Rewrote %d stale row-id update file(s) against snapshot %d "

Review Comment:
   **[P2] Check the commit timeout after the eager rebase**
   
   With `commit.timeout=100ms` I staged the same FileIO update and compaction 
on the base and this head, and added a 250ms delay to each real rewriter. The 
base reported a timeout before publishing the update. This head completed the 
Ray rebase and published successfully after 2.39s. The elapsed checks only run 
after a failed commit, so the successful eager path skips the timeout entirely. 
Could we apply the same deadline after this pre-commit rebase before opening a 
publisher?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to