JingsongLi commented on code in PR #11048:
URL: https://github.com/apache/paimon/pull/11048#discussion_r4523203898
##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -357,17 +399,26 @@ def _rewrite_group(
num_partitions=num_partitions,
ray_remote_args=ray_remote_args,
base_snapshot_id=snapshot_id,
+ estimated_size_bytes=estimated_size_bytes,
Review Comment:
**[P2] Keep the rewrite's routing plan scoped to affected partitions**
On a three-partition FileIO table I staged an update only in `a` and
compacted `a`. The new precheck planned only `a`, but this call then built
another `plan_all()` over `a`, `b`, and `c` via `distributed_update_apply` and
put the entire routing plan into Ray. The base's local rewrite used the
affected partition's commit entries. A localized update on a large partitioned
table now incurs whole-table planning and serialization on each rebase
group/retry. Could this call reuse a routing plan restricted to the candidate
partitions and complete file groups?
##########
paimon-python/pypaimon/ray/row_id_conflict_rewriter.py:
##########
@@ -86,20 +92,29 @@ def commit_self_merge_with_compaction_retry(
if result is not None:
current_updates = result.update_messages
logger.info(
- "Rewrote %d stale self-merge file(s) against snapshot %d "
+ "Rewrote %d stale row-id update file(s) against snapshot %d "
Review Comment:
**[P2] Check the commit timeout after the eager rebase**
With `commit.timeout=100ms` I staged the same FileIO update and compaction
on the base and this head, and added a 250ms delay to each real rewriter. The
base reported a timeout before publishing the update. This head completed the
Ray rebase and published successfully after 2.39s. The elapsed checks only run
after a failed commit, so the successful eager path skips the timeout entirely.
Could we apply the same deadline after this pre-commit rebase before opening a
publisher?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]