JingsongLi commented on code in PR #9357:
URL: https://github.com/apache/paimon/pull/9357#discussion_r3838026956
##########
paimon-python/pypaimon/ray/data_evolution_merge_into.py:
##########
@@ -84,14 +89,39 @@ def merge_into(
read_columns: Optional[Sequence[str]] = None,
) -> Dict[str, int]:
_require_ray_join()
- num_partitions = _resolve_num_partitions(num_partitions)
table, source_ds, matched_specs, not_matched_specs, ctx = _prepare(
target, source, catalog_options,
list(when_matched), list(when_not_matched), on,
read_columns,
)
base_snapshot = table.snapshot_manager().get_latest_snapshot()
+ target_empty = _is_target_empty(base_snapshot)
+ estimated_size_bytes = None
+ if num_partitions is None:
+ estimated_size_bytes = _estimate_merge_input_size_bytes(
+ source_ds, ctx,
+ )
+ min_partitions = 1
+ unknown_num_partitions = None
+ data_context = (
+ None
+ if ctx.is_self_merge
+ else getattr(source_ds, "context", None)
Review Comment:
**[P2] Resolve matched MERGE partitions from the target-left context**
This always derives the adaptive default from `source_ds.context`, but the
matched update/delete path executes the join as
`target_renamed.join(source_renamed, ...)`. Ray seals the context on each
Dataset and `Dataset.join` inherits the left operand context, so if the source
was created under context A and the target under context B, Paimon selects the
partition count using A while the actual matched join runs with B. This can
significantly over- or under-partition the shuffle. Please resolve the internal
partition count per branch from that branch’s actual left operand context while
continuing to honor an explicit `num_partitions`, and add a matched-path test
where the source and target contexts differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
