leaves12138 commented on code in PR #10418: URL: https://github.com/apache/paimon/pull/10418#discussion_r4212550025
########## paimon-python/pypaimon/write/native_merge_into.py: ########## @@ -0,0 +1,97 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Transport existing MERGE inputs into the core operation.""" + +import pyarrow as pa + +from pypaimon.snapshot.snapshot import BATCH_COMMIT_IDENTIFIER +from pypaimon.write.native_commit import from_native_commit_messages +from pypaimon.write.native_update import ( + _native_row_id_table, _native_update_columns_supported, _supported_upsert_key_type, +) + + +def create_native_merge_into(table, source, on, matched, not_matched, commit_user, commit_identifier): + from pypaimon.table.data_evolution_merge_into import _is_self_merge, _is_table_like, _prepare + from pypaimon.ray.data_evolution_merge_into import _normalize_on, _union_update_cols + from pypaimon.ray.data_evolution_merge_transform import SourceColumnRef, TargetColumnRef, LiteralValue + from pypaimon.ray.merge_condition import rewrite_condition + + target_keys, source_keys = _normalize_on(on) + self_merge = _is_self_merge(table, source, target_keys, source_keys) + if not_matched and table.options.video_frame_fields(): + return None + if _is_table_like(source) and not self_merge: + return None + native_table = _native_row_id_table(table) + if native_table is None: + return None + source, matched, not_matched, context = _prepare(table, source, list(matched), list(not_matched), on) Review Comment: Non-blocking dependency-contract clarification Keeping the standalone Python DataFusion dependency for conditional MERGE is an acceptable design choice. The existing pypaimon[sql] extra already includes pypaimon[datafusion] and pypaimon-rust, and the current missing-dependency error explains how to install DataFusion. Requiring this documented extra is not itself a correctness bug, and importing the Python package during validation does not mean that native conditions execute in Python. I withdraw the P2/blocking request to decouple this in the current PR. The smallest recommendation is to document that conditional native MERGE keeps this dependency requirement and use the existing SQL extra rather than adding DataFusion to all core installations. The paired Rust runtime must of course include the new MERGE API. If a native-only installation without the standalone Python DataFusion package becomes a requirement later, move the dependency check out of shared clause normalization and into the Python execution path. That is an optional dependency-footprint improvement, not required for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
