leaves12138 commented on code in PR #10418: URL: https://github.com/apache/paimon/pull/10418#discussion_r4212550025
########## paimon-python/pypaimon/write/native_merge_into.py: ########## @@ -0,0 +1,97 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Transport existing MERGE inputs into the core operation.""" + +import pyarrow as pa + +from pypaimon.snapshot.snapshot import BATCH_COMMIT_IDENTIFIER +from pypaimon.write.native_commit import from_native_commit_messages +from pypaimon.write.native_update import ( + _native_row_id_table, _native_update_columns_supported, _supported_upsert_key_type, +) + + +def create_native_merge_into(table, source, on, matched, not_matched, commit_user, commit_identifier): + from pypaimon.table.data_evolution_merge_into import _is_self_merge, _is_table_like, _prepare + from pypaimon.ray.data_evolution_merge_into import _normalize_on, _union_update_cols + from pypaimon.ray.data_evolution_merge_transform import SourceColumnRef, TargetColumnRef, LiteralValue + from pypaimon.ray.merge_condition import rewrite_condition + + target_keys, source_keys = _normalize_on(on) + self_merge = _is_self_merge(table, source, target_keys, source_keys) + if not_matched and table.options.video_frame_fields(): + return None + if _is_table_like(source) and not self_merge: + return None + native_table = _native_row_id_table(table) + if native_table is None: + return None + source, matched, not_matched, context = _prepare(table, source, list(matched), list(not_matched), on) Review Comment: [P2] Do not require Python DataFusion to prepare a native condition This shared _prepare() call invokes pypaimon.ray.merge_condition._load_datafusion() whenever any clause has a condition. Therefore a native-capable installation with pypaimon and pypaimon-rust, but without the optional standalone datafusion package, still raises ImportError for even condition="s.id > 0", before the Rust condition compiler is called. The unconditional case reaches NativeTableMergeInto successfully. I isolated this by making native-table eligibility succeed and blocking only the standalone datafusion import; skipping the unused _load_datafusion check lets the same native clause normalize successfully. Neither the core PyPaimon requirements nor the Rust wheel require that optional Python package. Please separate clause normalization/validation from the Python execution dependency check, keeping the latter on the Python fallback only, and test native conditional MERGE with the standalone import unavailable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
