JingsongLi commented on code in PR #11163: URL: https://github.com/apache/paimon/pull/11163#discussion_r4583532847
########## paimon-python/pypaimon/common/native_compatibility.py: ########## @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Shared Native eligibility checks that must agree for readers and writers. + +Current Native/Rust runtimes do not implement +``nested-sequence-overwrite-on-equal=false`` for ``nested_update``. When that +option is configured, both Native read and Native write must yield to Python +so equal-sequence retention stays consistent. Default / explicit ``true`` +keeps Native eligible. +""" + +from typing import Optional + +from pypaimon.common.options.core_options import CoreOptions +from pypaimon.common.options.options_utils import OptionsUtils + + +_FIELDS_PREFIX = CoreOptions.FIELDS_PREFIX + "." +_AGG_SUFFIX = ".aggregate-function" +_OVERWRITE_SUFFIX = "." + CoreOptions.NESTED_SEQUENCE_OVERWRITE_ON_EQUAL +_NESTED_UPDATE = "nested_update" + + +def nested_update_overwrite_on_equal_false_reason( + options: Optional[CoreOptions]) -> Optional[str]: + """Return why Native must not run, or ``None`` when Native stays eligible. + + Only ``aggregate-function=nested_update`` fields with an explicit + ``nested-sequence-overwrite-on-equal=false`` force Python. Unset / ``true`` + leave Native eligible. + """ + if options is None or not isinstance(options, CoreOptions): + return None + raw = options.options.to_map() + for key, value in raw.items(): + if not (key.startswith(_FIELDS_PREFIX) and key.endswith(_AGG_SUFFIX)): + continue + if value != _NESTED_UPDATE: + continue + field = key[len(_FIELDS_PREFIX):-len(_AGG_SUFFIX)] Review Comment: [P1] Resolve inherited default aggregators before allowing Native This only examines explicit fields.<name>.aggregate-function keys, so it misses a valid table with fields.default-aggregate-function=nested_update and fields.updates.nested-key/sequence-field/nested-sequence-overwrite-on-equal=false. Both Java and Python resolve the updates field to nested_update while exempting the scalar primary key, but this helper returns None. On this exact head with Rust0.5.0, a real NativeBatchTableWrite containing two versions of one nested key at seq=7 persists only 'second'; reopening with the Python reader still returns 'second', although the false policy requires retaining 'first'. The explicit-field and Python-writer controls retain 'first'. On two Python-written files, Python reads 'first' while Native reads 'second' with Python fallback forbidden. Thus the earlier Native data-loss issue is still reachable through the default configuration. Resolve each field's effective aggregation function (including the default and field overrides), or conservativ ely gate the configured false-policy fields, and add this real-table read/write case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
