JingsongLi commented on code in PR #11163:
URL: https://github.com/apache/paimon/pull/11163#discussion_r4643615240


##########
paimon-python/pypaimon/common/native_compatibility.py:
##########
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Shared Native eligibility checks that must agree for readers and writers.
+
+Current Native/Rust runtimes do not implement
+``nested-sequence-overwrite-on-equal=false`` for ``nested_update``. When that
+option is configured, both Native read and Native write must yield to Python
+so equal-sequence retention stays consistent. Default / explicit ``true``
+keeps Native eligible.
+"""
+
+from typing import Optional
+
+from pypaimon.common.options.core_options import CoreOptions
+
+
+_FIELDS_PREFIX = CoreOptions.FIELDS_PREFIX + "."
+_AGG_SUFFIX = ".aggregate-function"
+_DEFAULT_AGG_KEY = _FIELDS_PREFIX + "default-aggregate-function"
+_OVERWRITE_SUFFIX = "." + CoreOptions.NESTED_SEQUENCE_OVERWRITE_ON_EQUAL
+_NESTED_UPDATE = "nested_update"
+# Matches aggregation_merge_function.resolve_agg_func_name when neither a
+# field-level nor table-wide aggregate-function is set.
+_DEFAULT_AGG_FALLBACK = "last_non_null_value"
+
+
+def _effective_aggregate_function(raw, field):
+    """Resolve the aggregator that will actually run for ``field``.
+
+    Same precedence as ``resolve_agg_func_name`` for non-PK / non-sequence
+    fields: field-level ``aggregate-function``, then
+    ``fields.default-aggregate-function``, then the factory default
+    ``last_non_null_value``.
+    """
+    return (
+        raw.get(_FIELDS_PREFIX + field + _AGG_SUFFIX)
+        or raw.get(_DEFAULT_AGG_KEY)
+        or _DEFAULT_AGG_FALLBACK
+    )
+
+
+def nested_update_overwrite_on_equal_false_reason(
+        options: Optional[CoreOptions]) -> Optional[str]:
+    """Return why Native must not run, or ``None`` when Native stays eligible.
+
+    Fields with an explicit ``nested-sequence-overwrite-on-equal=false`` force
+    Python when their effective aggregator is ``nested_update`` (field-level
+    ``aggregate-function``, otherwise ``fields.default-aggregate-function``).
+    Unset / ``true``, and false on non-``nested_update`` fields, leave Native
+    eligible.
+    """
+    if options is None or not isinstance(options, CoreOptions):
+        return None
+    raw = options.options.to_map()
+    for key in raw:
+        if not (key.startswith(_FIELDS_PREFIX) and 
key.endswith(_OVERWRITE_SUFFIX)):
+            continue
+        field = key[len(_FIELDS_PREFIX):-len(_OVERWRITE_SUFFIX)]
+        if 
options.field_nested_update_agg_nested_sequence_overwrite_on_equal(field):

Review Comment:
   **[P2] Validate the explicit option dependency before choosing the PyPaimon 
writer**
   
   For a real nested_update table with a nested key, explicit 
overwrite-on-equal=true and no nested-sequence-field, Native 0.5.0 writes and 
commits successfully. The same table shape with Python write selected raises 
the documented dependency error, as Java does. The true-policy fast path skips 
validation. Please apply the dependency check for explicitly configured true 
and false consistently before writer selection.



##########
paimon-python/pypaimon/execution/python/read/reader/aggregate/aggregators.py:
##########
@@ -229,16 +231,56 @@ def _compare_objects(left: Any, right: Any) -> int:
     if right is None:
         return 1
 
+    if isinstance(data_type, RowType):
+        return _compare_row_values(left, right, data_type)
+
     return (left > right) - (left < right)

Review Comment:
   **[P1] Match Java ordering throughout a ROW-valued sequence**
   
   I wrote and reopened real Parquet PK aggregation tables with a nested 
sequence `ROW<part ARRAY<BIGINT>>` or `ROW<part DOUBLE>`, and the same nested 
key in both updates. With overwrite-on-equal=false, Python `prepare_commit` 
raises TypeError for `[NULL] -> [1]`. For `-0.0 -> +0.0` or `1.0 -> NaN`, 
Python silently retains `first`, while Java with the same false setting commits 
and reads `second` in all three cases. Recursive ROW traversal still delegates 
member ordering to Python operators. Please use schema-aware ordering through 
the member types, including Java floating-point ordering and nulls-first within 
arrays.



##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldNestedUpdateAgg.java:
##########
@@ -196,6 +217,14 @@ public Object retract(Object accumulator, Object 
retractField) {
         }
     }
 
+    private boolean shouldUpdate(InternalRow newRow, InternalRow oldRow) {
+        if (!hasSequenceField) {
+            return true;
+        }
+        int comparison = compareSequence(newRow, oldRow);
+        return comparison > 0 || (comparison == 0 && overwriteOnEqualSequence);

Review Comment:
   **[P1] Keep the equal-sequence winner independent of commit boundaries**
   
   In a Java Parquet aggregation table with `sequence.field=ord` and nested 
first-wins enabled, I wrote updates A, C, B with outer ord values 1, 3, 2 and 
the same nested key and seq=7. One batch kept A. Writing A+C first commits a 
visible A with outer ord=3, then committing B alone replaces that visible value 
with B at the same nested sequence. The two results persist after full 
compaction. Explicit true returned C in both layouts. The outer row carries the 
higher ordinal but the retained nested payload originated earlier, so 
re-reading separate files orders the partial aggregate differently. Keep enough 
ordering information to preserve the policy across partial aggregation, or 
reject the incompatible sequence-field combination explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to