JingsongLi commented on code in PR #1406:
URL: https://github.com/apache/paimon-rust/pull/1406#discussion_r4656566794


##########
crates/paimon/src/table/aggregator/nested.rs:
##########
@@ -293,6 +316,18 @@ impl NestedAgg {
         Ok(Ordering::Equal)
     }
 
+    /// Whether `new` may replace `old` for `nested_update` (Java 
`shouldUpdate`).
+    fn should_update(&self, new: &StructArray, old: &StructArray) -> 
crate::Result<bool> {
+        if self.sequence_indices.is_empty() {
+            return Ok(true);
+        }
+        match self.compare_sequence(new, old)? {
+            Ordering::Greater => Ok(true),
+            Ordering::Equal => Ok(self.overwrite_on_equal_sequence),

Review Comment:
   **[P1] Preserve equal-sequence winners across concurrent writer flushes**
   
   This is reachable with fixed bucket 1, `merge-engine=aggregation`, no global 
`sequence.field`, and overwrite-on-equal=false. I prepared two writers against 
the same empty snapshot. A wrote the target nested key with amount 10, a 
different PK, then the same target and nested sequence with amount 30. B wrote 
a different PK followed by the same target and nested sequence with amount 20. 
In both runs I committed A, then B. Flushing all of A together returned 20. 
Preparing A's first row in a separate file and publishing both of A's files in 
the same commit returned 10.
   
   The locally reduced row carries A's last KV sequence, even when its nested 
value came from A's first KV. At the next merge, the strict tie rule uses that 
changed order. Preserve the winning nested record's ordering information across 
file boundaries, or narrow the supported configuration until it can produce 
deterministic results.



##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -298,11 +310,12 @@ fn parse_field_scoped_option_key(key: &str) -> 
Option<(&str, FieldScopedOptionKi
 /// `SchemaManager.applyRenameColumnsToOptions` (Java's case-2 list; the
 /// aggregation/partial-update engines do not accept every entry at create
 /// time, but the keys can still arrive in Java-written metadata).
-const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 7] = [
+const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 8] = [
     AGG_FUNCTION_SUFFIX,
     IGNORE_RETRACT_SUFFIX,
     DISTINCT_SUFFIX,
     LIST_AGG_DELIMITER_SUFFIX,
+    NESTED_SEQUENCE_OVERWRITE_ON_EQUAL_SUFFIX,

Review Comment:
   **[P1] Keep nested partial updates readable after a column rename**
   
   I wrote two versions to an `items` column using `nested_partial_update`, 
with a configured nested key, nested sequence field, and the new flag set to 
false. `ALTER RENAME items TO lines` succeeds. The base still reads the renamed 
column, but this revision fails with `requires ... nested-sequence-field`. The 
rename moves the flag and nested key while leaving the nested sequence option 
under the old field name. The constructor then validates the flag even in 
`nested_partial_update`, where it has no effect. Keep related option names 
synchronized and restrict the new runtime checks to `nested_update`.



##########
docs/src/sql.md:
##########
@@ -4014,9 +4014,27 @@ Field-scoped options used by these functions:
 - `fields.<field>.ts-field` selects the key-time field for
   `merge_map_with_keytime` (defaults to the last ROW field).
 - `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
+  `fields.<field>.nested-sequence-overwrite-on-equal` (default `true`;
+  requires `nested-sequence-field`; applies only to `nested_update`),
   `fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
   for nested array aggregators. `nested_partial_update` requires
-  `nested-key`.
+  `nested-key`. When `nested-sequence-field` is set for `nested_update`, a
+  nested ROW overwrites an equal-key accumulator if its sequence is greater,
+  or equal when `nested-sequence-overwrite-on-equal` is `true` (default).
+  Set it to `false` to overwrite only when the new sequence is strictly
+  greater. That strict equal-sequence policy is currently enforced by Rust
+  writer/compaction paths that implement the option; a Java runtime that has
+  not implemented it may still overwrite the nested ROW when sequences are
+  equal. Tables that rely on `false` to keep equal sequences from overwriting
+  should not mix writers or compaction from a Java runtime that does not yet

Review Comment:
   **[P2] Include Java readers in the interoperability restriction**
   
   I read the same uncompacted snapshot with Rust and the Java Paimon 2.2 
snapshot build `20261118.220748-71`. Two Rust commits used the same nested key 
and sequence, with amounts 10 and 20. Rust read 10. Java read 20 without 
performing any writes or compaction. Please document that Java snapshot reads 
can also see different winners while equal-sequence versions coexist. Changing 
the option after compaction cannot restore values that have already been 
discarded.



##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -272,6 +280,10 @@ fn parse_field_scoped_option_key(key: &str) -> 
Option<(&str, FieldScopedOptionKi
             NESTED_SEQUENCE_FIELD_SUFFIX,
             FieldScopedOptionKind::NestedSequenceField,
         ),
+        (
+            NESTED_SEQUENCE_OVERWRITE_ON_EQUAL_SUFFIX,
+            FieldScopedOptionKind::NestedSequenceOverwriteOnEqual,

Review Comment:
   **[P1] Reject invalid tie-policy changes before publishing the schema**
   
   On a populated `nested_update` table with a nested key and no nested 
sequence field, a public filesystem `ALTER SET 
fields.items.nested-sequence-overwrite-on-equal=false` succeeded and persisted 
schema 1. Its previously readable rows then failed with `requires ... 
nested-sequence-field`. This option kind currently checks the owner column but 
leaves its value and dependency checks to runtime. Validate the boolean and its 
prerequisite during both CREATE and ALTER for fields that use the option.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to