JingsongLi commented on code in PR #1406:
URL: https://github.com/apache/paimon-rust/pull/1406#discussion_r4656566794
##########
crates/paimon/src/table/aggregator/nested.rs:
##########
@@ -293,6 +316,18 @@ impl NestedAgg {
Ok(Ordering::Equal)
}
+ /// Whether `new` may replace `old` for `nested_update` (Java
`shouldUpdate`).
+ fn should_update(&self, new: &StructArray, old: &StructArray) ->
crate::Result<bool> {
+ if self.sequence_indices.is_empty() {
+ return Ok(true);
+ }
+ match self.compare_sequence(new, old)? {
+ Ordering::Greater => Ok(true),
+ Ordering::Equal => Ok(self.overwrite_on_equal_sequence),
Review Comment:
**[P1] Preserve equal-sequence winners across concurrent writer flushes**
This is reachable with fixed bucket 1, `merge-engine=aggregation`, no global
`sequence.field`, and overwrite-on-equal=false. I prepared two writers against
the same empty snapshot. A wrote the target nested key with amount 10, a
different PK, then the same target and nested sequence with amount 30. B wrote
a different PK followed by the same target and nested sequence with amount 20.
In both runs I committed A, then B. Flushing all of A together returned 20.
Preparing A's first row in a separate file and publishing both of A's files in
the same commit returned 10.
The locally reduced row carries A's last KV sequence, even when its nested
value came from A's first KV. At the next merge, the strict tie rule uses that
changed order. Preserve the winning nested record's ordering information across
file boundaries, or narrow the supported configuration until it can produce
deterministic results.
##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -298,11 +310,12 @@ fn parse_field_scoped_option_key(key: &str) ->
Option<(&str, FieldScopedOptionKi
/// `SchemaManager.applyRenameColumnsToOptions` (Java's case-2 list; the
/// aggregation/partial-update engines do not accept every entry at create
/// time, but the keys can still arrive in Java-written metadata).
-const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 7] = [
+const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 8] = [
AGG_FUNCTION_SUFFIX,
IGNORE_RETRACT_SUFFIX,
DISTINCT_SUFFIX,
LIST_AGG_DELIMITER_SUFFIX,
+ NESTED_SEQUENCE_OVERWRITE_ON_EQUAL_SUFFIX,
Review Comment:
**[P1] Keep nested partial updates readable after a column rename**
I wrote two versions to an `items` column using `nested_partial_update`,
with a configured nested key, nested sequence field, and the new flag set to
false. `ALTER RENAME items TO lines` succeeds. The base still reads the renamed
column, but this revision fails with `requires ... nested-sequence-field`. The
rename moves the flag and nested key while leaving the nested sequence option
under the old field name. The constructor then validates the flag even in
`nested_partial_update`, where it has no effect. Keep related option names
synchronized and restrict the new runtime checks to `nested_update`.
##########
docs/src/sql.md:
##########
@@ -4014,9 +4014,27 @@ Field-scoped options used by these functions:
- `fields.<field>.ts-field` selects the key-time field for
`merge_map_with_keytime` (defaults to the last ROW field).
- `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
+ `fields.<field>.nested-sequence-overwrite-on-equal` (default `true`;
+ requires `nested-sequence-field`; applies only to `nested_update`),
`fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
for nested array aggregators. `nested_partial_update` requires
- `nested-key`.
+ `nested-key`. When `nested-sequence-field` is set for `nested_update`, a
+ nested ROW overwrites an equal-key accumulator if its sequence is greater,
+ or equal when `nested-sequence-overwrite-on-equal` is `true` (default).
+ Set it to `false` to overwrite only when the new sequence is strictly
+ greater. That strict equal-sequence policy is currently enforced by Rust
+ writer/compaction paths that implement the option; a Java runtime that has
+ not implemented it may still overwrite the nested ROW when sequences are
+ equal. Tables that rely on `false` to keep equal sequences from overwriting
+ should not mix writers or compaction from a Java runtime that does not yet
Review Comment:
**[P2] Include Java readers in the interoperability restriction**
I read the same uncompacted snapshot with Rust and the Java Paimon 2.2
snapshot build `20261118.220748-71`. Two Rust commits used the same nested key
and sequence, with amounts 10 and 20. Rust read 10. Java read 20 without
performing any writes or compaction. Please document that Java snapshot reads
can also see different winners while equal-sequence versions coexist. Changing
the option after compaction cannot restore values that have already been
discarded.
##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -272,6 +280,10 @@ fn parse_field_scoped_option_key(key: &str) ->
Option<(&str, FieldScopedOptionKi
NESTED_SEQUENCE_FIELD_SUFFIX,
FieldScopedOptionKind::NestedSequenceField,
),
+ (
+ NESTED_SEQUENCE_OVERWRITE_ON_EQUAL_SUFFIX,
+ FieldScopedOptionKind::NestedSequenceOverwriteOnEqual,
Review Comment:
**[P1] Reject invalid tie-policy changes before publishing the schema**
On a populated `nested_update` table with a nested key and no nested
sequence field, a public filesystem `ALTER SET
fields.items.nested-sequence-overwrite-on-equal=false` succeeded and persisted
schema 1. Its previously readable rows then failed with `requires ...
nested-sequence-field`. This option kind currently checks the owner column but
leaves its value and dependency checks to runtime. Validate the boolean and its
prerequisite during both CREATE and ALTER for fields that use the option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]