JingsongLi commented on code in PR #1406:
URL: https://github.com/apache/paimon-rust/pull/1406#discussion_r4586084530
##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -272,6 +274,10 @@ fn parse_field_scoped_option_key(key: &str) ->
Option<(&str, FieldScopedOptionKi
NESTED_SEQUENCE_FIELD_SUFFIX,
FieldScopedOptionKind::NestedSequenceField,
),
+ (
+ NESTED_SEQUENCE_OVERWRITE_ON_EQUAL_SUFFIX,
Review Comment:
[P2] Register the new field option in rename and drop handling too
This adds the suffix to field validation, but
`FIELD_SCOPED_RENAMEABLE_SUFFIXES`, used by both `rename_field_scoped_options`
and `remove_field_scoped_options`, omits it. Consequently the old
`fields.items.nested-sequence-overwrite-on-equal=false` key survives
renaming/dropping items, and final schema validation rejects the ALTER because
items no longer exists.
I isolated this from the pre-existing omissions for older nested options
with public `TableSchema::apply_changes`: explicitly migrate
`nested-sequence-field` alongside the rename (nested-key is renamed by the
existing field-list handler), or explicitly remove `nested-key` and
`nested-sequence-field` before the drop. Both controls without the new option
succeed; adding only this new option makes both operations fail with
`Aggregation field 'items' ... is not declared`. Please include this suffix in
the lifecycle registry and add rename/drop regressions.
##########
crates/paimon/src/table/aggregator/nested.rs:
##########
@@ -293,6 +316,18 @@ impl NestedAgg {
Ok(Ordering::Equal)
}
+ /// Whether `new` may replace `old` for `nested_update` (Java
`shouldUpdate`).
+ fn should_update(&self, new: &StructArray, old: &StructArray) ->
crate::Result<bool> {
+ if self.sequence_indices.is_empty() {
+ return Ok(true);
+ }
+ match self.compare_sequence(new, old)? {
+ Ordering::Greater => Ok(true),
+ Ordering::Equal => Ok(self.overwrite_on_equal_sequence),
Review Comment:
[P1] Also reject or support false with aggregation and an outer sequence
field
The unsafe provenance case remains reachable outside partial-update.
Reproduced with PK `pk`, `bucket=1`, `merge-engine=aggregation`,
`sequence.field=version`, and `items ARRAY<ROW<item_id INT, amount INT>>`
configured with `aggregate-function=nested_update`, `nested-key=item_id`,
`nested-sequence-field=item_id`, `nested-sequence-overwrite-on-equal=false`.
For one PK and item_id=1, input `(version,amount)` in arrival order
`(3,30),(1,10),(2,20)` reads amount 10 when committed separately or in one
batch, but 20 when versions 3 and 1 are flushed together before committing
version 2. Explicit compaction preserves 10 versus 20. The true control returns
30 for every grouping; without the outer sequence field both option values are
grouping-consistent.
The first flush retains the version-1 nested winner but records version 3 in
the aggregate row. The next merge orders version 2 before that summary; this
equal branch then retains 20. The new guard only covers PartialUpdateConfig.
Please preserve the nested winner's ordering/provenance or reject this
aggregation + outer-sequence combination until supported, and add a public
regression with identical arrival order and different flush groupings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]