JingsongLi commented on code in PR #1406:
URL: https://github.com/apache/paimon-rust/pull/1406#discussion_r4581215025
##########
crates/paimon/src/table/aggregator/nested.rs:
##########
@@ -293,6 +316,18 @@ impl NestedAgg {
Ok(Ordering::Equal)
}
+ /// Whether `new` may replace `old` for `nested_update` (Java
`shouldUpdate`).
+ fn should_update(&self, new: &StructArray, old: &StructArray) ->
crate::Result<bool> {
+ if self.sequence_indices.is_empty() {
+ return Ok(true);
+ }
+ match self.compare_sequence(new, old)? {
+ Ordering::Greater => Ok(true),
+ Ordering::Equal => Ok(self.overwrite_on_equal_sequence),
Review Comment:
[P1] Reject or correctly support false with partial-update sequence groups
The new tie rule makes query results depend on flush boundaries. Reproduced
through public Table write/commit/read/compact APIs with `id INT, version INT,
items ARRAY<ROW<item_id INT, amount INT>>`, PK `id`, `bucket=1`,
`merge-engine=partial-update`, `fields.version.sequence-group=items`, and
`fields.items.{aggregate-function=nested_update, nested-key=item_id,
nested-sequence-field=item_id, nested-sequence-overwrite-on-equal=false}`. For
one PK, write `(version,items)` in order `(3,[(1,30)]), (1,[(1,10)]),
(2,[(1,20)])`. Three separate commits read `[(1,20)]`; committing version 3
first and flushing versions 1 and 2 together reads `[(1,10)]`. The true control
returns 30 with both groupings; aggregation returns 10 with both false
groupings.
After merging outer versions 3 and 1, the nested winner is 10, but
PartialUpdateMergeFunction still records group max version 3. Version 2 then
enters `agg_reversed`, incorrectly prepending 20 ahead of the version-1 winner;
this equal branch prevents 10 from replacing it. Full compaction preserves each
differing result. Please retain the ordering/provenance needed for each nested
winner, or reject false with partial-update sequence groups until supported.
Add a regression with identical arrival order and different flush groupings.
##########
docs/src/sql.md:
##########
@@ -3556,9 +3556,14 @@ Field-scoped options used by these functions:
- `fields.<field>.ts-field` selects the key-time field for
`merge_map_with_keytime` (defaults to the last ROW field).
- `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
+ `fields.<field>.nested-sequence-overwrite-on-equal` (default `true`;
+ requires `nested-sequence-field`),
`fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
for nested array aggregators. `nested_partial_update` requires
- `nested-key`.
+ `nested-key`. When `nested-sequence-field` is set, a nested ROW overwrites
Review Comment:
[P2] Limit the sequence-overwrite guarantee to nested_update
The new rule follows both nested aggregators, but
`NestedMode::PartialUpdate` bypasses `should_update`. `nested_partial_update`
still applies equal/lower-sequence non-null fields even with false, as
`nested_partial_update_unaffected_by_overwrite_on_equal` confirms. Explicitly
scope this guarantee to `nested_update` so users do not rely on it for
stale-event protection with `nested_partial_update`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]