JingsongLi commented on code in PR #11163:
URL: https://github.com/apache/paimon/pull/11163#discussion_r4581461583
##########
paimon-python/pypaimon/execution/python/read/reader/aggregate/aggregators.py:
##########
@@ -1021,15 +1024,31 @@ def _check_option_dependencies(
"requires 'fields.<field-name>.nested-key' to be configured."
)
- if (
- options.field_nested_update_agg_nested_sequence_field(field)
- and not nested_key
- ):
+ nested_sequence_field = (
+ options.field_nested_update_agg_nested_sequence_field(field)
+ )
+ if nested_sequence_field and not nested_key:
raise ValueError(
"Option 'fields.<field-name>.nested-sequence-field' "
"requires 'fields.<field-name>.nested-key' to be configured."
)
+ overwrite_on_equal_configured = options.options.contains_key(
+ f"{CoreOptions.FIELDS_PREFIX}.{field}."
+ f"{CoreOptions.NESTED_SEQUENCE_OVERWRITE_ON_EQUAL}"
+ )
+ if overwrite_on_equal_configured and not nested_sequence_field:
+ raise ValueError(
+ "Option
'fields.<field-name>.nested-sequence-overwrite-on-equal' "
+ "requires 'fields.<field-name>.nested-sequence-field' to be
configured."
+ )
+
+ def _should_update(self, new_row: Record, old_row: Record) -> bool:
+ if not self.has_sequence_field:
+ return True
+ comparison = self._compare_sequence(new_row, old_row)
+ return comparison > 0 or (comparison == 0 and
self.overwrite_on_equal_sequence)
Review Comment:
[P1] Gate Native reads and writes until they honor this policy
This new policy is only applied by the Python aggregator, while Native read
and write remain eligible (and enabled by default). With
`fields.updates.aggregate-function=nested_update`, nested key `key`, nested
sequence `seq`, and `nested-sequence-overwrite-on-equal=false`, I committed two
versions of one PK/nested key at the same `seq=7`: Python reads `first`, but
Native Rust 0.5.0 reads `second`, with Python fallback forbidden. More
seriously, a single public `write_arrow` batch containing those two versions
through `NativeBatchTableWrite` persists only `second`; reopening via the
Python reader cannot recover `first`. The Python-writer control retains
`first`, and explicit `true` controls agree on `second`.
Please gate both Native read and write for this setting until the supported
runtime implements it (or update/require that runtime), and add real-table
regression coverage for both paths. Otherwise normal backend selection silently
changes the result and can irreversibly discard the value this option promises
to retain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]