LsomeYeah opened a new pull request, #11086:
URL: https://github.com/apache/paimon/pull/11086
### Purpose
Add `fields.<field>.aggregate-function=merge_top_level` for JSON Object
strings in primary-key tables, providing top-level key merging similar to
`merge_map` without requiring a MAP-typed column.
- Newer inputs replace entire values of matching top-level properties;
omitted properties are retained. Nested objects and arrays are not recursively
merged.
- SQL NULL does not update a field. JSON property null is retained as a
value, not a deletion marker; a JSON null root is rejected. Empty objects do
not clear existing properties.
- Follow the engine's aggregation/sequence ordering, including reverse
aggregation. Property names are decoded, literal, and case-sensitive.
- Reject invalid/non-Object JSON, trailing values, duplicate keys at any
nesting level, and unrepresentable numbers. Preserve integer/decimal precision
without converting through double.
- Support STRING (unbounded VARCHAR). Retraction is unsupported by default;
the existing explicit ignore-retract mechanism is unchanged.
The reducer's single-input shortcut otherwise bypasses validation for a lone
record. Add an internal opt-out used by this aggregator and propagated through
aggregation, partial-update, and lookup wrappers. Existing aggregators retain
their shortcut by default.
### Tests
Verified with JDK 8 and Flink 1.20.1: **208 tests passed, 0
failures/errors/skips**.
- Six focused unit tests cover top-level replacement, null/empty semantics,
reverse ordering, associativity/idempotence, literal keys, strict validation,
numeric precision/round trips, type validation, and unsupported retraction.
- Core table tests cover buffered merges, cross-commit reads, older-sequence
contributions, and forced compaction for both aggregation and partial-update.
- Flink SQL IT covers both merge engines and rejects a single invalid JSON
input. Without the single-input opt-out, this assertion fails for both engines
because the invalid input is accepted.
- Existing field aggregation, aggregation/partial-update merge, lookup
merge, and table tests are included in the regression scope.
- Checkstyle, Spotless, and `git diff --check` passed.
```sh
mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
-DwildcardSuites=none -DfailIfNoTests=false \
'-Dtest=FieldAggregatorTest,AggregateMergeFunctionTest,PartialUpdateMergeFunctionTest,LookupMergeFunctionTest,PrimaryKeySimpleTableTest#testMergeTopLevel+testMergeMapOverwriteOnEqualTs,BasicAggregateITCase'
\
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]