LsomeYeah opened a new pull request, #11086:
URL: https://github.com/apache/paimon/pull/11086

   ### Purpose
   
   Add `fields.<field>.aggregate-function=merge_top_level` for JSON Object 
strings in primary-key tables, providing top-level key merging similar to 
`merge_map` without requiring a MAP-typed column.
   
   - Newer inputs replace entire values of matching top-level properties; 
omitted properties are retained. Nested objects and arrays are not recursively 
merged.
   - SQL NULL does not update a field. JSON property null is retained as a 
value, not a deletion marker; a JSON null root is rejected. Empty objects do 
not clear existing properties.
   - Follow the engine's aggregation/sequence ordering, including reverse 
aggregation. Property names are decoded, literal, and case-sensitive.
   - Reject invalid/non-Object JSON, trailing values, duplicate keys at any 
nesting level, and unrepresentable numbers. Preserve integer/decimal precision 
without converting through double.
   - Support STRING (unbounded VARCHAR). Retraction is unsupported by default; 
the existing explicit ignore-retract mechanism is unchanged.
   
   The reducer's single-input shortcut otherwise bypasses validation for a lone 
record. Add an internal opt-out used by this aggregator and propagated through 
aggregation, partial-update, and lookup wrappers. Existing aggregators retain 
their shortcut by default.
   
   ### Tests
   
   Verified with JDK 8 and Flink 1.20.1: **208 tests passed, 0 
failures/errors/skips**.
   
   - Six focused unit tests cover top-level replacement, null/empty semantics, 
reverse ordering, associativity/idempotence, literal keys, strict validation, 
numeric precision/round trips, type validation, and unsupported retraction.
   - Core table tests cover buffered merges, cross-commit reads, older-sequence 
contributions, and forced compaction for both aggregation and partial-update.
   - Flink SQL IT covers both merge engines and rejects a single invalid JSON 
input. Without the single-input opt-out, this assertion fails for both engines 
because the invalid input is accepted.
   - Existing field aggregation, aggregation/partial-update merge, lookup 
merge, and table tests are included in the regression scope.
   - Checkstyle, Spotless, and `git diff --check` passed.
   
   ```sh
   mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
     -DwildcardSuites=none -DfailIfNoTests=false \
     
'-Dtest=FieldAggregatorTest,AggregateMergeFunctionTest,PartialUpdateMergeFunctionTest,LookupMergeFunctionTest,PrimaryKeySimpleTableTest#testMergeTopLevel+testMergeMapOverwriteOnEqualTs,BasicAggregateITCase'
 \
     test
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to