PyRSA opened a new pull request, #8374:
URL: https://github.com/apache/paimon/pull/8374
### Purpose
[Feature] Add `fields.{fieldName}.nested-key-null-strategy` configuration
for `nested_update` function
[Feature] Support nested-key null handling strategies in
FieldNestedUpdateAgg operator to control behavior when nested-key does not
satisfy primary key semantics
Expected behavior when nested-key contains null values or does not satisfy
primary key semantics:
In the following examples, `nested-key` is defined as `k0,k1`.
| **case** | **Input A** | **Input B** |
**Current Behavior** | **Expected Behavior** |
| ---------------------------- | ------------------- | ---------------- |
-------------------- | ----------------------------- |
| Default behavior (no config) | [(0, null, "A", 1)] | [(0, 1, "B", 2)] |
Merge | Merge |
| MERGE strategy | [(0, null, "A", 1)] | [(0, 1, "B", 2)] |
Merge | Merge |
| IGNORE strategy | [(0, null, "A", 1)] | [(0, 1, "B", 2)] |
Merge ❌ | Ignore invalid nested-key row |
| ERROR strategy | [(0, null, "A", 1)] | [(0, 1, "B", 2)] |
Merge ❌ | Throw exception |
---
### Behavior Definition
- `merge`
Merge rows even if nested-key does not satisfy primary key semantics.
This is equivalent to not configuring `nested-key-null-strategy`.
- `ignore`
Ignore rows whose nested-key does not satisfy primary key semantics.
- `error`
Throw an exception when nested-key does not satisfy primary key semantics.
---
### Tests
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithNestedKeyNullStrategyArgumentCheck
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithoutNestedKeyNullStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseMergeStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseIgnoreStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseThrowErrorStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseMergeStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseIgnoreStrategy
-
org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseThrowErrorStrategy
---
### API and Format
No Changes.
---
### Documentation
docs/docs/primary-key-table/merge-engine/aggregation.mdx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
