PDGGK opened a new pull request, #9532:
URL: https://github.com/apache/paimon/pull/9532
### Purpose
Adaptive variant shredding inference throws and fails the write when a
node's evidence is gone but its selected schema is still a `ROW`.
`finalizeAdaptiveSchema` substitutes the previously *selected* schema when
the current file has no evidence for a node:
```java
if (combined == null || combined instanceof VariantType) {
if (current != null && !(current instanceof VariantType)) {
combined = current;
} else if (previousSelected != null) {
combined = previousSelected; //
InferVariantShreddingSchema:576-577
} else {
return DataTypes.VARIANT();
}
}
```
Execution then falls into the `RowType` branch, which reads that substitute
as if it were evidence:
```java
double ratio = rootValueCount == 0 ? 0 : getFieldCount(field) /
rootValueCount; // :592
```
Selected schemas carry no per-field counts — `finalizeSimpleSchema` clears
them, and the adaptive `RowType` branch rebuilds fields with the
description-free three-argument `DataField` constructor — so `getFieldCount`
throws:
```
java.lang.IllegalStateException: Field 'x' is missing count in description.
This should not happen during schema inference.
at
InferVariantShreddingSchema.getFieldCount(InferVariantShreddingSchema.java:394)
at
InferVariantShreddingSchema.finalizeAdaptiveSchema(InferVariantShreddingSchema.java:592)
at
InferVariantShreddingSchema.finalizeAdaptiveSchema(InferVariantShreddingSchema.java:615)
at
InferVariantShreddingSchema.inferAdaptive(InferVariantShreddingSchema.java:134)
at
VariantShreddingInferenceSession.inferSchema(VariantShreddingInferenceSession.java:74)
```
Getting there needs three things in one rolling writer with
`variant.inferShreddingSchema=true` and
`variant.shredding.inferenceMode=adaptive`: a node drifts across a type family,
which degrades its combined evidence to `VARIANT` while the schema selected for
it is a `ROW`; and it is then absent from the next file, so there is no current
evidence either. Two files cannot do it — when a node is absent,
`combineEvidence` falls back to the previous file's *evidence*, which still
carries counts — so it takes three.
The exception comes out of `InferShreddingWritePlanWriter` uncaught, so the
file fails to write at all.
Three files of a `ROW<v VARIANT>` table, at the default options, are enough:
| file | rows | result |
|---|---|---|
| 1 | `{"k":1,"p":5}` | plans, `p` selected as `BIGINT` |
| 2 | `{"k":1,"p":{"x":1}}` | plans, `p` selected as `ROW<x BIGINT>`,
combined evidence for `p` degrades to `VARIANT` |
| 3 | `{"k":1}` | **throws** |
A root-level variant of the same shape — `42`, then `{"a":1}`, then SQL
`NULL` — fails the same way with `Field 'a'`.
### Summary and Changelog
Return `previousSelected` instead of assigning it to `combined`. It is
already a finalized selection: it has been through admission, retention and the
field budget once, and with no evidence for this file there is nothing to
re-threshold it against. Carrying it forward unchanged is also what the
retention path does for a node whose evidence is still a `RowType`.
### Tests
`InferVariantShreddingSchemaTest#testAdaptiveInferenceKeepsSelectedRowWhenEvidenceDegradedAndNodeIsAbsent`
drives the three files above and asserts `p` keeps `ROW<x BIGINT>`. On
`master` it errors with the `IllegalStateException` above, and the other 24
cases in the class are unaffected either way.
Four negative controls, run against a build of this ref, confirm the trigger
is that specific combination rather than absence alone — no drift and `p`
merely disappears; drift but `p` still present in file 3; object all the way,
which is the shape every existing adaptive test uses; and root scalar to scalar
to NULL. All four pass before and after.
The reason no existing test catches this: the four adaptive cases in
`InferVariantShreddingSchemaTest` and `InferVariantShreddingWriteTest` all keep
the root variant an object across every file, so combined evidence stays a
`RowType` and line 577 is never reached with a `RowType` in `previousSelected`.
`testAdaptiveInferenceWidensScalarSelectedFromPriorEvidence` comes closest —
its `second` column has no current evidence in round 2 — but its combined
evidence there is the previous file's evidence, which still carries counts.
`mvn test -pl paimon-common -Dtest='org.apache.paimon.data.variant.**'` — 52
tests, and `-pl paimon-format -Dtest=InferVariantShreddingWriteTest` — 18
tests, all passing. `spotless:check` and `checkstyle:check` clean.
### Tests API and Compatibility
No API, format or configuration change. Only the path that currently throws
behaves differently; every schema that infers successfully today infers the
same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
