zhuxiangyi opened a new pull request, #9813:
URL: https://github.com/apache/paimon/pull/9813
### Purpose
**This is a bug fix.** With `variant.inferShreddingSchema=true`, a Variant
object whose key is empty or whitespace-only fails the whole file write:
```sql
CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES
('variant.inferShreddingSchema' = 'true');
INSERT INTO T VALUES (1, parse_json('{"":1,"a":2}'));
-- Job aborted ... java.lang.IllegalArgumentException: Field names must
contain at least one non-whitespace character.
-- at org.apache.paimon.types.RowType.validateFields(RowType.java:283)
-- at
org.apache.paimon.data.variant.InferVariantShreddingSchema.schemaOf(InferVariantShreddingSchema.java:318)
```
`InferVariantShreddingSchema.schemaOf` turns every key of a sampled object
into a `DataField` and builds a `RowType` from them, and `RowType` rejects a
blank field name. An empty key is a valid Variant object key (the same row
writes and reads back fine with inference off), so inference must not turn it
into a write failure. Other unusual keys (`a.b`, keys with spaces,
`typed_value` / `value` / `metadata`, non-ASCII) already work.
The fix skips blank keys when building the inferred object type, so they
stay in the unshredded `value` while the remaining keys are still shredded; an
object made only of blank keys stays unshredded. The alphabetical-order check
now iterates the kept fields rather than the original object size.
### Tests
The new tests reproduce the bug: on `master` they fail with the
`IllegalArgumentException` above, and pass with this change.
- `InferVariantShreddingSchemaTest#testInferSchemaSkipsBlankKeys`: `{"":1,"
":2,"a":3}` infers to a typed object with only `a`; `{"":1}` alone stays
unshredded.
- `InferVariantShreddingWriteTest#testInferSchemaSkipsBlankKeys`: writes
such rows with inference on, checks the physical Parquet schema, the
reconstructed Variant (blank keys preserved) and typed extraction of `$.a`.
- `VariantTestBase`: inserts `{"":1,"a":2}` and reads it back with `to_json`
and `variant_get`; runs under all four Spark 4.x configurations. Verified on
Spark 4.1.2: 116 tests pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
