SteNicholas opened a new pull request, #379:
URL: https://github.com/apache/paimon-cpp/pull/379
### Purpose
Linked issue: close #344
The generated physical schema of a shredded Variant column carried no
`paimon.id`, so the parquet
writer emitted no field id for `metadata`, `value`, `typed_value` or any of
their descendants. Java
Paimon converts the whole Variant group with its generic
`ParquetSchemaConverter`, which
dereferences the field id of every converted field, so reading such a file
threw a
`NullPointerException` before the first row. Configured, inferred and
adaptive shredding share this
schema construction, so all three produced files that only paimon-cpp could
read.
This numbers the generated fields the way the Java `PaimonShreddingUtils`
does: each level is
numbered from 0 in insertion order, an object field keeps the id from the
shredding schema or its
position when it has none, and an array element takes the id
`ParquetSchemaConverter` generates for
it (`SpecialFields.getArrayElementFieldId`).
### Tests
- `VariantShreddingTest.ShreddingSchemaFieldIds`
- `VariantShreddingTest.ShreddingSchemaRejectsInvalidFieldIds`
- `VariantShreddingTest.ShreddingArraySchemaFieldIds`
- `VariantParquetTest.ShreddedWriteAndReadRoundTrip`, extended to assert the
raw parquet footer
field ids of the whole shredded subtree under the configured, per-file and
adaptive shredding
modes
- `SpecialFieldsTest.TestGetArrayElementFieldId`, pinning the generated
element id against the
values the Java `SpecialFields` produces
### API and Format
No public API change. The storage format of shredded Variant columns
changes: the generated
physical fields now carry parquet field ids, matching what the Java writer
produces. Files written
before this change remain readable by paimon-cpp.
### Documentation
No, this is a bug fix and introduces no new feature.
### Generative AI tooling
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
