Gabriel39 opened a new pull request, #69127:
URL: https://github.com/apache/doris/pull/69127
### What problem does this PR solve?
Annotated Parquet Variant columns can be rejected before reading any rows
when a shredded array element has only a required BYTE_ARRAY `value`. This
layout was already accepted through Paimon schema overrides but rejected for
VARIANT(1)-annotated files, including files read through Iceberg.
Remove `allow_paimon_shredded_layout` and share the existing compatibility
rules across annotated Variant columns and table-format overrides:
- Accept a required fallback only for array elements without a `typed_value`
carrier.
- Accept optional metadata only in shredded layouts, and optional shredded
wrappers.
- Preserve runtime rejection of missing metadata and null array elements,
and require optional fallbacks for object fields and arrays with typed carriers.
Add schema tests and a native Parquet reader regression with two entirely
synthetic six-row fixtures and a reproducible generator. They cover integer and
Variant-null elements, empty arrays, outer SQL NULL, root Variant null, absent
object fields, and multiple row groups. The fixtures contain no external data.
### Release note
Fix reading annotated Parquet Variant columns with compatible shredded
layouts previously accepted only through Paimon schema overrides.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test
Validation: the new schema and file-reader cases fail on the original
implementation and pass with this change. A focused ASAN test build passed 119
schema/materialization/footer tests and the new native file-reader test. A
separate native reader probe also reconstructed all Variant values in an
independently supplied file. The focused build reused existing dependency
libraries; a full product build and catalog-level SQL regression were not run
locally. All affected C++ files pass clang-format 16. Synthetic fixture
regeneration is byte-for-byte reproducible.
- Behavior changed:
- [x] Yes. Annotated Variant columns share the existing shredded-layout
compatibility rules.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
