Gabriel39 opened a new pull request, #68883:
URL: https://github.com/apache/doris/pull/68883
### What problem does this PR solve?
Forward-port only the requested changes whose missing behavior is still
present on master, in branch-4.1 merge order: #68614, then #68667, followed by
#68859.
| Source PR | Evidence on master before this change | Action |
| --- | --- | --- |
| #68614 | Arrow ARRAY/MAP/STRUCT child fields lose Doris logical type
markers; JSON/VARIANT markers are absent. | Preserve recursive logical metadata
and use protocol-specific schema converters. |
| #68667 | Flight maps Variant V2 to UTF8 instead of native binary values. |
Return `arrow.parquet.variant` with `struct<metadata: binary not null, value:
binary not null>` storage; preserve SQL NULL, physical scalar types and
row-local dictionaries. |
| #68852 | The obsolete external-table queries are already removed. All 276
active query tags match the 276 expected output blocks in execution order. |
Excluded; no expected-output changes. |
| #68856 | The preload module is already removed. The Hudi plugin resolves
Parquet readers and format structures consistently to 1.18.0. | Excluded; no
dependency changes. |
| #68859 | Block metadata omits VARBINARY length, and deserialization
constructs `DataTypeVarbinary()` with an unspecified length. | Preserve
declared lengths, including nested and nullable columns, using optional
protobuf field 13. |
Adapt the ports to master's existing architecture: retain native UUID and
nanosecond/timestamp mappings, reuse the shared FE Arrow mapper, and wrap
extension storage correctly for nested Variant arrays. Master already uses
Variant V2, so the removed legacy Variant switch and SerDe are not restored.
Branch-4.1-only Iceberg Variant writing and prepared-statement schema machinery
are outside this port.
The Flight integration tests and Python samples cover native Variant and
recursive metadata. The Flink regression explicitly casts Variant to STRING for
a connector that declares that column as STRING.
### Release note
Preserve nested logical type metadata in Arrow Flight SQL results, return
Variant V2 as native Arrow Variant binary, and retain declared VARBINARY
lengths across block serialization. Clients requiring Variant JSON text should
explicitly cast it to STRING.
### Validation
- FE targeted unit tests: 64 passed, including nested metadata, native
Variant, UUID preservation and shared Arrow mappings.
- FE Checkstyle: passed.
- BE targeted ASAN unit tests: 326 passed, covering Arrow, Variant, UUID and
VARBINARY.
- Changed-line clang-tidy, clang-format 16 and build hygiene checks: passed.
- Groovy regression scripts compiled; Python samples passed syntax checks.
- Checked modified Thrift/protobuf field IDs against master and branch-3.1,
branch-4.0, branch-4.1 and branch-4.2.
- Confirmed the ES baseline query/output alignment and Hudi Parquet
dependency tree before excluding those ports.
- Live cluster, ADBC and Flink integration tests were not run locally.
### Check List (For Author)
- Test
- [x] Unit tests
- [x] Regression tests added/updated
- Behavior changed:
- [x] Yes, native Variant Flight output and preserved type metadata.
- Does this need documentation?
- [x] Yes, Python Flight sample documentation updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
