Gabriel39 opened a new pull request, #68667:
URL: https://github.com/apache/doris/pull/68667
### What problem does this PR solve?
Flight SQL currently exposes VARIANT as UTF8, so ADBC clients cannot receive
the canonical binary Variant representation. Add the default-off session
variable `enable_arrow_flight_sql_native_variant` on branch-4.1. When enabled
on the querying Flight connection, VARIANT fields use the
`arrow.parquet.variant` extension with non-nullable binary `metadata` and
`value` children.
Pin the option in the result sink, propagate it through nested schema
mapping, align GetTables metadata, and register the extension before remote BE
schema decoding. Reuse the V2 canonical binary writer. Legacy scalar roots use
typed encoding where supported (including exact decimals), while legacy
documents use JSON conversion. Decimal256 roots are explicitly rejected. SQL
NULL remains a null struct; V2 Variant null remains an encoded non-null value.
Clients without an extension implementation can read the physical struct and
extension metadata.
### Release note
Support opt-in native VARIANT results over Arrow Flight SQL / ADBC using
`SET enable_arrow_flight_sql_native_variant = true`. The default UTF8 mode is
unchanged.
### Validation
- 31 BE tests passed, including six new Flight Variant tests and existing
converter, V2 output, and Paimon tests. Changed sources were compiled and
linked against cached build dependencies; this was not a clean full build.
- Two new FE tests passed for schema IPC metadata and result-sink option
capture. Changed FE sources and generated Thrift classes compiled; changed BE
sink and remote-reader sources also compiled.
- ADBC 1.12.0 / PyArrow 24.0.0 direct and partition reads preserved
BE-generated schemas and bytes through a local Flight transport fixture for
legacy and V2 data.
- FE Checkstyle and clang-format 16 passed. The new Flight regression suite
compiled, and the Python ADBC test passed syntax and result-assertion checks.
- Full Doris cluster execution of the new Groovy and Python integration
tests remains for CI / cluster validation. The transport fixture above does not
substitute for that execution.
### Check List (For Author)
- Test
- [x] Regression test added
- [x] Unit Test
- [x] Manual transport test described above
- Behavior changed:
- [x] Yes, only when native Variant output is enabled.
- Does this need documentation?
- [x] Yes, usage and client decoding limitations are documented in the
Python Flight sample README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
