Gabriel39 opened a new pull request, #68614:
URL: https://github.com/apache/doris/pull/68614
### What problem does this PR solve?
Flight SQL attaches `doris_type` metadata to top-level LARGEINT/IP fields,
but ARRAY items, MAP keys/values and STRUCT children lose it. A nested LARGEINT
therefore arrives as an Arrow string that clients cannot distinguish from
ordinary text. JSON and VARIANT also lack the marker at the top level.
For example, in `SELECT named_struct('number', CAST(17 AS LARGEINT), 'text',
'17')`, both children currently appear as unannotated strings. This change
annotates only the LARGEINT child, allowing clients to restore its logical type
safely.
Related PR: #67530 (closed while awaiting type-mapping refactoring). This
change targets the current `branch-4.1` schema and converter implementation.
Build nested fields through the existing metadata helper and add
JSON/VARIANT to that helper. Preserve Arrow storage types, child names,
nullability and encoded values; in particular, MAP keys remain non-nullable and
LARGEINT retains its string encoding for the full signed 128-bit range.
Metadata is carried at every nesting depth and survives Arrow IPC
serialization. No session setting is required.
PyArrow does not automatically interpret custom field metadata. Applications
can use `doris_type=LARGEINT` to convert the corresponding values to integers
without guessing from their text.
### Release note
Preserve Doris logical type metadata for LARGEINT, IPV4, IPV6, JSON and
VARIANT inside Arrow arrays, maps and structs. JSON and VARIANT now also
include the marker at the top level.
### Check List (For Author)
- Test:
- BE ASAN: all 20 tests passed with `bash run-be-ut.sh --run
--filter='*MetadataTest*:ArrowBlockConvertorTest.*' -j 32` (9 new metadata
tests plus 11 existing converter tests).
- Before the fix, 8 of the 9 new tests failed on missing metadata; the
native-type negative control passed.
- Coverage includes all five logical types at the root and in
ARRAY/STRUCT/MAP, LARGEINT map keys, deep nesting, names/nullability, signed
128-bit extremes, NULL values, and Arrow IPC round trips.
- clang-format 16 and `git diff --check` passed.
- The optional ADBC integration script passes syntax/import checks; both
live tests were skipped because no cluster URI was provided. No live-cluster
result is claimed.
- Behavior changed: additional Arrow field metadata only; physical types and
values remain compatible.
- Documentation: describe nested metadata and client conversion in the
Python Flight SQL sample README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
