Gabriel39 opened a new pull request, #68611:
URL: https://github.com/apache/doris/pull/68611
### What problem does this PR solve?
A valid Doris MAP containing a NULL key cannot be returned through Flight
SQL as an Arrow MAP, because Arrow forbids NULL map keys. For example, `SELECT
map(CAST(NULL AS STRING), 100)` fails during DoGet.
Add the opt-in session variable `arrow_flight_sql_map_as_list` (default
`false`). When enabled, MAP values are returned as Arrow `List<Struct<key,
value>>`, recursively through arrays, structs, and map values. Schema selection
happens before batches are read, so encountering a NULL key in a later batch
does not change the schema. The Flight converter shares immutable key/value
columns, null maps, and offsets to preserve entries without copying the Doris
column data; it adds an all-valid null map for the list entries. Flight SQL
GetTables metadata follows the same session setting.
The default Arrow MAP representation, MySQL results, Python Arrow bindings,
and external table writers retain their existing behavior. Enable the option
after upgrading the FE and BE nodes that serve the connection.
### Release note
Add `SET arrow_flight_sql_map_as_list = true` to read MAP values with NULL
keys through Flight SQL using a lossless list-of-structs representation. The
default remains disabled for compatibility with existing clients.
### Check List (For Author)
- Test:
- FE: all 10 `FlightSqlSchemaHelperArrowTypeTest` tests passed, including
recursive list schemas, IPC round trips, default compatibility, and Thrift
option forwarding.
- BE: all 14 `ArrowBlockConvertorTest` tests passed under ASAN, including
the 3 new regressions for NULL keys and values, NULL/empty maps, sliced
batches, nested containers, constant DATETIME values, and unchanged
native/Python bindings. The 3 new regressions fail with the original writer.
- FE Checkstyle and clang-format 16 checks passed.
- ADBC integration test script added under
`samples/arrow-flight-sql/python/` and syntax-checked. It has not been run
against a live cluster.
- Behavior changed: Yes, only when the new session option is enabled.
- Documentation: Usage and compatibility notes are included in the Python
Flight SQL sample README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
