Doris-Breakwater commented on issue #67372: URL: https://github.com/apache/doris/issues/67372#issuecomment-5490655659
Breakwater-GitHub-Analysis-Slot: slot_ad34727ee091 ## Initial triage This is a confirmed server-side type-model incompatibility, not an ADBC/PyArrow client issue. The supplied reproducer is sufficient for triage, and source inspection at `31263df4dc1d4d3a27517d264802cd4d6b92c874` matches the reported error exactly. One NULL key makes the affected `DoGet` stream unreadable, although it does not mutate or corrupt stored data. The documented `map_entries(m)` workaround is lossless for this case. The issue currently has no labels. Suggested labels are `kind/bug` and `area/rpc`; it is also already indexed by #65615. ## Verified mechanism 1. [`convert_to_arrow_type()`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/format/arrow/arrow_row_batch.cpp#L141-L151) maps every Doris `TYPE_MAP` to `arrow::MapType` based only on the expression type. 2. The Flight result sink builds and registers that schema during [`init()` / `prepare()`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/exec/operator/result_sink_operator.cpp#L57-L65), before result rows are available. 3. Both the local and remote Flight readers convert Doris blocks to Arrow record batches during `DoGet`; for example, the [local reader](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/service/arrow_flight/arrow_flight_batch_reader.cpp#L93-L118) calls `convert_to_arrow_batch()` only after fetching a result block. 4. [`DataTypeMapSerDe::write_column_to_arrow()`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_map_serde.cpp#L340-L386) inspects the actual key null map at that point and returns `INVALID_ARGUMENT` with the exact reported message when it finds a NULL key. This is required by Arrow's MAP layout: the Arrow format states that neither the `entries` field nor the `key` field may be nullable ([Arrow schema definition](https://github.com/apache/arrow/blob/apache-arrow-24.0.0/format/Schema.fbs#L118-L145)). Doris, by contrast, deliberately stores MAP keys in nullable child columns; `map()` also constructs a nullable key type. Static inspection of the current local `master` still finds the same rejection path, so there is no apparent master-side fix yet, although I did not run a live cluster reproduction. `map_entries(m)` works for a concrete reason: it changes the Doris result type to `ARRAY<STRUCT<key, value>>`, and its key struct field is nullable. Arrow then receives a List/Struct schema rather than logical MAP, so the NULL key is representable. ## Design constraints and recommendation A per-row or per-batch fallback from Arrow MAP to List/Struct is not valid: the Flight stream schema is established before batches and must remain stable. Similarly, rejecting only MAPs that *will* contain a NULL key at plan/schema time is generally not possible from the current Doris type alone; determining that for a table result requires inspecting data. A type-time rejection would therefore have to reject all raw MAP outputs (or introduce a stronger non-null-key contract), including maps whose actual keys are valid for Arrow. The maintainer decision should therefore be explicit and query-wide: - For lossless Doris semantics, map every Doris MAP result (recursively, including nested MAPs) to `list<struct<key nullable, value nullable>>` for Flight SQL, or initially expose that mapping behind an explicit connection/session compatibility option. This changes the client-visible Arrow type for existing non-NULL maps, so it needs a compatibility and release-note decision. - If preserving the existing Arrow MAP type is more important, keep the limitation but reject conservatively before streaming where feasible and return a protocol-specific error that names the result column and recommends `map_entries()`. Document that Doris MAP values containing NULL keys are not representable. Do not silently drop, replace, or filter NULL-key entries. Whichever policy is chosen, the stale comment in `DataTypeMapSerDe` saying the NULL entry is ignored should be corrected; the implementation returns an error. ## Missing information No additional logs, profile, or client metadata are needed to establish this mechanism. The exact `adbc_driver_flightsql` and PyArrow versions would only be useful for pinning an end-to-end regression environment, not for determining the root cause. Runtime confirmation on current `master` remains useful before closing the issue, but it is not blocking design or implementation. ## Suggested next steps 1. Decide the stable Flight SQL mapping policy for Doris MAP and document it in the Arrow Flight SQL type-compatibility section. 2. Add an end-to-end `arrow_flight_sql` regression using the provided table case; assert both the returned Arrow schema and preservation of the NULL key under the chosen policy. 3. Cover a mixed map with NULL and non-NULL keys, a NULL value with a non-NULL key, a NULL key appearing in a later result batch, and nested MAPs. Exercise both local and remote result-reader paths if the test framework supports them. 4. Remove the blanket skip added by #65182 only when the selected behavior is implemented and verified; otherwise keep the skip linked to the documented limitation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
