Doris-Breakwater commented on issue #67364: URL: https://github.com/apache/doris/issues/67364#issuecomment-5490565527
## Initial triage I reviewed the reported commit (`31263df4dc1d4d3a27517d264802cd4d6b92c874`) read-only. This is a real server-side protocol failure, but the strongest code-backed diagnosis is **an Arrow Flight SQL / SQL-cache replay bug, not a missing Arrow representation for `HLL` or `QUANTILE_STATE`**. ### Verified facts - At this commit, FE schema construction explicitly maps `HLL`, `BITMAP`, and `QUANTILE_STATE` to [`ArrowType.Binary`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlSchemaHelper.java#L151-L154). The BE likewise maps all three to [`arrow::binary()`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/format/arrow/arrow_row_batch.cpp#L170-L174), and has binary Arrow writers for both [HLL](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_hll_serde.cpp#L128-L145) and [QuantileState](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_quantilestate_serde.h#L101-L120). Therefore, the intended raw Flight result is a binary Arrow column; a clean “unsupported type” error is not the appropriate first fix for this code versio n. - The MySQL `NULL` result is separate, expected compatibility behavior. `return_object_data_as_binary` defaults to false, and the MySQL text serializers for these object types decline to serialize in that mode. It does not mean the stored value is null and does not show that Arrow cannot represent it. - [`StmtExecutor.sendFields()` is MySQL-only](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1794-L1800) and begins by asserting `ConnectType.MYSQL`. A normal BE-backed Flight query returns through the Arrow path before any MySQL fields are sent ([lines 1420-1439](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1420-L1439)). - There is, however, an unguarded cached-plan branch: for Flight, `MysqlChannel channel` remains null, but a `SqlCache` plan still calls `sendCachedValues(...)` ([lines 1337-1349](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1337-L1349)), which then calls `sendFields()`. The later cache-analysis path already has a `TODO support arrow flight sql` and is correctly gated on `channel != null` ([lines 1353-1363](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1353-L1363)); the earlier cached-plan replay is not. - `enable_sql_cache` defaults to true. Consequently, an identical query previously cached through MySQL can deterministically enter this branch and raise the reported `IllegalStateException` for **any output type**. The apparent type specificity can be explained if the raw-state SQL was used on the MySQL control path first, while the scalar-conversion SQL was a different cache key. ### Confirmation still needed The issue does not include the complete FE stack or cache evidence, so the code mechanism above is highly specific but not yet proven to be the trigger in this run. Please provide: 1. The complete FE exception stack and query ID. The decisive frame is whether `sendFields()` was called by `sendCachedValues()` from the `SqlCache` branch. 2. The matching `fe.audit.log` entry, including `HitSqlCache`, plus `SHOW VARIABLES LIKE 'enable_sql_cache'` for both the MySQL and Flight sessions. 3. The exact order and timing of the MySQL control query and Flight query. The default result-cache freshness guard is 30 seconds after the latest table version, but deployments can override it. 4. Exact `adbc_driver_flightsql` and `pyarrow` versions. These are unlikely to cause a Java-side `Preconditions` failure, but they are still needed for a complete reproduction record. The fastest discriminator is: ```sql -- Run in the Flight SQL session. SET enable_sql_cache = false; SELECT h, q FROM adbc_opaque_state; ``` Also retry on a freshly created table with Flight issuing the raw `SELECT` before any identical MySQL `SELECT`. Conversely, once the table is cache-eligible, prime an identical scalar query through MySQL and then run it through Flight; if that also reaches the same exception, it confirms that the failure is cache/protocol-specific rather than type-specific. ### Recommended next steps - Treat this as `kind/bug`, involving `area/rpc` plus the SQL-cache owner (the issue currently has no labels). - Prevent an Arrow Flight session from consuming the current MySQL-row SQL-cache replay path, either by bypassing `PhysicalSqlCache` for Flight or by adding a protocol-neutral/Arrow-native cached-result path. Add a regression that exercises an actual cache hit, for both a simple scalar result and raw `HLL` / `QUANTILE_STATE` binary results. - Add direct Flight coverage asserting that uncached raw `HLL` and `QUANTILE_STATE` columns produce Arrow `binary` arrays. - If the query still fails with SQL cache disabled, attach the new full FE/BE logs and test `h` and `q` separately plus `WHERE 1 = 0`; that will distinguish schema construction from value serialization without guessing at another root cause. Current workarounds are the scalar conversions already listed in the report, or `HLL_TO_BASE64(h)` / `QUANTILE_STATE_TO_BASE64(q)` when the opaque state must be preserved. Disabling SQL cache for the Flight session is also a targeted workaround if the discriminator above confirms the cache path. Breakwater-GitHub-Analysis-Slot: slot_12cbbf321ecc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
