morningman opened a new issue, #65615: URL: https://github.com/apache/doris/issues/65615
## Goal Use this issue as the single index for user-visible bugs, protocol gaps, reliability work, and ecosystem improvements related to Doris Arrow Flight SQL and the ADBC or Flight SQL clients that use it. ADBC is the client API and driver layer. Doris implements the Arrow Flight SQL server path that those clients call. This tracker covers both sides when a client report identifies a Doris interoperability or server behavior problem. Foundational context: #25514 and #19812. ## Scope and search method - Review window: **2025-07-15 through 2026-07-15**, inclusive. - Primary scope: `apache/doris` issues, PR title/body/comments, and history of the FE/BE Arrow Flight SQL implementation and regression suites. - Carry-over scope: older items that were updated or closed during the review window and still help explain an open problem. - Cross-repository scope: Doris Website, Doris Spark/Flink connectors, and Apache Arrow client reports that explicitly name Doris. - Search terms included `ADBC`, `Arrow Flight`, `Flight SQL`, `arrow_flight`, `flight_sql`, `FlightSQL`, and relevant implementation paths. - Generic Arrow file-format, UDF, dependency, and release-note items with no Flight SQL behavior were excluded. Status was checked on **2026-07-15**. ### Status convention - [ ] Open, unmerged, closed without a verified fix, duplicated but not closed, or otherwise needs follow-up. - [x] The referenced change was merged or the work item was confirmed complete. Backports are listed beside their primary PR. ## Open or needs follow-up ### Reliability, lifecycle, and correctness - [ ] **FE direct-memory leak during prepared-statement creation:** #65305. Current fix: #65311. An earlier fix attempt, #52245, was closed without merge. Add a repeat-query memory test and verify failed/cancelled statements and session teardown, not only the happy path. - [ ] **External-table split lifecycle across `GetFlightInfo` and `DoGet`:** #62259 is closed, but its root-cause fix #64799 is still open. #64797 and its 4.1 backport #64890 harden the error path so the BE does not crash; they explicitly do not keep the `SplitSource` alive. Do not consider the root cause complete until #64799 or a replacement is merged and the required release branches are covered. - [ ] **Corrupted large string/JSON results:** #61124 reports invalid StringArray offsets and an empty data buffer. - [ ] **DDL regression on 4.0.5:** #62017. Proposed fix #62018 was closed without merge. Reproduce on supported 4.0/4.1 builds and land or replace the fix. - [ ] **BE crash with `public_host` and an ILB proxy:** #62217. Separate the endpoint-routing problem from the Arrow/gRPC status-conversion crash path and verify both. - [ ] **LargeUtf8 offset handling can crash remote Doris reads:** #62681. - [ ] **Complex-type fidelity:** #64976 proposed preserving array child nullability and #65044 added focused IPC tests, but both closed without merge. #65182 documents additional Arrow Map null-key and raw BITMAP incompatibilities by skipping those regression cases. Decide which cases require engine fixes and which require documented protocol limitations. - [ ] **Flight SQL JDBC `RST_STREAM` failures on 4.0.1:** #59304 was closed as stale without a linked fix. Reproduce against current supported versions or link the resolving change. - [ ] **Flight SQL with the distributed planner disabled:** #57318 was closed as stale. #63136 appears to address the same empty-result-receiver failure on master and #63155 backports it to 4.1, but the 4.0 backport #63154 closed without merge. Verify the version matrix and close the loop explicitly. - [ ] **Sessions remain in `SHOW PROCESSLIST` after an ADBC connection closes:** #36331 was closed as stale. #55008/#55396 improve timeout cleanup and #65040 keeps the timeout checker alive, but explicit client-close behavior still needs a current-version verification. ### Authentication, endpoint advertisement, and session semantics - [ ] **Duplicate authentication reports:** #61743, #61744, and #61757 all report authentication using `[email protected]`. Verify the merged fix #63506 and backports #63591/#63592 against the reported read-only user case, then close or deduplicate the three issues. - [ ] **FE cannot advertise a public Flight endpoint:** #62538 requests FE equivalents of BE `public_host` and the Flight proxy port. - [ ] **FE/BE advertises `127.0.0.1:8050` to a remote JDBC client:** #59490 was closed as stale. Check whether it is the same endpoint-advertisement gap as #62538. - [ ] **Kubernetes and reverse-proxy topology:** #44599 is an older, stale-closed precursor to #62217/#62538. Decide whether the remaining work belongs in Doris, Doris Operator, or both, and cross-link the owner issue. - [ ] **Doris Catalog session-variable passthrough in Arrow Flight mode:** #59742. ### Protocol coverage and architecture - [ ] **Variant over cross-cluster Arrow Flight:** #61883. The discussion proposes a LargeUtf8 first step and leaves a more efficient representation as follow-up. - [ ] **Prepared-statement parameter binding:** [apache/arrow-java#1009](https://github.com/apache/arrow-java/issues/1009) was closed upstream as a Doris server-side implementation gap. Create or link a Doris issue that defines the required Flight SQL parameter schema and execution behavior. - [ ] **Initial database/schema in a Flight SQL JDBC URL:** [apache/arrow-java#829](https://github.com/apache/arrow-java/issues/829) remains open. Determine client-versus-server ownership and document the supported connection option. - [ ] **ADBC ingestion:** users requested `adbc_ingest`/DataFrame writes in #25514, but no dedicated Doris work item tracks scope, protocol methods, atomicity, or type coverage. Create one if ingestion remains on the roadmap. - [ ] **Protocol SPI extraction:** #60355. - [ ] **Optional BE Arrow Flight worker pool:** #60388 adds an `enable_arrow_flight` switch so unused deployments do not allocate the pool. ### Tests and ecosystem - [ ] **Arrow regression cleanup:** #62536 remains open. - [ ] **Arrow JDBC close-time allocator leak:** #64024 changed the authentication regression test to avoid the prepared path, and #64677 explicitly says the intermittent close-time leak needs a separate fix. Create or link that follow-up and check whether newer Arrow/Netty versions resolve it. - [ ] **Spark column projection:** [apache/doris-spark-connector#333](https://github.com/apache/doris-spark-connector/issues/333) reports that `doris.read.fields` is ignored in Arrow read mode. - [ ] **Spark array work cleanup:** [apache/doris-spark-connector#345](https://github.com/apache/doris-spark-connector/pull/345) remains open although the narrower opt-in implementation [#362](https://github.com/apache/doris-spark-connector/pull/362) merged. Mark the older PR superseded or document the remaining element-type work. ## Completed or merged during the review window ### Core feature and query path - [x] Doris Catalog and cross-cluster query over Arrow Flight: #55274 and #57898. - [x] Connection cleanup and thread-local correctness: #55008 with 3.1 backport #55396, plus #55809. - [x] Non-destructive Flight ticket/query-id parsing: #57240 with 4.0 backport #57330. - [x] Flight processor close ordering and noisy warning fix: #57858 with 4.0 backport #58011, closing #57859. - [x] Queries containing `&` no longer produce malformed tickets: #58830 with 4.0 backport #58884. - [x] Async result completion avoids dirty reads: #60282 with 4.0 backport #60325. - [x] Remote Flight SQL result receiver initialization: #63136 with 4.1 backport #63155. - [x] Arrow UTF8/String size limit corrected to the intended 2 GiB threshold: #63137 with backports #63148 and #63558. - [x] Timeout checking continues after one connection throws: #65040. ### Authentication, diagnostics, and crash hardening - [x] Better malformed-ticket logging: #58811 with 4.0 backport #58842. - [x] Workload-group values are populated in audit records for early query failures: #62651 with 4.1 backport #62697. - [x] Arrow status initialization crash fix: #63191 with backports #63266/#63267. The build-mode root cause was also addressed by #64390 and documented upstream in [apache/arrow#49781](https://github.com/apache/arrow/issues/49781). - [x] Real client IP is used for Flight authentication: #63506 with backports #63591/#63592. - [x] External split-fetch failures now fail gracefully instead of crashing the BE: #64797 with 4.1 backport #64890. This is only the robustness half of #62259; the lifecycle item remains open above. ### Regression framework and ecosystem - [x] Remote-IP auth regression uses a stable JDBC statement path: #64024 with backports #64047/#64049. - [x] Regression framework Arrow/Netty versions aligned with FE and JDK 17: #64677. - [x] JDBC complex values are materialized before result-set teardown: #65084. - [x] Regression cases now account for Arrow-incompatible Map null keys and protocol-specific BITMAP bytes: #65182. - [x] Spark connector FE/BE load balancing: [apache/doris-spark-connector#329](https://github.com/apache/doris-spark-connector/pull/329). - [x] Spark connector Flight read logging: [apache/doris-spark-connector#354](https://github.com/apache/doris-spark-connector/pull/354). - [x] Spark connector opt-in native array reads: [apache/doris-spark-connector#362](https://github.com/apache/doris-spark-connector/pull/362), closing [#341](https://github.com/apache/doris-spark-connector/issues/341). - [x] Flink connector BackendClient typo that could affect Flight reads: [apache/doris-flink-connector#622](https://github.com/apache/doris-flink-connector/pull/622). - [x] Website corrected the Flink connector `source.use-flight-sql` default for current/4.x docs: [apache/doris-website#3725](https://github.com/apache/doris-website/pull/3725). ## Closed without merge or superseded attempts Keep these for history so the same investigation is not repeated: - Prepared-statement resource release: #52245. - Query profile and connection finalization: #45416. - Early Arrow regression experiments: #53284, #53285, and #53286. - Alternative generic Arrow Flight Catalog implementation: #59270. The merged Doris Catalog path is #55274/#57898. - Logging experiment: #61990. - DDL regression fix: #62018. - 4.0 result-receiver backport: #63154. - Array child nullability and focused IPC tests: #64976 and #65044. ## Maintenance rules - Add each new issue or PR under the closest topic and link it back to this tracker. - Keep the primary fix and all required release-branch backports on the same line. - Check an item only after the fix is merged and the affected supported versions are verified. - If a stale-closed report still reproduces, reopen it or create a replacement and link both directions. - Record user-visible protocol limitations in the Arrow Flight SQL documentation, not only in regression-test skips. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
