raghav-reglobe opened a new issue, #62259: URL: https://github.com/apache/doris/issues/62259
## Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ## Version 4.1.0-rc01 (built from source, tag `apache-iceberg-4.1.0-rc01`, ARM64) ## What's Wrong Arrow Flight SQL queries against Iceberg external catalog tables crash the BE with SIGSEGV. MySQL protocol works perfectly for the same queries. **Root cause**: FE releases the Iceberg `SplitSource` after `GetFlightInfo` returns, before BE's `DoGet` calls `fetchSplitBatch`. Arrow Flight's 2-phase execution (`GetFlightInfo` → `DoGet`) doesn't align with the SplitSource lifecycle. MySQL protocol is unaffected because plan + execute share one coordinator lifecycle. ## Reproduction 1. Set up Iceberg REST catalog (Polaris) with a Silver SCD2 table (~775K rows, ~48 columns) 2. Query via Arrow Flight SQL (ADBC Python driver): ```python import adbc_driver_flightsql.dbapi as flight_sql conn = flight_sql.connect("grpc://doris-fe:8040", db_kwargs={"username": "root", "password": ""}) cursor = conn.cursor() cursor.execute("SELECT * FROM silver.cashify_mart.orders LIMIT 100") table = cursor.fetch_arrow_table() # → BE SIGSEGV ``` 3. Same query via MySQL protocol: works fine (2.2s) 4. `SELECT 1` via Arrow Flight: works fine (internal table, no SplitSource) ## FE Error Log ``` Caused by: org.apache.thrift.TException: Split source 0 is released at org.apache.doris.service.FrontendServiceImpl.fetchSplitBatch(FrontendServiceImpl.java:1009) RuntimeLogger ERROR (thrift-server-pool-2) [ProcessFunction.process():47] Internal error processing fetchSplitBatch java.lang.reflect.UndeclaredThrowableException at org.apache.doris.thrift.FrontendService$Processor$fetchSplitBatch.getResult(FrontendService.java:5668) Caused by: org.apache.thrift.TException: Split source 0 is released at org.apache.doris.service.FrontendServiceImpl.fetchSplitBatch(FrontendServiceImpl.java:1009) at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:592) ``` ## BE Crash Stack Trace ``` 0# doris::signal::FailureSignalHandler(int, siginfo_t*, void*) 4# std::__cxx11::basic_string constructor at basic_string.h:707 5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&) 6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*) 7# GrpcServiceHandler::DoGet(grpc::ServerContext*, Ticket const*, ServerWriter<FlightData>*) ``` The SIGSEGV is a secondary bug: `TransportStatus::FromStatus` crashes on a null/invalid error string when converting the `fetchSplitBatch` IO_ERROR to a gRPC status. Even without the crash, the query would fail with "Split source 0 is released". ## Analysis The batch split source mechanism (PR #35107, #46398) creates an async `SplitSource` in FE during query planning. BE fetches splits via Thrift RPC `fetchSplitBatch`. The lifecycle issue: 1. `GetFlightInfo` → FE plans query, creates SplitSource, returns FlightInfo 2. `Coordinator.close()` is called after GetFlightInfo returns → releases SplitSource 3. `DoGet` → BE starts scan, calls `fetchSplitBatch` → "Split source 0 is released" In MySQL protocol, plan + execute happen in one request, so the coordinator (and SplitSource) stays alive throughout. In Arrow Flight, the 2-phase execution means the coordinator is closed between phases. ## Workaround Setting extremely high batch mode threshold forces the old `getSplits()` path (all splits materialized during planning, before GetFlightInfo returns): ```sql SET GLOBAL num_partitions_in_batch_mode = 999999999; SET GLOBAL num_files_in_batch_mode = 999999999; ``` This bypasses the async SplitSource entirely. Verified working on 4.1.0-rc01 with 2 BEs. ## Suggested Fix **Option A (minimal, safe for RC patch)**: Detect Arrow Flight protocol and force non-batch mode: ```java if (connectContext.getProtocolType() == ProtocolType.ARROW_FLIGHT) { sessionVariable.setNumPartitionsInBatchMode(0); } ``` **Option B (proper)**: Defer `Coordinator.close()` / SplitSource cleanup until `DoGet` completes. Tie the coordinator lifecycle to the query ID, not the `GetFlightInfo` request. ## Environment - Doris: 4.1.0-rc01 (built from source, ARM64/Graviton4) - Iceberg: REST catalog (Apache Polaris 1.3.0), Iceberg 1.10.1 - Client: adbc-driver-flightsql 1.4.0 (Python) - K8s: EKS, 1 FE + 2 BEs cc @AshinGau (PR #35107 batch split source) @xinyiZzz (Arrow Flight SQL maintainer) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
