Doris-Breakwater commented on issue #70614: URL: https://github.com/apache/doris/issues/70614#issuecomment-6909783994
Initial triage: **the supplied stacks strongly support BE HTTP event-loop starvation caused by synchronous Stream Load callbacks. There is a known fix for the completion-wait component, but I cannot confirm a complete patch for both reported blocking paths or recommend a generic upgrade to 3.x as the solution.** The issue currently has no labels; suggested classification is BE / loading / availability. This is source-based analysis, not a reproduced failure, and the vendor build still needs to be matched to its exact source revision. **Verified code behavior and related fixes** - In the public `2.1.11-rc01` source, `EvHttpServer::start()` dispatches one libevent loop per HTTP worker. Stream Load, `/api/health`, and `/metrics` register on the same server. Blocking all workers explains an unresponsive HTTP port while the separate heartbeat service remains available. See [HTTP workers](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/http/ev_http_server.cpp#L111-L139) and [endpoint registration](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/service/http_service.cpp). - `StreamLoadAction::_handle()` calls `body_sink->finish()` **before** waiting on `ctx->future.get()`. `on_chunk_data()` synchronously appends to the pipe. The pipe has a default 4 MiB buffering threshold; readers wait for data/finish/cancellation, and writers wait for capacity/cancellation. These waits have no local deadline. `StreamLoadExecutor` fulfills the promise from the fragment completion callback. See [Stream Load](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/http/action/stream_load.cpp#L154-L190) and [pipe implementation](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/io/fs/stream_load_pipe.cpp#L199-L252). - [#53139](https://github.com/apache/doris/issues/53139) describes the same completion-wait problem, including Group Commit sync mode. Its fix, [PR #53144](https://github.com/apache/doris/pull/53144), merged on January 14, 2026 as `3ec56d7a796b9361d67203fcd9b7e2a32bbbeab4`, moves completion/commit/reply work into a callback and changes HTTP request lifetime handling. This is a relevant patch to evaluate with the vendor, not a one-line `future.get()` replacement. The older attempt [#36772](https://github.com/apache/doris/pull/36772) was reverted by [#36798](https://github.com/apache/doris/pull/36798); do not use that earlier patch alone. - **3.x / upgrade status:** I checked the public `3.0.8-rc02` and `3.1.4-rc02` source tags and the live `branch-3.0` and `branch-3.1` files: `_handle()` still waits on `ctx->future.get()`, and body append remains synchronous. The async-completion change is present on master and was backported to `branch-4.2` in [PR #70171](https://github.com/apache/doris/pull/70171), merged November 5, 2026. A merged branch backport does **not** establish the first released binary containing it; release/build inclusion must be verified separately. - **Remaining coverage gap:** the current master and `branch-4.2` HTTP callbacks still call blocking body append. [#68973](https://github.com/apache/doris/issues/68973) remains open and tracks full-pipe and header-stage event-loop blocking. Therefore #53144/#70171 address an important part of this report, but are not evidence of a complete fix for the 32 full-pipe waiters. **What the current evidence does not yet prove** The proposed cycle needs to be expressed across requests: a request already inside `future.get()` has finished its own input pipe. It may wait for execution/scanner capacity while other requests' scanners wait for bodies whose HTTP loops are blocked. Full and empty pipe stacks alone do not establish which requests depend on which others. Mapping load/query IDs, pipes, scanner queues, and loop assignments would establish that dependency. There is also an independent escape path in the reference source: `FragmentMgr::cancel_worker()` checks fragment deadlines; pipeline cancellation looks up the Stream Load pipe and calls `cancel()`, which wakes both readers and writers. See [timeout processing](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/runtime/fragment_mgr.cpp#L1302-L1524) and [pipeline cancellation](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp#L138-L187). A blocked HTTP loop delays its own disconnect processing, but does not by itself disable this background cancellation. The reported 43-hour persistence needs investigation of the effective deadlines, cancel-worker stack, and whether cancellation reaches the affected pipes and completes the fragments. The default scanner configuration also deserves correction: public 2.1.11 resolves `doris_scanner_thread_pool_thread_num=-1` to `max(48, 2 × CPU cores)`, already at least the default 48 HTTP workers. Remote and workload-group scanner limits can differ. Making the configured scanner count greater than the HTTP worker count is not a general deadlock-prevention guarantee; neither “request 33” nor concurrency above 48 is a universal trigger. The aggregate Publish:Compaction ratio, zero-row fraction, and waiting-thread percentage do not establish the initial bottleneck. Compactions can merge multiple versions, so comparable per-tablet version/backlog measurements are needed. Similar four-hour durations do not imply identical start/end times: the reference FE configuration has `insert_load_default_timeout_second=14400`, which is a candidate to correlate with the actual LOAD records, not proof of the timeout responsible here. **Minimum additional material** Please attach compressed text files or link a sanitized gist, with timestamps and a consistent timezone: 1. Full FE/BE build strings and vendor commit/patch identifiers; effective BE/FE configuration, CPU count, table DDL, workload-group scanner limits, and whether Pipeline X is enabled. 2. A minimal client script and sanitized request headers: actual `group_commit` mode, `timeout`, format, Content-Length versus chunked transfer, batch sizes, concurrency **after FE redirects**, connection reuse, and retry policy. Table Group Commit interval/size properties alone do not identify the request mode. 3. Two or three complete symbolized process dumps 30–60 seconds apart before restart, including the fragment cancellation worker and scanner pool names. A symbolized core, if feasible, would help map blocked contexts/pipe IDs and their cancellation state. 4. FE and affected BE INFO/WARNING logs around the onset and expected timeout boundary, retaining load labels/query IDs/transaction IDs consistently, plus client HTTP outcomes. Include cancellation messages, effective timeout values, scanner active/queued counts, WAL replay progress, and representative per-tablet version/compaction backlog trends. These are more useful initially than uncorrelated 20-hour totals. **Next steps:** bound total per-BE in-flight loads and retries at the client/gateway; the proposed limit of 24 is a starting experiment, not a guaranteed safe threshold. Increasing worker pools may provide headroom but cannot remove blocking callbacks. During an existing stall, stop new admissions, capture evidence, then use a controlled restart if necessary; retain WAL and reconcile uncertain load outcomes before retrying. For a permanent fix, evaluate the async-completion patch together with nonblocking per-connection body backpressure, bounded admission, and cancellation/request-lifetime handling. Validate both completed small bodies waiting for scanners and full pipes, while checking health/metrics responsiveness, timeout/disconnect behavior, WAL replay, and data correctness. Adding `wait_for()` alone does not cover append blocking. I suggest keeping this issue open, linked to #53139/#53144 and #68973, until the vendor revision, cancellation failure, and coverage of both blocking paths are established. Breakwater-GitHub-Analysis-Slot: slot_ad312e285724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
