Doris-Breakwater commented on issue #70614:
URL: https://github.com/apache/doris/issues/70614#issuecomment-6909783994

   Initial triage: **the supplied stacks strongly support BE HTTP event-loop 
starvation caused by synchronous Stream Load callbacks. There is a known fix 
for the completion-wait component, but I cannot confirm a complete patch for 
both reported blocking paths or recommend a generic upgrade to 3.x as the 
solution.** The issue currently has no labels; suggested classification is BE / 
loading / availability. This is source-based analysis, not a reproduced 
failure, and the vendor build still needs to be matched to its exact source 
revision.
   
   **Verified code behavior and related fixes**
   
   - In the public `2.1.11-rc01` source, `EvHttpServer::start()` dispatches one 
libevent loop per HTTP worker. Stream Load, `/api/health`, and `/metrics` 
register on the same server. Blocking all workers explains an unresponsive HTTP 
port while the separate heartbeat service remains available. See [HTTP 
workers](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/http/ev_http_server.cpp#L111-L139)
 and [endpoint 
registration](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/service/http_service.cpp).
   - `StreamLoadAction::_handle()` calls `body_sink->finish()` **before** 
waiting on `ctx->future.get()`. `on_chunk_data()` synchronously appends to the 
pipe. The pipe has a default 4 MiB buffering threshold; readers wait for 
data/finish/cancellation, and writers wait for capacity/cancellation. These 
waits have no local deadline. `StreamLoadExecutor` fulfills the promise from 
the fragment completion callback. See [Stream 
Load](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/http/action/stream_load.cpp#L154-L190)
 and [pipe 
implementation](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/io/fs/stream_load_pipe.cpp#L199-L252).
   - [#53139](https://github.com/apache/doris/issues/53139) describes the same 
completion-wait problem, including Group Commit sync mode. Its fix, [PR 
#53144](https://github.com/apache/doris/pull/53144), merged on January 14, 2026 
as `3ec56d7a796b9361d67203fcd9b7e2a32bbbeab4`, moves completion/commit/reply 
work into a callback and changes HTTP request lifetime handling. This is a 
relevant patch to evaluate with the vendor, not a one-line `future.get()` 
replacement. The older attempt 
[#36772](https://github.com/apache/doris/pull/36772) was reverted by 
[#36798](https://github.com/apache/doris/pull/36798); do not use that earlier 
patch alone.
   - **3.x / upgrade status:** I checked the public `3.0.8-rc02` and 
`3.1.4-rc02` source tags and the live `branch-3.0` and `branch-3.1` files: 
`_handle()` still waits on `ctx->future.get()`, and body append remains 
synchronous. The async-completion change is present on master and was 
backported to `branch-4.2` in [PR 
#70171](https://github.com/apache/doris/pull/70171), merged November 5, 2026. A 
merged branch backport does **not** establish the first released binary 
containing it; release/build inclusion must be verified separately.
   - **Remaining coverage gap:** the current master and `branch-4.2` HTTP 
callbacks still call blocking body append. 
[#68973](https://github.com/apache/doris/issues/68973) remains open and tracks 
full-pipe and header-stage event-loop blocking. Therefore #53144/#70171 address 
an important part of this report, but are not evidence of a complete fix for 
the 32 full-pipe waiters.
   
   **What the current evidence does not yet prove**
   
   The proposed cycle needs to be expressed across requests: a request already 
inside `future.get()` has finished its own input pipe. It may wait for 
execution/scanner capacity while other requests' scanners wait for bodies whose 
HTTP loops are blocked. Full and empty pipe stacks alone do not establish which 
requests depend on which others. Mapping load/query IDs, pipes, scanner queues, 
and loop assignments would establish that dependency.
   
   There is also an independent escape path in the reference source: 
`FragmentMgr::cancel_worker()` checks fragment deadlines; pipeline cancellation 
looks up the Stream Load pipe and calls `cancel()`, which wakes both readers 
and writers. See [timeout 
processing](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/runtime/fragment_mgr.cpp#L1302-L1524)
 and [pipeline 
cancellation](https://github.com/apache/doris/blob/2.1.11-rc01/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp#L138-L187).
 A blocked HTTP loop delays its own disconnect processing, but does not by 
itself disable this background cancellation. The reported 43-hour persistence 
needs investigation of the effective deadlines, cancel-worker stack, and 
whether cancellation reaches the affected pipes and completes the fragments.
   
   The default scanner configuration also deserves correction: public 2.1.11 
resolves `doris_scanner_thread_pool_thread_num=-1` to `max(48, 2 × CPU cores)`, 
already at least the default 48 HTTP workers. Remote and workload-group scanner 
limits can differ. Making the configured scanner count greater than the HTTP 
worker count is not a general deadlock-prevention guarantee; neither “request 
33” nor concurrency above 48 is a universal trigger.
   
   The aggregate Publish:Compaction ratio, zero-row fraction, and 
waiting-thread percentage do not establish the initial bottleneck. Compactions 
can merge multiple versions, so comparable per-tablet version/backlog 
measurements are needed. Similar four-hour durations do not imply identical 
start/end times: the reference FE configuration has 
`insert_load_default_timeout_second=14400`, which is a candidate to correlate 
with the actual LOAD records, not proof of the timeout responsible here.
   
   **Minimum additional material**
   
   Please attach compressed text files or link a sanitized gist, with 
timestamps and a consistent timezone:
   
   1. Full FE/BE build strings and vendor commit/patch identifiers; effective 
BE/FE configuration, CPU count, table DDL, workload-group scanner limits, and 
whether Pipeline X is enabled.
   2. A minimal client script and sanitized request headers: actual 
`group_commit` mode, `timeout`, format, Content-Length versus chunked transfer, 
batch sizes, concurrency **after FE redirects**, connection reuse, and retry 
policy. Table Group Commit interval/size properties alone do not identify the 
request mode.
   3. Two or three complete symbolized process dumps 30–60 seconds apart before 
restart, including the fragment cancellation worker and scanner pool names. A 
symbolized core, if feasible, would help map blocked contexts/pipe IDs and 
their cancellation state.
   4. FE and affected BE INFO/WARNING logs around the onset and expected 
timeout boundary, retaining load labels/query IDs/transaction IDs consistently, 
plus client HTTP outcomes. Include cancellation messages, effective timeout 
values, scanner active/queued counts, WAL replay progress, and representative 
per-tablet version/compaction backlog trends. These are more useful initially 
than uncorrelated 20-hour totals.
   
   **Next steps:** bound total per-BE in-flight loads and retries at the 
client/gateway; the proposed limit of 24 is a starting experiment, not a 
guaranteed safe threshold. Increasing worker pools may provide headroom but 
cannot remove blocking callbacks. During an existing stall, stop new 
admissions, capture evidence, then use a controlled restart if necessary; 
retain WAL and reconcile uncertain load outcomes before retrying. For a 
permanent fix, evaluate the async-completion patch together with nonblocking 
per-connection body backpressure, bounded admission, and 
cancellation/request-lifetime handling. Validate both completed small bodies 
waiting for scanners and full pipes, while checking health/metrics 
responsiveness, timeout/disconnect behavior, WAL replay, and data correctness. 
Adding `wait_for()` alone does not cover append blocking.
   
   I suggest keeping this issue open, linked to #53139/#53144 and #68973, until 
the vendor revision, cancellation failure, and coverage of both blocking paths 
are established.
   
   Breakwater-GitHub-Analysis-Slot: slot_ad312e285724
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to