morningman opened a new issue, #66796: URL: https://github.com/apache/doris/issues/66796
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master (observed on commit `0b3406309dc`-based CI run of #66789, 2026-08-15; the involved code paths are unchanged on current master and the bug is independent of that PR's changes) ### What's Wrong? A graceful BE shutdown (`stop_be.sh --grace`) can hang for up to 10 minutes and then fail the pipeline's "stop grace" check. The hang is a shutdown-ordering problem around `StreamLoadRecorderManager`: 1. `doris_main` stops the servers first — at `16:45:10-15` the log shows `Http service stopped` -> `Brpc service stopped` -> `Backend Service stopped` — and only then calls `ExecEnv::destroy()`. 2. `destroy()` reaches `SAFE_STOP(_stream_load_recorder_manager)`. `StreamLoadRecorderManager::stop()` sets the (atomic) `_stop` flag and `join()`s its worker `std::thread`. 3. The worker polls `_stop` once per second, but each iteration may call `_load_if_necessary()` -> `_send_stream_load()`, which PUTs the buffered audit records to the FE as an HTTP stream load with `client.set_timeout_ms(DEFAULT_STREAM_LOAD_TIMEOUT_SEC * 1000)` where `DEFAULT_STREAM_LOAD_TIMEOUT_SEC = 600`. 4. If such a load is in flight (or starts) in the window after the BE's own HTTP server has already been torn down, the load can never complete — the redirect target of the stream load is this BE's http endpoint — so the request only returns when the **600 s curl timeout** expires. 5. `stop()`'s `join()` therefore blocks for up to 600 s. The regression pipeline's grace budget is `timeout 10m` and starts earlier, so the stop check always loses the race: `Stop grace fail`, and the pipeline `kill -ABRT`s the BE. Evidence from the failing run (NonConcurrentRegression build, PR #66789 round; 474/474 test cases had passed, the only failure was the stop): - ABRT-time stack of the main thread: ``` 2# __pthread_clockjoin_ex 3# __interceptor_pthread_join 4# std::thread::join() 5# doris::ExecEnv::destroy() 6# main ``` - `be.INFO`: main thread's last destroy-progress line is `16:45:15.813 GroupCommitMgr is stopped`, then silence for 10 minutes. In `ExecEnv::destroy()`'s order, the next steps after that stop are `_routine_load_task_executor` (ThreadPool-based; a stuck doris `Thread::join` prints `Waited for ...ms trying to join` warnings every second — none appear) and then `_stream_load_recorder_manager`, which is the only object in that stretch owning a raw `std::thread` (raw `std::thread::join` is silent, matching the log). - The recorder worker was active during the run (`15:47:52 Failed to load stream load records to audit log table: ...`), and there is no completion/failure log for its last send after `16:45` — the final request never returned before the ABRT. The window is narrow (an audit batch must be in flight within roughly one worker iteration of the service teardown), which is why CI rarely hits it — in the last 30 NonConcurrentRegression runs it fired once. But any BE with a pending audit-log batch at stop time can hang its graceful shutdown this way. ### What You Expected? Graceful shutdown completes within seconds regardless of whether an audit-table stream load is in flight. ### How to Reproduce? Race-dependent; the deterministic recipe is: 1. Enable the stream load audit recorder and run enough stream loads that the recorder buffers a batch. 2. Arrange for `_send_stream_load()` to be in flight when `stop_be.sh --grace` runs (e.g. pause the FE side of the audit load, or inject a sync point between the HTTP-server stop and `ExecEnv::destroy()`). 3. Observe the BE blocked in `ExecEnv::destroy -> std::thread::join` for up to `DEFAULT_STREAM_LOAD_TIMEOUT_SEC` (600 s). ### Anything Else? Possible fixes, any one of which suffices (they compose too): 1. **Ordering**: stop `StreamLoadRecorderManager` (and other workers that depend on live FE/BE services) *before* tearing down the HTTP/brpc servers, instead of inside `ExecEnv::destroy()` after them. 2. **Interruptible send**: give the shutdown path a way to abort the in-flight `HttpClient` request (or check `_stop` and use a short timeout, e.g. a few seconds, once shutdown has begun). 3. **Bounded join**: make `stop()` cap how long it waits for the worker and detach/abandon the final request past the cap, so graceful stop stays bounded even if the request hangs. The 600 s request timeout being exactly equal to the common 10-minute grace budget of deployment scripts makes this failure mode particularly unlucky: whenever the race fires, the grace check is guaranteed to fail. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
