morningman opened a new pull request, #66797:
URL: https://github.com/apache/doris/pull/66797
### What problem does this PR solve?
Issue Number: close #66796
Problem Summary:
A graceful BE shutdown could hang for up to 10 minutes and fail the
pipeline's
"stop grace" check.
`doris_main()` stops the servers first (`Http service stopped` -> `Brpc
service
stopped` -> `Backend Service stopped`) and only then calls
`ExecEnv::destroy()`,
which reaches `SAFE_STOP(_stream_load_recorder_manager)` and joins the
recorder's
worker `std::thread`. If that worker is inside `_send_stream_load()` at the
time,
its request is a stream load against this BE's own http endpoint, which has
already been torn down, so the load can never complete and the request only
returns once the `DEFAULT_STREAM_LOAD_TIMEOUT_SEC = 600` curl timeout
expires.
The join blocks for that long. Deployment scripts give graceful stop a 10
minute
budget, so whenever the race fires the stop check is guaranteed to lose it
and
the BE gets `kill -ABRT`ed.
The window is narrow (an audit batch has to be in flight within about one
worker
iteration of the service teardown), which is why it fired only once in the
last
30 NonConcurrentRegression runs, but any BE with a pending audit-log batch at
stop time can hang this way.
The PR applies three changes, each of which bounds the wait on its own:
1. **Ordering** — `doris_main()` stops the recorder manager *before* tearing
down
the http service, so on the normal path the last audit batch is flushed
against a live server and the worker is already gone by the time
`ExecEnv::destroy()` runs. `stop()` is idempotent, so the existing
`SAFE_STOP()` in `destroy()` becomes a no-op.
2. **Interruptible send** — new `HttpClient::set_abort_callback()` registers
a
functor that libcurl polls through `CURLOPT_XFERINFOFUNCTION`, roughly
once
per second while the connection is idle. Returning true aborts the
transfer
with `CURLE_ABORTED_BY_CALLBACK` instead of running to
`CURLOPT_TIMEOUT_MS`.
`StreamLoadRecorderManager` hooks its `_stop` flag up to it, which covers
both
a request that is already in flight when `stop()` is called and one that
races its way past the check in 3.
3. **No new work after stop** — the worker no longer starts an audit load
once
shutdown has begun, and waits on a condition variable instead of an
unconditional 1s sleep, so `stop()` wakes it immediately rather than
after up
to a second.
`stop()` now also logs on completion. The original hang was hard to locate in
the log precisely because a raw `std::thread::join()` is silent, unlike doris
`Thread::join`, which prints `Waited for ...ms trying to join`.
The 600s curl timeout itself is left alone: it is a sane bound for a load
that
is not racing shutdown, and the abort hook makes it irrelevant for shutdown.
### Release note
Fix a graceful BE shutdown that could hang for up to 10 minutes when an audit
log stream load was in flight while the BE's http service was being stopped.
### Check List (For Author)
- Test
- [x] Unit Test
`HttpClientTest.abort_in_flight_request` points an `HttpClient` at a
listening
socket that is never accepted from — the kernel completes the handshake and
buffers the request, but no response ever comes, which is exactly what an
audit
stream load looks like once the http service serving it is gone. With
`CURLOPT_TIMEOUT_MS` at 60s, the abort callback is raised 300ms in and the
request has to end well before the timeout.
Verified separately against the libcurl the BE links (8.2.1) that the
progress
callback is polled while idle-waiting: the request ends in ~1.2s with
`CURLE_ABORTED_BY_CALLBACK` (callback invoked 13 times) rather than at 60s.
The shutdown ordering itself is not unit-testable — it needs a full BE stop
with an audit batch in flight, which is the race described in the issue.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
