github-actions[bot] commented on code in PR #69422:
URL: https://github.com/apache/doris/pull/69422#discussion_r4345147666
##########
be/src/runtime/exec_env.cpp:
##########
@@ -191,20 +194,41 @@ std::map<TNetworkAddress, FrontendInfo>
ExecEnv::get_running_frontends() {
return res;
}
+void ExecEnv::wait_for_shutdown_heartbeat() {
+ int32_t elapsed = 0;
+ while (!k_shutdown_heartbeat_sent && elapsed <
config::grace_shutdown_heartbeat_wait_seconds) {
+ sleep(1);
+ ++elapsed;
+ }
+ LOG(INFO) << "shutdown heartbeat prepared: " <<
k_shutdown_heartbeat_sent.load() << ", waited "
+ << elapsed << " seconds";
+ // The response may still be in transit. This buffer also gives followers
time
+ // to replay shutdown state. It does not prove that every FE has applied
it.
+ sleep(std::max(0, config::grace_shutdown_heartbeat_propagation_seconds));
+}
+
void ExecEnv::wait_for_all_tasks_done() {
- // For graceful shutdown, need to wait for all running queries to stop
+ // Both receiver implementations and the load coordinator must finish. For
example,
+ // an HTTP upload or transaction commit can outlive all of its pipeline
fragments.
int32_t wait_seconds_passed = 0;
while (true) {
int num_queries = _fragment_mgr->running_query_num();
- if (num_queries < 1) {
+ size_t num_channels = _load_channel_mgr->num_active_channels();
+ size_t num_streams = _load_stream_mgr->get_load_stream_num();
Review Comment:
[P2] Clean up partially opened receivers before counting them as active
`LoadStreamMgr` can retain a receiver after every actual stream has closed.
For example, with `total_streams=2`, let the first open succeed and the second
be rejected by the receiver's full heavy-work queue before registration
(`PInternalService::open_load_stream`). The sender cancels its accepted
streams, but `LoadStream::on_closed()` only runs once and waits for the planned
total of two before calling `clear_load()`. There is no manager sweeper or
other cancellation path to erase this entry, so `get_load_stream_num()` remains
positive indefinitely. Adding it to this shutdown predicate therefore makes a
later idle shutdown consume all of `grace_shutdown_wait_seconds` (120 seconds
by default) for a failed load that cannot progress. Handle partial-open/aborted
receiver cleanup before relying on this census, and cover an expected stream
that never opens. The new transactional-context cleanup does not cover this
receiver path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
