github-actions[bot] commented on code in PR #69422:
URL: https://github.com/apache/doris/pull/69422#discussion_r4640065880


##########
be/src/load/channel/load_channel_mgr.cpp:
##########
@@ -64,6 +64,13 @@ LoadChannelMgr::LoadChannelMgr() : 
_stop_background_threads_latch(1) {
     });
 }
 
+size_t LoadChannelMgr::num_active_channels() {
+    // Construction can release the last QueryContext and block on spill 
cleanup.
+    // A busy manager counts as pending work so shutdown can still check its 
deadline.
+    std::unique_lock<std::mutex> lock(_lock, std::try_to_lock);
+    return lock.owns_lock() ? _load_channels.size() : 1;
+}

Review Comment:
   [P1] Preserve an existing channel when a duplicate open fails
   
   `open()` reuses the `_load_channels[load_id]` entry whenever the load ID is 
already present, then calls `channel->open(params)` outside the manager lock. 
The new error branch calls `_cancel_load_channel(channel, status)` for every 
failure, so a malformed retry (for example cloud `txn_expiration <= 0`) or a 
second open that fails in `BaseTabletsChannel::open()` cancels and tombstones 
the already-valid channel created by the first request. The old path returned 
this error while keeping that channel usable. Track whether this call inserted 
the channel and only cancel that exact newly-created instance (and add a 
duplicate-open regression test).



##########
be/src/runtime/exec_env.cpp:
##########
@@ -205,20 +208,38 @@ std::map<TNetworkAddress, FrontendInfo> 
ExecEnv::get_running_frontends() {
     return res;
 }
 
+void ExecEnv::wait_for_shutdown_heartbeat() {
+    int32_t elapsed = 0;
+    while (!k_shutdown_heartbeat_sent && elapsed < 
config::grace_shutdown_heartbeat_wait_seconds) {
+        sleep(1);
+        ++elapsed;

Review Comment:
   [P1] Include legacy load streams in the shutdown census
   
   `wait_for_all_tasks_done()` now checks only query, `LoadChannelMgr`, and 
`StreamLoadContext` counts. A `PInternalService::open_load_stream()` request 
instead owns a `LoadStream` in `LoadStreamMgr::_load_streams_map`; it is not a 
`StreamLoadContext`, and `LoadStream::on_closed()` is what eventually erases 
it. With an accepted stream in flight, the three new counts can all be zero, so 
shutdown breaks out and later destroys `_load_stream_mgr` while the load is 
still writing. Add a nonblocking/atomic active-stream count (with partial-open 
cleanup) to this predicate before claiming all loads are drained.



##########
be/src/runtime/fragment_mgr.cpp:
##########
@@ -413,9 +412,20 @@ Status FragmentMgr::exec_plan_fragment(const 
TPipelineFragmentParams& params,
         RETURN_IF_ERROR(
                 _exec_env->new_load_stream_mgr()->put(stream_load_ctx->id, 
stream_load_ctx));
 
-        RETURN_IF_ERROR(
-                
_exec_env->stream_load_executor()->execute_plan_fragment(stream_load_ctx, 
parent));
-        return Status::OK();

Review Comment:
   [P1] Scope the late cleanup callback to its original context
   
   The new failure path removes the failed context with `remove(id, 
stream_load_ctx.get())`, allowing a retry to register the same ID. However, 
`StreamLoadExecutor`'s captured completion callback still calls 
`remove(ctx->id)` without an expected pointer. 
`PipelineFragmentContext::submit()` can fail after some tasks were submitted, 
leaving the old PFC to invoke that callback later during asynchronous teardown; 
it can then erase the retry's context and leave the retry without its 
pipe/cancellation owner. Pass `ctx.get()` through the callback removal (or 
otherwise defer cleanup until the old PFC cannot run).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to