github-actions[bot] commented on code in PR #69421:
URL: https://github.com/apache/doris/pull/69421#discussion_r4639054453
##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -246,6 +246,10 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env,
return Status::Aborted("Missing vital param: table_name");
}
+ // TODO: Balance this reservation when setup fails before a callback
exists
Review Comment:
[P1] Balance the in-flight reservation on plan setup failure
The increment immediately before exec_plan_fragment is not paired when setup
returns an error before the callback is installed. The caller then decrements
only the initial consume reservation, leaving _inflight_cnt nonzero;
_handle_consumer_finished() is never reached and load_status_future.get() can
wait forever for a multi-table routine load. Use a scope guard or equivalent
failure path to decrement this plan and remove/cancel its child context before
returning.
##########
be/src/load/group_commit/group_commit_mgr.cpp:
##########
@@ -505,13 +505,13 @@ Status GroupCommitTable::_create_group_commit_load(
} else {
LOG(WARNING) << "_exec_env->cluster_info not set backend_id";
}
- TNetworkAddress master_addr =
_exec_env->cluster_info()->master_fe_addr;
- st = ThriftRpcHelper::rpc<FrontendServiceClient>(
- master_addr.hostname, master_addr.port,
+ st = ThriftRpcHelper::rpc_fe_with_master_refresh(
Review Comment:
[P1] Keep group-commit completion on the elected master
Only the plan request is routed through rpc_fe_with_master_refresh;
_finish_group_commit_load still sends commit and rollback to the snapshotted
master_addr with the old one-shot helper. A master election after the plan
succeeds can therefore reject or disconnect these completion RPCs without
refreshing the address, leaving the admitted transaction unresolved. Route both
completion phases through the refresh helper with the stable transaction id.
##########
be/src/util/thrift_rpc_helper.cpp:
##########
@@ -23,14 +23,17 @@
#include <thrift/protocol/TBinaryProtocol.h>
#include <thrift/transport/TTransportException.h>
// IWYU pragma: no_include <bits/chrono.h>
+#include <algorithm>
#include <chrono> // IWYU pragma: keep
#include <sstream>
#include <thread>
#include "common/status.h"
+#include "runtime/cluster_info.h"
#include "runtime/exec_env.h" // IWYU pragma: keep
Review Comment:
[P1] Enable master-refresh retries when BE shutdown starts
rpc_fe_with_master_refresh selects the long retry/DNS-refresh path from
cluster_info()->in_graceful_shutdown(), but the production shutdown path only
sets k_doris_exit and never calls set_in_graceful_shutdown(true). The flag
therefore stays false for every real SIGTERM, so a transport failure while the
FE is being stopped returns from the legacy RPC immediately and the load can
fail during the exact election window this retry budget is intended to cover.
Set the flag as part of the shutdown transition (or derive this mode from
k_doris_exit).
##########
be/src/util/thrift_rpc_helper.cpp:
##########
@@ -23,14 +23,17 @@
#include <thrift/protocol/TBinaryProtocol.h>
#include <thrift/transport/TTransportException.h>
// IWYU pragma: no_include <bits/chrono.h>
+#include <algorithm>
#include <chrono> // IWYU pragma: keep
#include <sstream>
#include <thread>
#include "common/status.h"
+#include "runtime/cluster_info.h"
#include "runtime/exec_env.h" // IWYU pragma: keep
#include "util/client_cache.h"
#include "util/debug_points.h"
+#include "util/dns_cache.h"
#include "util/network_util.h"
Review Comment:
[P1] Retry healthy NOT_MASTER responses during FE election
When the old master is still reachable it can return NOT_MASTER during an
election. Because this branch only falls through for FE_SHUTTING_DOWN, the
helper returns transport OK with the NOT_MASTER result and the Stream Load
caller aborts; _retry_fe_rpc's address refresh is never used. Route NOT_MASTER
through the same bounded retry loop so a heartbeat with the new master can
recover the request.
##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -76,6 +79,8 @@
import java.time.LocalDate;
import java.util.List;
import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
Review Comment:
[P1] Fence HTTP and ordinary Thrift admission before declaring quiescence
This fence stops only the MySQL listener and the two explicit counters. HTTP
and the general FE Thrift listener stay open until after
waitForGracefulShutdown, while their planning/stream-load handlers are not
counted. A request accepted after this point can run with no active-request
counter, transaction-RPC counter, or Coordinator, so the loop can exit and the
subsequent listener shutdown can cut off its response or load. Stop/fence those
listeners or account for every accepted handler before the final zero check.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/QueryAdmissionController.java:
##########
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.base.Preconditions;
+
+/** Admission and accounting for client requests during FE graceful shutdown.
*/
+public class QueryAdmissionController {
+ public static final QueryAdmissionController INSTANCE = new
QueryAdmissionController();
+
+ private boolean accepting = true;
+ private int activeRequests;
+
+ // Admission and the shutdown transition share a lock. A request either
contributes
+ // to the drain count before shutdown, or is refused before it starts
planning.
+ public synchronized boolean tryAcquire() {
+ if (!accepting) {
+ return false;
+ }
+ activeRequests++;
+ return true;
+ }
+
+ public synchronized void release() {
+ Preconditions.checkState(activeRequests > 0, "no admitted request to
release");
+ activeRequests--;
+ }
+
+ // Result retrieval and session cleanup must remain possible after
admission closes.
Review Comment:
[P1] Fence Flight continuations with the final drain snapshot
acquireContinuation() still increments after stopAccepting(), and the drain
loop only performs a non-atomic final read of activeRequests before breaking. A
DoGet/CloseSession can arrive in that gap, perform teardown or stream a ticket,
and then be cut off when qeService.stop() closes Flight. Coordinate
continuation admission with the final fence (or close/fence Flight before the
zero check) so no continuation can begin after quiescence is declared.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -1458,6 +1458,17 @@ private void checkPassword(String user, String passwd,
String clientIp)
@Override
public TLoadTxnBeginResult loadTxnBegin(TLoadTxnBeginRequest request)
throws TException {
+ try (TxnRpcAdmissionController.Guard guard =
TxnRpcAdmissionController.INSTANCE.tryEnter()) {
Review Comment:
[P1] Admit multi-table planning into the shutdown fence
streamLoadMultiTablePut is the endpoint used by
MultiTablePipe::request_and_exec_plans, but only streamLoadPut and the
transaction commit/rollback methods use TxnRpcAdmissionController. Once
stopAcceptingIfIdle() observes zero guarded RPCs, this uncounted planner can
still start, update routine-load state, and lose its response when the Thrift
server stops. Wrap it in the same guard and return FE_SHUTTING_DOWN before any
plan work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]