github-actions[bot] commented on code in PR #69423:
URL: https://github.com/apache/doris/pull/69423#discussion_r4639182831
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -2166,6 +2210,17 @@ private boolean commitTxnImpl(TCommitTxnRequest request)
throws UserException {
@Override
public TLoadTxnRollbackResult loadTxnRollback(TLoadTxnRollbackRequest
request) throws TException {
+ try (TxnRpcAdmissionController.Guard guard =
TxnRpcAdmissionController.INSTANCE.tryEnter()) {
+ if (guard == null) {
+ TStatus status = new TStatus(TStatusCode.FE_SHUTTING_DOWN);
+ status.addToErrorMsgs("FE is shutting down; retry this RPC on
the current master");
+ return new TLoadTxnRollbackResult().setStatus(status);
+ }
+ return loadTxnRollbackWithAdmission(request);
+ }
+ }
+
+ private TLoadTxnRollbackResult
loadTxnRollbackWithAdmission(TLoadTxnRollbackRequest request) throws TException
{
String clientAddr = getClientAddrAsString();
if (LOG.isDebugEnabled()) {
LOG.debug("receive txn rollback request: {}, backend: {}",
Review Comment:
[P1] Guard multi-table Stream Load planning during FE drain:
streamLoadMultiTablePut() still calls generatePlan() without
TxnRpcAdmissionController.tryEnter(), while the six adjacent transaction RPCs
now use that fence. A request can start after stopAcceptingIfIdle() and remain
uncounted, then be interrupted by thriftServerStarter.stop(), aborting
routine-load planning. Wrap this method with the same admission guard and
return FE_SHUTTING_DOWN when closed.
##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -246,6 +246,10 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env,
return Status::Aborted("Missing vital param: table_name");
}
+ // TODO: Balance this reservation when setup fails before a callback
exists
Review Comment:
[P1] Balance the multi-table in-flight reservation on setup errors:
exec_plans() increments _inflight_cnt before exec_plan_fragment(), but
RETURN_IF_ERROR exits when FragmentMgr fails before creating a completion
callback, so the callback decrement never runs and _handle_consumer_finished()
never fulfills load_status_promise. A routine-load multi-table request can wait
indefinitely after a query-context setup failure. Decrement/cancel this
reservation (and propagate the first error) before returning.
##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -741,19 +749,86 @@ public static boolean isServerReady() {
}
private static void gracefulShutdown() {
- // wait for all queries to finish
+ // Health is already 503, but keep serving during LB / DNS convergence.
try {
- long now = System.currentTimeMillis();
- List<Coordinator> allCoordinators =
QeProcessorImpl.INSTANCE.getAllCoordinators();
- while (!allCoordinators.isEmpty() && System.currentTimeMillis() -
now < 300 * 1000L) {
- Thread.sleep(1000);
- allCoordinators =
QeProcessorImpl.INSTANCE.getAllCoordinators();
- LOG.info("waiting {} queries to finish before shutdown",
allCoordinators.size());
- }
- } catch (Throwable t) {
- LOG.error("", t);
+ LOG.info("graceful shutdown: serving during drain for {} seconds",
Config.graceful_shutdown_drain_seconds);
+ TimeUnit.SECONDS.sleep(Math.max(0,
Config.graceful_shutdown_drain_seconds));
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.warn("graceful shutdown drain interrupted", e);
}
+ // Close admission before the listener: an accepted connection can
race its closure.
+ QueryAdmissionController.INSTANCE.stopAccepting();
+ if (qeService != null) {
+ qeService.stopAcceptNewConnections();
+ }
+ try {
Review Comment:
[P1] Count HTTP Stream Load work in the FE drain: waitForGracefulShutdown()
has no HTTP admission counter, while LoadAction's upload/planning handlers
remain active until httpServer.shutdown() runs after the wait. An accepted
upload can therefore be invisible to the zero check and be cut off with its
transaction/body response when shutdown closes Jetty. Add an HTTP
admission/active-request fence covering body and response lifetime, or stop
HTTP admission before draining and wait for existing handlers.
##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -741,19 +749,86 @@ public static boolean isServerReady() {
}
private static void gracefulShutdown() {
- // wait for all queries to finish
+ // Health is already 503, but keep serving during LB / DNS convergence.
try {
- long now = System.currentTimeMillis();
- List<Coordinator> allCoordinators =
QeProcessorImpl.INSTANCE.getAllCoordinators();
- while (!allCoordinators.isEmpty() && System.currentTimeMillis() -
now < 300 * 1000L) {
- Thread.sleep(1000);
- allCoordinators =
QeProcessorImpl.INSTANCE.getAllCoordinators();
- LOG.info("waiting {} queries to finish before shutdown",
allCoordinators.size());
- }
- } catch (Throwable t) {
- LOG.error("", t);
+ LOG.info("graceful shutdown: serving during drain for {} seconds",
Config.graceful_shutdown_drain_seconds);
+ TimeUnit.SECONDS.sleep(Math.max(0,
Config.graceful_shutdown_drain_seconds));
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.warn("graceful shutdown drain interrupted", e);
}
+ // Close admission before the listener: an accepted connection can
race its closure.
+ QueryAdmissionController.INSTANCE.stopAccepting();
+ if (qeService != null) {
+ qeService.stopAcceptNewConnections();
+ }
+ try {
+ long deadline = System.nanoTime()
+ + TimeUnit.SECONDS.toNanos(Math.max(0,
Config.grace_shutdown_wait_seconds));
+ waitForGracefulShutdown(ExecuteEnv.getInstance().getScheduler(),
deadline);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.warn("graceful shutdown wait interrupted", e);
+ } catch (Throwable t) {
+ LOG.error("graceful shutdown failed while draining requests", t);
+ } finally {
+ TxnRpcAdmissionController.INSTANCE.stopAccepting();
+ }
LOG.info("graceful shutdown finished");
}
+
+ static void waitForGracefulShutdown(ConnectScheduler scheduler, long
deadline) throws InterruptedException {
+ // XNIO invokes close listeners synchronously, including transaction
rollback RPCs.
+ // Keep those waits on a daemon worker so even an idle session cannot
stall the deadline.
+ ExecutorService idleCloser =
ThreadPoolManager.newDaemonFixedThreadPool(
+ 1, 1, "graceful-idle-mysql-closer", false);
+ try {
+ Future<?> idleCleanup = null;
+ boolean cleanupStartedIdle = false;
+ while (true) {
+ if (idleCleanup == null) {
+ cleanupStartedIdle =
QueryAdmissionController.INSTANCE.getActiveRequests() == 0;
+ idleCleanup = idleCloser.submit(() ->
scheduler.closeIdleMysqlConnections(deadline));
+ }
+ int activeRequests =
QueryAdmissionController.INSTANCE.getActiveRequests();
+ int activeTxnRpcs =
TxnRpcAdmissionController.INSTANCE.getActiveRequests();
+ List<Coordinator> coordinators =
QeProcessorImpl.INSTANCE.getAllCoordinators();
+ boolean pendingFlightResults =
scheduler.hasPendingFlightResults();
+ boolean pendingIdleCleanup = !idleCleanup.isDone();
Review Comment:
[P2] Fence Flight continuations before declaring FE drained: the final zero
check samples QueryAdmissionController and then breaks, but
runContinuation()/closeSession() still call acquireContinuation() after
stopAccepting and can race into the gap. The Flight listener is stopped
immediately afterward, so a late DoGet or close-session cleanup can be cut off.
Atomically close continuation admission with the final check and wait for
holders before stopping Flight.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/QeService.java:
##########
@@ -66,6 +66,16 @@ public void start() throws Exception {
LOG.info("QE service start.");
}
+ public void stopAcceptNewConnections() {
Review Comment:
[P2] Fence new Flight session creation during drain:
stopAcceptNewConnections() stops only mysqlStarter, while Flight authentication
still opens/registers ConnectContext sessions without QueryAdmissionController.
Clients can create idle bearer sessions after command admission closes, consume
FE/Flight connection quotas, then have commands rejected; idle cleanup excludes
ARROW_FLIGHT_SQL. Gate initial Flight authentication or stop only new-session
auth while retaining already-issued continuations.
##########
be/src/runtime/exec_env.cpp:
##########
@@ -205,20 +208,38 @@ std::map<TNetworkAddress, FrontendInfo>
ExecEnv::get_running_frontends() {
return res;
}
+void ExecEnv::wait_for_shutdown_heartbeat() {
+ int32_t elapsed = 0;
+ while (!k_shutdown_heartbeat_sent && elapsed <
config::grace_shutdown_heartbeat_wait_seconds) {
+ sleep(1);
+ ++elapsed;
+ }
+ LOG(INFO) << "shutdown heartbeat prepared: " <<
k_shutdown_heartbeat_sent.load() << ", waited "
+ << elapsed << " seconds";
+ // The response may still be in transit. This buffer also gives followers
time
+ // to replay shutdown state. It does not prove that every FE has applied
it.
+ sleep(std::max(0, config::grace_shutdown_heartbeat_propagation_seconds));
+}
+
void ExecEnv::wait_for_all_tasks_done() {
- // For graceful shutdown, need to wait for all running queries to stop
+ // The cloud load receiver and coordinator must finish. For example,
+ // an HTTP upload or transaction commit can outlive all of its pipeline
fragments.
int32_t wait_seconds_passed = 0;
while (true) {
int num_queries = _fragment_mgr->running_query_num();
- if (num_queries < 1) {
+ size_t num_channels = _load_channel_mgr->num_active_channels();
Review Comment:
[P1] Fence new HTTP loads before the BE shutdown census:
wait_for_all_tasks_done() breaks on a zero sample, but the HTTP starter remains
live and Stream Load handlers do not check k_doris_exit or any admission gate.
A request arriving just after that sample creates a StreamLoadContext and can
be terminated by the subsequent BE exit/HTTP stop, losing an accepted load.
Close HTTP admission atomically before the final census (and count in-flight
body/response work), or recheck/fence context creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]