JackieTien97 commented on code in PR #19224:
URL: https://github.com/apache/iotdb/pull/19224#discussion_r4545201604


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -161,12 +192,20 @@ public void start() {
 
   @Override
   public void stop(Throwable t) {
+    boolean terminateImmediately;
+    synchronized (this) {
+      if (stopped) {
+        return;
+      }
+      stopped = true;
+      terminateImmediately = dispatchFailed || !dispatchComplete;
+    }
     dispatcher.abort();
     if (stateTracker != null) {
       stateTracker.abort();
     }
     if (queryTerminator != null) {
-      if (dispatchFailed) {
+      if (terminateImmediately) {
         // The FragmentInstances dispatched before the failure can't finish on 
their own and hold
         // operator memory until they are cancelled, so don't wait for the 
grace period.

Review Comment:
   Nit: this comment still describes only the failed-dispatch case. 
`terminateImmediately` is now also true when `stop()` arrives before `start()` 
or while the dispatch is in flight. The Javadoc of 
`IQueryTerminator.terminateImmediately` ("Used when the dispatch failed: …") 
has the same gap.
   
   ```suggestion
         if (terminateImmediately) {
           // Either the dispatch failed or the stop arrived before the 
dispatch completed. The
           // FragmentInstances dispatched so far can't finish on their own and 
hold operator memory
           // until they are cancelled, so don't wait for the grace period.
   ```
   



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/FragmentInstanceDispatcherImpl.java:
##########
@@ -767,7 +797,37 @@ private void dispatchLocally(final FragmentInstance 
instance)
   }
 
   @Override
-  public void abort() {}
+  public synchronized void abort() {
+    aborted = true;
+    LinkedBlockingQueue<Pair<SubPlan, Boolean>> queue = dispatchQueue;
+    if (queue != null) {
+      queue.offer(new Pair<>(null, false));

Review Comment:
   Nit: the queue is unbounded now, so `offer` cannot fail here, but SonarCloud 
reports the ignored return value (java:S899) as one of the new reliability 
items behind the failed quality gate. `add` matches the other call sites in 
this class.
   
   ```suggestion
         queue.add(new Pair<>(null, false));
   ```
   



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -114,17 +133,29 @@ && isQuery()
 
   @Override
   public void start() {
+    synchronized (this) {
+      if (started || stopped || stateMachine.getState().isDone()) {
+        return;
+      }
+      started = true;
+    }
     stateMachine.transitionToDispatching();
     long startTime = System.nanoTime();
-    Future<FragInstanceDispatchResult> dispatchResultFuture =
-        dispatcher.dispatch(rootSubPlan, instances);
 
     // NOTICE: the FragmentInstance may be dispatched to another Host due to 
consensus redirect.
     // So we need to start the state fetcher after the dispatching stage.
     try {
+      Future<FragInstanceDispatchResult> dispatchResultFuture =
+          dispatcher.dispatch(rootSubPlan, instances);
       FragInstanceDispatchResult result = dispatchResultFuture.get();
+      synchronized (this) {
+        if (stopped) {
+          return;

Review Comment:
   Three additions from a second pass.
   
   **This is new with the change.** On base, a stop that arrived during 
dispatch was lost because the scheduler had not been published yet, so 
`start()` went on to RUNNING and `getStatus()` returned. Delivering that stop 
is the point of this PR; what is missing is completing the query state 
afterwards.
   
   **Why the stale-query sweep reaches a query that is still dispatching.** 
`cleanUpStaleQueries()` skips executions for which `isActive()` is true, and 
`isActive()` returns `startTimeOfCurrentRpc == -1`, which is true when *no* RPC 
is in flight. So the sweep leaves idle executions alone and acts on the ones 
whose RPC is still running, including the initial `executeStatement` call that 
is inside `start()`. That looks inverted relative to the Javadoc of 
`cleanUpStaleQueries()`. It dates from #17358 and is outside this PR, so it 
probably deserves its own issue. The trigger here still needs dispatch to 
outlast `timeout + 60 s`.
   
   **A candidate fix, checked.** Remember the cause in `stop()`. In this 
branch, if the state is not terminal yet, call `transitionToFailed(cause)`, or 
`transitionToAborted()` when the cause is null, before returning. The probe 
uses a real `QueryExecution` and a real `ClusterScheduler` with a mocked 
dispatcher, and calls `stopAndCleanup(cause)` while `dispatch()` is blocked:
   
   | | state after `start()` returns | `getStatus()` |
   |---|---|---|
   | control, no stop | `RUNNING` | returns 200 |
   | this commit | `DISPATCHING` | still blocked after 5 s |
   | fix, stop with the sweep's `QueryTimeoutRuntimeException` | `FAILED` | 
returns 720 |
   | fix, stop with a null cause | `ABORTED` | returns 301 |
   
   `ClusterSchedulerLifecycleTest`, `QueryExecutionLifecycleTest` and 
`QueryExecutionRetryTest` still pass with it. Retries are not affected, because 
`retry()` only stops a scheduler whose `start()` has already returned.
   
   The variant I ran had no explicit `isDone()` check. Add one: when the stop 
comes from the terminal-state listener, `transitionToFailed(Throwable)` would 
otherwise fill an empty `failureException` on a query that failed with a status 
only.
   



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -114,17 +133,29 @@ && isQuery()
 
   @Override
   public void start() {
+    synchronized (this) {
+      if (started || stopped || stateMachine.getState().isDone()) {

Review Comment:
   **This clause is what turns CI red.** `unit-test (17, ubuntu-latest, 
datanode)`, `unit-test (17, windows-latest, datanode)` and `codecov` all fail 
on the same two cases, and the unit-test jobs are green on the base commit 
`509c395`:
   
   - 
`IoTDBInternalLocalReporterTest.testRemoteInterruptionDoesNotReplaceExistingFailure`
   - 
`IoTDBInternalLocalReporterTest.testRemoteInterruptionDoesNotHideExistingException`
   
   Both move the query state machine to FAILED, then call `start()` on a real 
`ClusterScheduler` and wait for the write batch to reach `sendBatchPlanNode` 
(`dispatched.await(5, SECONDS)` at `IoTDBInternalLocalReporterTest.java:489`). 
With this clause `start()` returns before dispatching, so the latch times out.
   
   Local runs on this commit:
   - as is: `IoTDBInternalLocalReporterTest` runs 25 cases, 2 fail
   - with only this clause removed: `IoTDBInternalLocalReporterTest` 25/25 and 
`ClusterSchedulerLifecycleTest` 4/4. These are the only two test classes that 
construct a `ClusterScheduler` directly.
   
   Two ways out:
   1. Drop the clause (suggestion below). `QueryExecution.schedule()` already 
turns "terminal before the handoff" into stop-before-start. The clause only 
adds the short window in which the state is terminal but the listener's 
`stop()` has not reached the scheduler yet. In that window a read dispatch is 
aborted and cancelled by the `stop()` that follows (`!dispatchComplete` selects 
`terminateImmediately`), so cleanup is still complete. A write batch would 
still be sent, but I found no production path that makes a write terminal 
before `schedule()`: only queries are registered in `queryExecutionMap`, which 
is what KILL QUERY and the stale-query sweep work from.
   2. Keep the clause, since `LoadTsFileScheduler.start()` has the same check, 
and change the two cases to record the failure after dispatch has started, for 
example from the `sendBatchPlanNode` stub. I have not tried this one.
   
   ```suggestion
         if (started || stopped) {
   ```
   



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryAttemptRegistry.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.execution.fragment;
+
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.db.i18n.DataNodeQueryMessages;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.QueryAttemptId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+
+import static com.google.common.math.LongMath.saturatedAdd;
+
+/**
+ * Owns shared DataNode state for one execution attempt. All 
registration/closing bookkeeping is
+ * serialized by compute on the attempt key; planning and state transitions 
run outside that lock.
+ * Closed entries retain no query data after their last actual user releases 
it. They only fence
+ * late dispatch until the request's immutable admission deadline, after which 
the request itself is
+ * rejected, even if the entry has been removed.
+ */
+class QueryAttemptRegistry {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(QueryAttemptRegistry.class);
+
+  private final Map<QueryAttemptId, Entry> attempts = new 
ConcurrentHashMap<>();
+  private final LongSupplier clock;
+  // Expiring a fence must remain irreversible even if the wall clock 
subsequently goes backwards.
+  private final AtomicLong observedTime = new AtomicLong();
+
+  QueryAttemptRegistry() {
+    this(System::currentTimeMillis);
+  }
+
+  QueryAttemptRegistry(LongSupplier clock) {
+    this.clock = clock;
+  }
+
+  private long now() {
+    return observedTime.accumulateAndGet(clock.getAsLong(), Math::max);
+  }
+
+  Registration register(
+      FragmentInstance instance, FragmentInstanceStateMachine stateMachine, 
boolean dataQuery) {
+    QueryAttemptId key =
+        new QueryAttemptId(
+            instance.getId().getQueryId(),
+            instance.hasQueryAttempt()
+                ? instance.getQueryAttemptId()
+                : QueryAttemptId.LEGACY_ATTEMPT);
+    long executionDeadline =
+        instance.hasQueryAttempt()
+            ? instance.getQueryDeadline()
+            : saturatedAdd(now(), instance.getTimeOut());
+    long admissionDeadline =
+        instance.hasQueryAttempt()
+            ? Math.min(executionDeadline, instance.getAdmissionDeadline())
+            : executionDeadline;
+    AtomicReference<Registration> result = new AtomicReference<>();
+    attempts.compute(
+        key,
+        (ignored, existing) -> {
+          long currentTime = now();
+          if (admissionDeadline <= currentTime
+              || (existing != null
+                  && (existing.closed || existing.executionDeadline <= 
currentTime))) {
+            throw rejected(key, admissionDeadline <= currentTime);
+          }
+          Entry entry =
+              existing == null
+                  ? new Entry(executionDeadline, Math.max(1, 
instance.getDataNodeFINum()))
+                  : existing;
+          // A transport retry must never extend the lifetime of an existing 
attempt.
+          entry.executionDeadline = Math.min(entry.executionDeadline, 
executionDeadline);
+          entry.fenceDeadline = Math.max(entry.fenceDeadline, 
admissionDeadline);
+          if (entry.active.containsKey(instance.getId())) {
+            throw new IoTDBRuntimeException(
+                String.format(
+                    DataNodeQueryMessages
+                        
.QUERY_EXCEPTION_REPEATED_RPC_CALL_DETECTED_FOR_FRAGMENTINSTANCE_S_REJECT_BF609A26,
+                    instance.getId().getFullId()),
+                TSStatusCode.REPEATED_RPC_CALL.getStatusCode());
+          }
+          if (dataQuery && entry.context == null) {
+            entry.context = new DataNodeQueryContext();
+          }
+          Registration registration =
+              new Registration(key, entry, instance.getId(), stateMachine, 
entry.context);
+          entry.active.put(instance.getId(), registration);
+          result.set(registration);
+          return entry;
+        });
+    return result.get();
+  }
+
+  void close(QueryAttemptId key, long deadline, boolean hasThrowable) {
+    List<Registration> toCancel = new ArrayList<>();
+    long currentTime = now();
+    attempts.compute(
+        key,
+        (ignored, existing) -> {
+          if (existing == null && deadline <= currentTime) {
+            return null;
+          }
+          Entry entry = existing == null ? new Entry(deadline, 0) : existing;
+          entry.closed = true;
+          entry.fenceDeadline = Math.max(entry.fenceDeadline, deadline);
+          if (entry.active != null) {
+            toCancel.addAll(entry.active.values());
+          }
+          return retireIfUnused(key, entry, currentTime);
+        });
+    terminate(toCancel, hasThrowable);
+  }
+
+  void removeExpired() {
+    long currentTime = now();
+    List<Registration> toCancel = new ArrayList<>();
+    for (QueryAttemptId key : attempts.keySet()) {
+      attempts.computeIfPresent(
+          key,
+          (ignored, entry) -> {
+            if ((entry.closed ? entry.fenceDeadline : entry.executionDeadline) 
> currentTime) {
+              return entry;
+            }
+            entry.closed = true;
+            if (entry.active != null) {
+              toCancel.addAll(entry.active.values());
+            }
+            return retireIfUnused(key, entry, currentTime);
+          });
+    }
+    terminate(toCancel, true);

Review Comment:
   **Expiry ends the FIs with `cancel()` and no cause, so a timeout may not 
surface as 720.**
   
   `executionDeadline` is the coordinator's `startTime + timeOut`. The FI's own 
deadline is set by `DriverTask` as arrival time plus the remaining timeout, 
which is later by the time between planning on the coordinator and submission 
on this node. This sweep runs every 2 s, so it can fire inside that gap, before 
`DriverTaskTimeoutSentinelThread` does. With synchronized clocks the gap is 
small and this is rare. Once this node's clock is ahead of the coordinator's by 
more than the sweep interval, it happens for every query that times out (see 
the wall-clock thread on `register`).
   
   What the FI reports in the two cases, from a unit probe on 
`FragmentInstanceContext.getInstanceInfo()`:
   
   | path | state | message | error code |
   |---|---|---|---|
   | this sweep | `CANCELLED` | empty | none |
   | existing driver timeout, `failed(new QueryTimeoutException())` | `FAILED` 
| `Query execution is time out` | 720 |
   
   `CANCELLED` counts as failed. For a failed FI with neither an error code nor 
failure info, `FixedRateFragInsStateTracker.updateQueryState` builds a plain 
`RuntimeException("FragmentInstance[…] is failed. …")`, so the client would get 
a generic failure instead of `QUERY_TIMEOUT`. That last step is read from the 
code, not run end to end.
   
   Suggestion: when a non-closed entry expires on `executionDeadline`, fail its 
FIs with a `QueryTimeoutException` instead of cancelling them. Closed entries 
whose fence expires should stay as they are: their FIs were already terminated 
by `close()`, and `failed()` would add a timeout cause to them.
   



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryAttemptRegistry.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.execution.fragment;
+
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.db.i18n.DataNodeQueryMessages;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.QueryAttemptId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+
+import static com.google.common.math.LongMath.saturatedAdd;
+
+/**
+ * Owns shared DataNode state for one execution attempt. All 
registration/closing bookkeeping is
+ * serialized by compute on the attempt key; planning and state transitions 
run outside that lock.
+ * Closed entries retain no query data after their last actual user releases 
it. They only fence
+ * late dispatch until the request's immutable admission deadline, after which 
the request itself is
+ * rejected, even if the entry has been removed.
+ */
+class QueryAttemptRegistry {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(QueryAttemptRegistry.class);
+
+  private final Map<QueryAttemptId, Entry> attempts = new 
ConcurrentHashMap<>();
+  private final LongSupplier clock;
+  // Expiring a fence must remain irreversible even if the wall clock 
subsequently goes backwards.
+  private final AtomicLong observedTime = new AtomicLong();
+
+  QueryAttemptRegistry() {
+    this(System::currentTimeMillis);
+  }
+
+  QueryAttemptRegistry(LongSupplier clock) {
+    this.clock = clock;
+  }
+
+  private long now() {
+    return observedTime.accumulateAndGet(clock.getAsLong(), Math::max);
+  }
+
+  Registration register(
+      FragmentInstance instance, FragmentInstanceStateMachine stateMachine, 
boolean dataQuery) {
+    QueryAttemptId key =
+        new QueryAttemptId(
+            instance.getId().getQueryId(),
+            instance.hasQueryAttempt()
+                ? instance.getQueryAttemptId()
+                : QueryAttemptId.LEGACY_ATTEMPT);
+    long executionDeadline =
+        instance.hasQueryAttempt()
+            ? instance.getQueryDeadline()
+            : saturatedAdd(now(), instance.getTimeOut());
+    long admissionDeadline =
+        instance.hasQueryAttempt()
+            ? Math.min(executionDeadline, instance.getAdmissionDeadline())
+            : executionDeadline;
+    AtomicReference<Registration> result = new AtomicReference<>();
+    attempts.compute(
+        key,
+        (ignored, existing) -> {
+          long currentTime = now();
+          if (admissionDeadline <= currentTime
+              || (existing != null
+                  && (existing.closed || existing.executionDeadline <= 
currentTime))) {
+            throw rejected(key, admissionDeadline <= currentTime);

Review Comment:
   One more consequence of the same comparison, for skew smaller than the 
admission window.
   
   The FI is admitted, but `removeExpired()` expires the entry when *this 
node's* clock passes `executionDeadline`, which is the coordinator's `startTime 
+ timeOut`. With the receiver ahead by δ, running FIs are cancelled δ early, so 
the effective query timeout becomes `timeout − δ`. On base the FI deadline is a 
relative duration applied to the local clock 
(`DriverTask.computeDeadlineTimeInMs(instance.getTimeOut())`), so it did not 
depend on the two clocks agreeing. A clock-injected probe confirms that the 
cut-off follows the receiver's clock only: the sweep leaves the FI `RUNNING` 1 
ms before `executionDeadline` and cancels it at `executionDeadline`.
   
   Two changes that would narrow this without a new protocol:
   - Take `executionDeadline` from the local relative timeout, as the legacy 
branch just above already does (`now() + instance.getTimeOut()`), and keep the 
coordinator's absolute timestamp for the admission decision only. That removes 
the early cancellation. The rejection described above remains.
   - When a request is rejected as expired, put the request's deadline and the 
local clock reading into the message. Today both the DataNode log and the 
client only show `Query attempt … is closed or expired` (720), which is hard to 
tell apart from a real timeout in the field.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to