JackieTien97 commented on code in PR #19224:
URL: https://github.com/apache/iotdb/pull/19224#discussion_r4545201604
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -161,12 +192,20 @@ public void start() {
@Override
public void stop(Throwable t) {
+ boolean terminateImmediately;
+ synchronized (this) {
+ if (stopped) {
+ return;
+ }
+ stopped = true;
+ terminateImmediately = dispatchFailed || !dispatchComplete;
+ }
dispatcher.abort();
if (stateTracker != null) {
stateTracker.abort();
}
if (queryTerminator != null) {
- if (dispatchFailed) {
+ if (terminateImmediately) {
// The FragmentInstances dispatched before the failure can't finish on
their own and hold
// operator memory until they are cancelled, so don't wait for the
grace period.
Review Comment:
Nit: this comment still describes only the failed-dispatch case.
`terminateImmediately` is now also true when `stop()` arrives before `start()`
or while the dispatch is in flight. The Javadoc of
`IQueryTerminator.terminateImmediately` ("Used when the dispatch failed: …")
has the same gap.
```suggestion
if (terminateImmediately) {
// Either the dispatch failed or the stop arrived before the
dispatch completed. The
// FragmentInstances dispatched so far can't finish on their own and
hold operator memory
// until they are cancelled, so don't wait for the grace period.
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/FragmentInstanceDispatcherImpl.java:
##########
@@ -767,7 +797,37 @@ private void dispatchLocally(final FragmentInstance
instance)
}
@Override
- public void abort() {}
+ public synchronized void abort() {
+ aborted = true;
+ LinkedBlockingQueue<Pair<SubPlan, Boolean>> queue = dispatchQueue;
+ if (queue != null) {
+ queue.offer(new Pair<>(null, false));
Review Comment:
Nit: the queue is unbounded now, so `offer` cannot fail here, but SonarCloud
reports the ignored return value (java:S899) as one of the new reliability
items behind the failed quality gate. `add` matches the other call sites in
this class.
```suggestion
queue.add(new Pair<>(null, false));
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -114,17 +133,29 @@ && isQuery()
@Override
public void start() {
+ synchronized (this) {
+ if (started || stopped || stateMachine.getState().isDone()) {
+ return;
+ }
+ started = true;
+ }
stateMachine.transitionToDispatching();
long startTime = System.nanoTime();
- Future<FragInstanceDispatchResult> dispatchResultFuture =
- dispatcher.dispatch(rootSubPlan, instances);
// NOTICE: the FragmentInstance may be dispatched to another Host due to
consensus redirect.
// So we need to start the state fetcher after the dispatching stage.
try {
+ Future<FragInstanceDispatchResult> dispatchResultFuture =
+ dispatcher.dispatch(rootSubPlan, instances);
FragInstanceDispatchResult result = dispatchResultFuture.get();
+ synchronized (this) {
+ if (stopped) {
+ return;
Review Comment:
Three additions from a second pass.
**This is new with the change.** On base, a stop that arrived during
dispatch was lost because the scheduler had not been published yet, so
`start()` went on to RUNNING and `getStatus()` returned. Delivering that stop
is the point of this PR; what is missing is completing the query state
afterwards.
**Why the stale-query sweep reaches a query that is still dispatching.**
`cleanUpStaleQueries()` skips executions for which `isActive()` is true, and
`isActive()` returns `startTimeOfCurrentRpc == -1`, which is true when *no* RPC
is in flight. So the sweep leaves idle executions alone and acts on the ones
whose RPC is still running, including the initial `executeStatement` call that
is inside `start()`. That looks inverted relative to the Javadoc of
`cleanUpStaleQueries()`. It dates from #17358 and is outside this PR, so it
probably deserves its own issue. The trigger here still needs dispatch to
outlast `timeout + 60 s`.
**A candidate fix, checked.** Remember the cause in `stop()`. In this
branch, if the state is not terminal yet, call `transitionToFailed(cause)`, or
`transitionToAborted()` when the cause is null, before returning. The probe
uses a real `QueryExecution` and a real `ClusterScheduler` with a mocked
dispatcher, and calls `stopAndCleanup(cause)` while `dispatch()` is blocked:
| | state after `start()` returns | `getStatus()` |
|---|---|---|
| control, no stop | `RUNNING` | returns 200 |
| this commit | `DISPATCHING` | still blocked after 5 s |
| fix, stop with the sweep's `QueryTimeoutRuntimeException` | `FAILED` |
returns 720 |
| fix, stop with a null cause | `ABORTED` | returns 301 |
`ClusterSchedulerLifecycleTest`, `QueryExecutionLifecycleTest` and
`QueryExecutionRetryTest` still pass with it. Retries are not affected, because
`retry()` only stops a scheduler whose `start()` has already returned.
The variant I ran had no explicit `isDone()` check. Add one: when the stop
comes from the terminal-state listener, `transitionToFailed(Throwable)` would
otherwise fill an empty `failureException` on a query that failed with a status
only.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java:
##########
@@ -114,17 +133,29 @@ && isQuery()
@Override
public void start() {
+ synchronized (this) {
+ if (started || stopped || stateMachine.getState().isDone()) {
Review Comment:
**This clause is what turns CI red.** `unit-test (17, ubuntu-latest,
datanode)`, `unit-test (17, windows-latest, datanode)` and `codecov` all fail
on the same two cases, and the unit-test jobs are green on the base commit
`509c395`:
-
`IoTDBInternalLocalReporterTest.testRemoteInterruptionDoesNotReplaceExistingFailure`
-
`IoTDBInternalLocalReporterTest.testRemoteInterruptionDoesNotHideExistingException`
Both move the query state machine to FAILED, then call `start()` on a real
`ClusterScheduler` and wait for the write batch to reach `sendBatchPlanNode`
(`dispatched.await(5, SECONDS)` at `IoTDBInternalLocalReporterTest.java:489`).
With this clause `start()` returns before dispatching, so the latch times out.
Local runs on this commit:
- as is: `IoTDBInternalLocalReporterTest` runs 25 cases, 2 fail
- with only this clause removed: `IoTDBInternalLocalReporterTest` 25/25 and
`ClusterSchedulerLifecycleTest` 4/4. These are the only two test classes that
construct a `ClusterScheduler` directly.
Two ways out:
1. Drop the clause (suggestion below). `QueryExecution.schedule()` already
turns "terminal before the handoff" into stop-before-start. The clause only
adds the short window in which the state is terminal but the listener's
`stop()` has not reached the scheduler yet. In that window a read dispatch is
aborted and cancelled by the `stop()` that follows (`!dispatchComplete` selects
`terminateImmediately`), so cleanup is still complete. A write batch would
still be sent, but I found no production path that makes a write terminal
before `schedule()`: only queries are registered in `queryExecutionMap`, which
is what KILL QUERY and the stale-query sweep work from.
2. Keep the clause, since `LoadTsFileScheduler.start()` has the same check,
and change the two cases to record the failure after dispatch has started, for
example from the `sendBatchPlanNode` stub. I have not tried this one.
```suggestion
if (started || stopped) {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryAttemptRegistry.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.execution.fragment;
+
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.db.i18n.DataNodeQueryMessages;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.QueryAttemptId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+
+import static com.google.common.math.LongMath.saturatedAdd;
+
+/**
+ * Owns shared DataNode state for one execution attempt. All
registration/closing bookkeeping is
+ * serialized by compute on the attempt key; planning and state transitions
run outside that lock.
+ * Closed entries retain no query data after their last actual user releases
it. They only fence
+ * late dispatch until the request's immutable admission deadline, after which
the request itself is
+ * rejected, even if the entry has been removed.
+ */
+class QueryAttemptRegistry {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(QueryAttemptRegistry.class);
+
+ private final Map<QueryAttemptId, Entry> attempts = new
ConcurrentHashMap<>();
+ private final LongSupplier clock;
+ // Expiring a fence must remain irreversible even if the wall clock
subsequently goes backwards.
+ private final AtomicLong observedTime = new AtomicLong();
+
+ QueryAttemptRegistry() {
+ this(System::currentTimeMillis);
+ }
+
+ QueryAttemptRegistry(LongSupplier clock) {
+ this.clock = clock;
+ }
+
+ private long now() {
+ return observedTime.accumulateAndGet(clock.getAsLong(), Math::max);
+ }
+
+ Registration register(
+ FragmentInstance instance, FragmentInstanceStateMachine stateMachine,
boolean dataQuery) {
+ QueryAttemptId key =
+ new QueryAttemptId(
+ instance.getId().getQueryId(),
+ instance.hasQueryAttempt()
+ ? instance.getQueryAttemptId()
+ : QueryAttemptId.LEGACY_ATTEMPT);
+ long executionDeadline =
+ instance.hasQueryAttempt()
+ ? instance.getQueryDeadline()
+ : saturatedAdd(now(), instance.getTimeOut());
+ long admissionDeadline =
+ instance.hasQueryAttempt()
+ ? Math.min(executionDeadline, instance.getAdmissionDeadline())
+ : executionDeadline;
+ AtomicReference<Registration> result = new AtomicReference<>();
+ attempts.compute(
+ key,
+ (ignored, existing) -> {
+ long currentTime = now();
+ if (admissionDeadline <= currentTime
+ || (existing != null
+ && (existing.closed || existing.executionDeadline <=
currentTime))) {
+ throw rejected(key, admissionDeadline <= currentTime);
+ }
+ Entry entry =
+ existing == null
+ ? new Entry(executionDeadline, Math.max(1,
instance.getDataNodeFINum()))
+ : existing;
+ // A transport retry must never extend the lifetime of an existing
attempt.
+ entry.executionDeadline = Math.min(entry.executionDeadline,
executionDeadline);
+ entry.fenceDeadline = Math.max(entry.fenceDeadline,
admissionDeadline);
+ if (entry.active.containsKey(instance.getId())) {
+ throw new IoTDBRuntimeException(
+ String.format(
+ DataNodeQueryMessages
+
.QUERY_EXCEPTION_REPEATED_RPC_CALL_DETECTED_FOR_FRAGMENTINSTANCE_S_REJECT_BF609A26,
+ instance.getId().getFullId()),
+ TSStatusCode.REPEATED_RPC_CALL.getStatusCode());
+ }
+ if (dataQuery && entry.context == null) {
+ entry.context = new DataNodeQueryContext();
+ }
+ Registration registration =
+ new Registration(key, entry, instance.getId(), stateMachine,
entry.context);
+ entry.active.put(instance.getId(), registration);
+ result.set(registration);
+ return entry;
+ });
+ return result.get();
+ }
+
+ void close(QueryAttemptId key, long deadline, boolean hasThrowable) {
+ List<Registration> toCancel = new ArrayList<>();
+ long currentTime = now();
+ attempts.compute(
+ key,
+ (ignored, existing) -> {
+ if (existing == null && deadline <= currentTime) {
+ return null;
+ }
+ Entry entry = existing == null ? new Entry(deadline, 0) : existing;
+ entry.closed = true;
+ entry.fenceDeadline = Math.max(entry.fenceDeadline, deadline);
+ if (entry.active != null) {
+ toCancel.addAll(entry.active.values());
+ }
+ return retireIfUnused(key, entry, currentTime);
+ });
+ terminate(toCancel, hasThrowable);
+ }
+
+ void removeExpired() {
+ long currentTime = now();
+ List<Registration> toCancel = new ArrayList<>();
+ for (QueryAttemptId key : attempts.keySet()) {
+ attempts.computeIfPresent(
+ key,
+ (ignored, entry) -> {
+ if ((entry.closed ? entry.fenceDeadline : entry.executionDeadline)
> currentTime) {
+ return entry;
+ }
+ entry.closed = true;
+ if (entry.active != null) {
+ toCancel.addAll(entry.active.values());
+ }
+ return retireIfUnused(key, entry, currentTime);
+ });
+ }
+ terminate(toCancel, true);
Review Comment:
**Expiry ends the FIs with `cancel()` and no cause, so a timeout may not
surface as 720.**
`executionDeadline` is the coordinator's `startTime + timeOut`. The FI's own
deadline is set by `DriverTask` as arrival time plus the remaining timeout,
which is later by the time between planning on the coordinator and submission
on this node. This sweep runs every 2 s, so it can fire inside that gap, before
`DriverTaskTimeoutSentinelThread` does. With synchronized clocks the gap is
small and this is rare. Once this node's clock is ahead of the coordinator's by
more than the sweep interval, it happens for every query that times out (see
the wall-clock thread on `register`).
What the FI reports in the two cases, from a unit probe on
`FragmentInstanceContext.getInstanceInfo()`:
| path | state | message | error code |
|---|---|---|---|
| this sweep | `CANCELLED` | empty | none |
| existing driver timeout, `failed(new QueryTimeoutException())` | `FAILED`
| `Query execution is time out` | 720 |
`CANCELLED` counts as failed. For a failed FI with neither an error code nor
failure info, `FixedRateFragInsStateTracker.updateQueryState` builds a plain
`RuntimeException("FragmentInstance[…] is failed. …")`, so the client would get
a generic failure instead of `QUERY_TIMEOUT`. That last step is read from the
code, not run end to end.
Suggestion: when a non-closed entry expires on `executionDeadline`, fail its
FIs with a `QueryTimeoutException` instead of cancelling them. Closed entries
whose fence expires should stay as they are: their FIs were already terminated
by `close()`, and `failed()` would add a timeout cause to them.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryAttemptRegistry.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.execution.fragment;
+
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.db.i18n.DataNodeQueryMessages;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.QueryAttemptId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+
+import static com.google.common.math.LongMath.saturatedAdd;
+
+/**
+ * Owns shared DataNode state for one execution attempt. All
registration/closing bookkeeping is
+ * serialized by compute on the attempt key; planning and state transitions
run outside that lock.
+ * Closed entries retain no query data after their last actual user releases
it. They only fence
+ * late dispatch until the request's immutable admission deadline, after which
the request itself is
+ * rejected, even if the entry has been removed.
+ */
+class QueryAttemptRegistry {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(QueryAttemptRegistry.class);
+
+ private final Map<QueryAttemptId, Entry> attempts = new
ConcurrentHashMap<>();
+ private final LongSupplier clock;
+ // Expiring a fence must remain irreversible even if the wall clock
subsequently goes backwards.
+ private final AtomicLong observedTime = new AtomicLong();
+
+ QueryAttemptRegistry() {
+ this(System::currentTimeMillis);
+ }
+
+ QueryAttemptRegistry(LongSupplier clock) {
+ this.clock = clock;
+ }
+
+ private long now() {
+ return observedTime.accumulateAndGet(clock.getAsLong(), Math::max);
+ }
+
+ Registration register(
+ FragmentInstance instance, FragmentInstanceStateMachine stateMachine,
boolean dataQuery) {
+ QueryAttemptId key =
+ new QueryAttemptId(
+ instance.getId().getQueryId(),
+ instance.hasQueryAttempt()
+ ? instance.getQueryAttemptId()
+ : QueryAttemptId.LEGACY_ATTEMPT);
+ long executionDeadline =
+ instance.hasQueryAttempt()
+ ? instance.getQueryDeadline()
+ : saturatedAdd(now(), instance.getTimeOut());
+ long admissionDeadline =
+ instance.hasQueryAttempt()
+ ? Math.min(executionDeadline, instance.getAdmissionDeadline())
+ : executionDeadline;
+ AtomicReference<Registration> result = new AtomicReference<>();
+ attempts.compute(
+ key,
+ (ignored, existing) -> {
+ long currentTime = now();
+ if (admissionDeadline <= currentTime
+ || (existing != null
+ && (existing.closed || existing.executionDeadline <=
currentTime))) {
+ throw rejected(key, admissionDeadline <= currentTime);
Review Comment:
One more consequence of the same comparison, for skew smaller than the
admission window.
The FI is admitted, but `removeExpired()` expires the entry when *this
node's* clock passes `executionDeadline`, which is the coordinator's `startTime
+ timeOut`. With the receiver ahead by δ, running FIs are cancelled δ early, so
the effective query timeout becomes `timeout − δ`. On base the FI deadline is a
relative duration applied to the local clock
(`DriverTask.computeDeadlineTimeInMs(instance.getTimeOut())`), so it did not
depend on the two clocks agreeing. A clock-injected probe confirms that the
cut-off follows the receiver's clock only: the sweep leaves the FI `RUNNING` 1
ms before `executionDeadline` and cancels it at `executionDeadline`.
Two changes that would narrow this without a new protocol:
- Take `executionDeadline` from the local relative timeout, as the legacy
branch just above already does (`now() + instance.getTimeOut()`), and keep the
coordinator's absolute timestamp for the admission decision only. That removes
the early cancellation. The rejection described above remains.
- When a request is rejected as expired, put the request's deadline and the
local clock reading into the message. Today both the DataNode log and the
client only show `Query attempt … is closed or expired` (720), which is hard to
tell apart from a real timeout in the field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]