JackieTien97 commented on code in PR #18869:
URL: https://github.com/apache/iotdb/pull/18869#discussion_r4300854997


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/SharedExchangeMemoryEligibility.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.plan.planner;
+
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.CollectNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.TopKNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.UnionNode;
+import 
org.apache.iotdb.db.queryengine.plan.relational.planner.node.ExchangeNode;
+
+import java.util.ArrayDeque;
+import java.util.Deque;
+import java.util.List;
+
+/** Select shared budgets only for FIs whose fan-in can consume any ready 
input. */
+final class SharedExchangeMemoryEligibility {
+  private SharedExchangeMemoryEligibility() {}
+
+  static boolean isEligible(PlanNode root) {
+    boolean hasCollect = false;
+    Deque<PlanNode> nodes = new ArrayDeque<>();
+    nodes.push(root);
+    while (!nodes.isEmpty()) {
+      PlanNode node = nodes.pop();
+      if (node instanceof ExchangeNode) {
+        // An upstream plan, if still attached, belongs to a different FI.
+        continue;
+      }
+      // TopK's input ordering assumptions remain unchanged, including its 
single-child form.
+      if (node instanceof TopKNode) {
+        return false;
+      }

Review Comment:
   **Order-sensitive single-input operators need an ordered Collect too.** TopK 
is excluded here because it depends on input order. Fill without FILL_GROUP and 
`diff()` depend on input order as well, but they are not excluded.
   
   PlanTester on 788a995c75 (the plans are the same as on base; only the 
runtime mode changed):
   
   | query | root FI | eligible |
   |---|---|---|
   | `SELECT time, tag1, tag2, tag3, s1 FROM table1 FILL METHOD PREVIOUS` | 
`PreviousFill → Collect(3 × Exchange)` | true |
   | `SELECT time, tag1, tag2, tag3, diff(s1) AS d FROM table1` | 
`Project[diff(s1)] → Collect(3 × Exchange)` | true |
   | `SELECT time, s1 FROM table1 WHERE tag1='shenzhen' AND tag2='B2' AND 
tag3='ZZ' FILL METHOD PREVIOUS` | `PreviousFill → Collect(3 × Exchange)` | true 
|
   | same, `FILL METHOD LINEAR` | `LinearFill → Collect` | true |
   | same, `FILL METHOD PREVIOUS ORDER BY time` | `Sort → PreviousFill → 
Collect` | true |
   | `SELECT time, diff(s1) AS d FROM table1 WHERE tag1='shenzhen' AND 
tag2='B2' AND tag3='ZZ'` | `Project[diff(s1)] → Collect` | true |
   
   **Why the values go wrong:**
   - In ready mode the Collect hands out one block per ready child in turn; 
`CollectOperatorTest#testAllReadyChildrenHaveLinearSelectionCost` asserts 
exactly that. So a device's rows from one region are interrupted by blocks from 
other regions.
   - Fill without FILL_GROUP and `diff()` carry state over from the previous 
row. After a block boundary, a NULL is filled from a row of whatever block 
arrived in between, and `diff()` is computed against that row. The values can 
change from run to run.
   - This hits multi-device queries without FILL_GROUP, and single devices 
whose time partitions live in different regions.
   - `ORDER BY time` does not help, because the Sort sits above the fill.
   - On base the children were drained one after another. Each region's rows 
stayed contiguous, and the same query returned the same values on every run.
   
   **COPY TO:** the check cannot see order requirements outside its own FI. 
`COPY table1 TO 'x.tsfile'` plans `CopyTo → Exchange` in the root FI, and 
`IdentitySink → Collect(3 × Exchange)` in a child FI that is eligible. The 
TsFile writer rejects non-increasing timestamps per device 
(`AlignedChunkGroupWriterImpl#checkIsHistoryData`). So COPY TO of a device 
whose partitions live in two regions can now fail depending on block arrival 
order. Base already depends on the fixed child order there, but it at least 
behaves the same on every run.
   
   **Coverage:** `IoTDBCollectFanInIT` only has a CONSTANT fill. It has no 
PREVIOUS/LINEAR fill, `diff()` or COPY TO case, so CI does not catch this.
   
   **Suggestion:** decide ordering in the distributed planner instead of 
inferring it from the FI shape on the DataNode.
   - Mark a Collect/Union as unordered only when nothing above it relies on 
input order, in this FI or in the parent FI.
   - Serialize that flag, with ordered as the default, and enable ready 
consumption only for unordered Collects.
   - With an ordered default, plans from a coordinator without this PR keep 
today's semantics during a rolling upgrade.
   - It also lets `visitGroup` and `validateAggregationGrouping` keep their 
streaming forms where no ready Collect is involved (see the comment there).
   
   A narrower fix would be to reject `PreviousFillNode`/`LinearFillNode` 
without grouping keys, and expressions that contain `diff()`, right here. COPY 
TO would still need information from the planner.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java:
##########
@@ -689,6 +689,12 @@ public List<PlanNode> visitGroup(GroupNode node, 
PlanContext context) {
     for (PlanNode child : childrenNodes) {
       if (canSortEliminated(node.getOrderingScheme(), 
nodeOrderingMap.get(child.getPlanNodeId()))) {
         result.add(child);
+      } else if (groupingInvalidatedByCollect(
+          child, node.getOrderingScheme().getOrderBy().subList(0, 
node.getPartitionKeyCount()))) {
+        SortNode sort =
+            new SortNode(queryId.genPlanNodeId(), child, 
node.getOrderingScheme(), false, false);
+        nodeOrderingMap.put(sort.getPlanNodeId(), sort.getOrderingScheme());
+        result.add(sort);

Review Comment:
   Non-blocking. `groupingInvalidatedByCollect` treats every multi-child 
Collect/Union as unordered. But the DataNode only enables ready consumption in 
eligible FIs. In an FI that is not eligible (TopK, joins, more than one 
pipeline), the Collect still drains its children one after another, as on base. 
There, this conversion mostly adds cost, and so does the `preGrouped` reset in 
`validateAggregationGrouping`.
   
   PlanTester, base d2b97bc845 vs 788a995c75. In both queries the root FI has 
`eligible=false` because of the TopK:
   
   ```sql
   -- base: GroupNode, this PR: SortNode (blocking full sort of the Collect 
output)
   SELECT tag1, tag2, tag3, s1, SUM(s1) OVER (PARTITION BY tag1, tag2, tag3 
ORDER BY s1) rs
   FROM (SELECT tag1, tag2, tag3, s1, ROW_NUMBER() OVER () n FROM table1) t
   WHERE n > 0 ORDER BY rs DESC LIMIT 10
   
   -- base: preGrouped=[tag1, tag2, tag3], this PR: preGrouped=[]
   SELECT tag1, tag2, tag3, SUM(s1) AS s
   FROM (SELECT tag1, tag2, tag3, s1, COUNT(*) OVER () AS n FROM table1 WHERE 
tag1 != 'shenzhen') t
   WHERE n > 0 GROUP BY tag1, tag2, tag3 ORDER BY s DESC LIMIT 10
   ```
   
   The second case matters more. `HashAggregationOperator` has no spill path 
(its `spillEnabled` branch throws `UnsupportedOperationException`) and keeps 
every group in memory. So a high-cardinality GROUP BY that streamed on base can 
now run out of operator memory.
   
   If some of these resets are also meant to fix sequential cases, for example 
devices that span regions, please limit them to those cases. Otherwise this 
goes away with the planner-side ordering decision suggested on 
`SharedExchangeMemoryEligibility`.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -241,6 +305,15 @@ public void deRegisterFragmentInstanceFromQueryMemoryMap(
         // forceDeregister == true, after all 
LocalSourceChannel/LocalSourceHandles are closed.
         return;
       }
+      // Pending requests must not outlive their FI registration. Cancel 
before removing the
+      // account so another query's free() cannot retry a request against a 
missing FI. Do not
+      // hold the registration lock while notifying cancellation listeners.
+      if (fragmentRelatedMemory != null) {
+        for (MemoryReservationFuture<Void> future :
+            fragmentRelatedMemory.pendingReservations.values()) {
+          tryCancel(future);
+        }
+      }

Review Comment:
   **Cancelling here breaks the code that owns these requests.** It does fix a 
real base bug. On d2b97bc845 a pending request outlives its FI, and the next 
`free()` from any other query hits it: `tryReserve` throws a 
`NullPointerException`, and the request's bytes stay allocated in the pool for 
good.
   
   The owners (`SourceHandle`, `SinkChannel`, `SharedTsBlockQueue`) rely on two 
rules: a pending future completes only after a successful reservation, and it 
is cancelled only by the owner itself through `tryCancel()`, which returns the 
pending bytes. This cancellation breaks both rules.
   
   **Probe setup:** a 64 B pool fully held by query `other`. A `SourceHandle` 
for (`q`, `0.0`) has one 64 B block pending. Then 
`deRegisterFragmentInstanceFromQueryMemoryMap("q", "0.0", true)`, then `free()` 
by `other`, then `handle.abort()`:
   
   | | base d2b97bc845 | 788a995c75 |
   |---|---|---|
   | `free()` by `other` | NPE; 64 B stay reserved with no live reservation | 
ok |
   | `handle.abort()` | ok, `onAborted` called | `IllegalArgumentException: 
RelatedMemoryReserved can't be null when freeing memory`; `aborted` stays 
false, `onAborted` not called |
   
   **What happens on 788a995c75:**
   - The cancellation runs the handle's `blockedOnMemory` listener while the 
handle is still live. That submits a `GetDataBlocksTask` for a block that has 
no reservation.
   - Later, `tryCancel()` returns 0 for the already-cancelled future. So 
`abort()` tries to `free()` the pending bytes against the removed account, and 
throws.
   - In `MPPDataExchangeManager#forceDeregisterFragmentInstance`, that 
exception escapes the loop; `DriverScheduler` only logs it. The FI's remaining 
handles are not aborted, and its `sourceHandles` entry is not removed.
   
   **Same pattern elsewhere (from reading the code):**
   - `SinkChannel.abort()/close()` call `tryCancel(blocked)` and then `free`.
   - In `SharedTsBlockQueue`, the memory listener enqueues the TsBlock with 
`reservedBytes` that were never reserved.
   
   
`SourceHandleTest#reproduceAbortAfterFragmentInstanceMemoryIsForceDeregistered` 
already covers the case where the handle holds reserved bytes. This change 
extends the failure to handles that only have a pending request.
   
   **Suggestion:** detach instead of cancel. Remove the FI's requests from both 
indexes without completing them. Then no listener fires, and the owner's later 
`tryCancel()` still returns the pending bytes. If you want to keep cancelling, 
the three owners need to treat a future cancelled by the pool as unreserved: no 
fetch or enqueue in the listener, and no `free` of its bytes. Either way, 
please add a regression test: a real `SourceHandle` blocked on memory, then 
force deregistration, then `abort()`.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -390,66 +472,103 @@ public void free(String queryId, String 
fragmentInstanceId, String planNodeId, l
     Validate.notNull(queryId, 
DataNodeQueryMessages.EXCEPTION_QUERYID_CAN_NOT_BE_NULL_DOT_16639DBE);
     Validate.isTrue(bytes > 0L);
 
-    try {
-      queryMemoryReservations
-          .get(queryId)
-          .get(fragmentInstanceId)
-          .computeIfPresent(
-              planNodeId,
-              (k, reservedMemory) -> {
-                if (reservedMemory < bytes) {
-                  throw new IllegalArgumentException(
-                      
DataNodeQueryMessages.FREE_MORE_MEMORY_THAN_HAS_BEEN_RESERVED);
-                }
-                return reservedMemory - bytes;
-              });
-    } catch (NullPointerException e) {
+    Map<String, FragmentMemoryReservation> fragments = 
queryMemoryReservations.get(queryId);
+    FragmentMemoryReservation fragment =
+        fragments == null ? null : fragments.get(fragmentInstanceId);
+    if (fragment == null) {
       throw new IllegalArgumentException(
           DataNodeQueryMessages
-              
.QUERY_EXCEPTION_RELATEDMEMORYRESERVED_CAN_T_BE_NULL_WHEN_FREEING_MEMORY_C80009F2,
-          e);
+              
.QUERY_EXCEPTION_RELATEDMEMORYRESERVED_CAN_T_BE_NULL_WHEN_FREEING_MEMORY_C80009F2);
     }
-
-    memoryBlock.release(bytes);
+    PlanNodeReservation reservation =
+        requireReservation(
+            fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, 
planNodeId);
+    Object accountingLock = fragment.sharedBudget ? fragment : reservation;
+    synchronized (accountingLock) {
+      if (reservation.bytesReserved < bytes) {
+        throw new IllegalArgumentException(
+            DataNodeQueryMessages.FREE_MORE_MEMORY_THAN_HAS_BEEN_RESERVED);
+      }
+      reservation.bytesReserved -= bytes;
+      fragment.getUsedBytes(reservation.side).addAndGet(-bytes);
+      memoryBlock.release(bytes);
+    }
+    // Complete futures outside the accounting lock: listeners can reserve or 
free other inputs.
 
     if (memoryReservationFutures.isEmpty()) {
       return;
     }
-    Iterator<MemoryReservationFuture<Void>> iterator = 
memoryReservationFutures.iterator();
+    Iterator<MemoryReservationFuture<Void>> iterator = 
memoryReservationFutures.values().iterator();
     while (iterator.hasNext()) {
       MemoryReservationFuture<Void> future = iterator.next();
-      synchronized (future) {
-        if (future.isCancelled() || future.isDone()) {
-          iterator.remove();
-          continue;
-        }
-        long bytesToReserve = future.getBytesToReserve();
-        String curQueryId = future.getQueryId();
-        String curFragmentInstanceId = future.getFragmentInstanceId();
-        String curPlanNodeId = future.getPlanNodeId();
-        long maxBytesCanReserve = future.getMaxBytesCanReserve();
-        if (tryReserve(
-            curQueryId, curFragmentInstanceId, curPlanNodeId, bytesToReserve, 
maxBytesCanReserve)) {
-          future.set(null);
-          iterator.remove();
-        } else {
-          rollbackReserve(curQueryId, curFragmentInstanceId, curPlanNodeId, 
bytesToReserve);
-        }
+      if (tryCompleteReservation(future)) {
+        removePendingReservation(future);
+      }

Review Comment:
   **A waiter whose FI account is gone makes every `free()` on the DataNode 
throw.** The cancellation in `deRegisterFragmentInstanceFromQueryMemoryMap` 
does not shut out `reserveWithPriority`. The race:
   1. Deregistration reads `pendingReservations` (311–316).
   2. A handle then publishes a request (401–402), and its retry at 406 fails 
without throwing.
   3. Deregistration removes the account at 318. The request is still in 
`memoryReservationFutures`.
   
   From then on, every `free()` releases its bytes and then throws from 
`tryCompleteReservation` → `tryReserve` → `getFragmentReservation`. Waiters 
queued behind that request are never reached. The try/catch at 409 only covers 
a retry that throws.
   
   I forced this interleaving in a probe on 788a995c75, by hooking the 
`pendingReservations.values()` read. Another query's `free()` then threw 
`IllegalArgumentException: Memory reservation is not registered for query x, 
fragment 0.0, plan node src`. The pool was back to 0 used bytes, but a later 32 
B waiter from query `y` was never woken. The window is narrow: a handle has to 
still be reserving while its FI is force-deregistered. But the failure lands on 
unrelated queries.
   
   **Suggestion:** in this loop, treat a missing FI account as a dead waiter: 
remove it from the index and continue, instead of throwing. Alternatively, 
publish and deregister under the same `FragmentMemoryReservation` monitor, and 
have `reserveWithPriority` refuse to publish into a removed account.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to