Copilot commented on code in PR #18869:
URL: https://github.com/apache/iotdb/pull/18869#discussion_r4278867643


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/SharedTsBlockQueue.java:
##########
@@ -265,7 +267,10 @@ public ListenableFuture<Void> add(TsBlock tsBlock) {
       localMemoryManager
           .getQueryPool()
           .registerPlanNodeIdToQueryMemoryMap(
-              localFragmentInstanceId.queryId, fullFragmentInstanceId, 
localPlanNodeId);
+              localFragmentInstanceId.queryId,
+              fullFragmentInstanceId,
+              localPlanNodeId,
+              MemorySide.OUTPUT);

Review Comment:
   Dependent pipeline queues now share the same bounded OUTPUT budget, which 
can prevent forwarding. An upstream queue can fill the budget and queue another 
reservation. When the parent pipeline removes one block, `free()` grants that 
upstream request; the parent then cannot reserve space for the removed block in 
its downstream local queue. SharedTsBlockQueue publishes that block only after 
reservation, while Driver waits on `sink.isFull()` before reading more upstream 
data. The downstream queue stays empty and the upstream queue stays full. 
Reserve forwarding capacity or isolate dependent pipeline budgets, and add a 
saturated two-queue pipeline-chain regression.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
       String planNodeId,
       long bytesToReserve,
       long maxBytesCanReserve) {
+    FragmentMemoryReservation fragment =
+        getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+    PlanNodeReservation reservation =
+        requireReservation(
+            fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, 
planNodeId);
     long tryUsedBytes = 
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
-    long queryRemainingBytes =
-        maxBytesCanReserve
-            - queryMemoryReservations
-                .get(queryId)
-                .get(fragmentInstanceId)
-                .merge(planNodeId, bytesToReserve, Long::sum);
-    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && 
queryRemainingBytes >= 0;
+    long sideUsedBytes = 
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);

Review Comment:
   The separate global and shared-side increments can reject both concurrent 
requests even when one fits. With a 1024-byte pool, 768 bytes held elsewhere, 
and an empty 256-byte INPUT budget, two 256-byte attempts can interleave as 
A-global=1024, B-global=1280, B-side=256, A-side=512. A fails the side check 
and B fails the global check. This schedule can recur during their 
post-publication retries, leaving both futures pending after rollback restores 
256 bytes of capacity. Rollback does not notify waiters, so progress then 
depends on an unrelated release. Make admission and rollback atomic against 
competing reservations, or guarantee a retry when rollback restores capacity, 
and test this deterministic interleaving.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
       String planNodeId,
       long bytesToReserve,
       long maxBytesCanReserve) {
+    FragmentMemoryReservation fragment =
+        getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+    PlanNodeReservation reservation =
+        requireReservation(
+            fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, 
planNodeId);
     long tryUsedBytes = 
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
-    long queryRemainingBytes =
-        maxBytesCanReserve
-            - queryMemoryReservations
-                .get(queryId)
-                .get(fragmentInstanceId)
-                .merge(planNodeId, bytesToReserve, Long::sum);
-    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && 
queryRemainingBytes >= 0;
+    long sideUsedBytes = 
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);
+    fragment.planNodes.compute(
+        planNodeId,
+        (key, value) -> {
+          value.bytesReserved += bytesToReserve;
+          return value;
+        });
+    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes()
+        && sideUsedBytes <= maxBytesCanReserve;

Review Comment:
   The shared INPUT limit can deadlock MergeSort under memory pressure. With a 
256-byte budget and 64-byte blocks, SourceHandle A can reserve four blocks and 
queue its fifth before B requests its first. When MergeSort caches one A block, 
`free()` grants A's earlier request and fills the budget again. MergeSort then 
skips its cached A input and waits for B, so neither input releases the memory 
B needs. The existing prefetch loop and reservation queue permit this schedule; 
ready-input Collect does not protect MergeSort. Preserve capacity for required 
inputs or make reservation scheduling distinguish missing head blocks from 
prefetch, and add a deterministic saturated-budget MergeSort regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to