ColinLeeo commented on code in PR #18869:
URL: https://github.com/apache/iotdb/pull/18869#discussion_r4279742022


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemorySide.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.memory;
+
+/** The exchange budget charged by a registered plan node. */
+public enum MemorySide {
+  INPUT,
+  OUTPUT;
+
+  public long getMaxBytes(long fragmentExchangeMaxBytes) {
+    long inputMaxBytes = fragmentExchangeMaxBytes / 2;
+    return this == INPUT ? inputMaxBytes : fragmentExchangeMaxBytes - 
inputMaxBytes;

Review Comment:
   Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63. Shared budgets now 
require both INPUT and OUTPUT to hold the configured maximum TsBlock. Otherwise 
the entire FI retains the original handle quotas and independent plan-node 
admission; exchange handles start with the original full-F limit until planning 
selects a policy. Sink-only FIs do not opt in, so the F = B boundary keeps its 
original output quota.
   
   Added `testSideSmallerThanOneBlockKeepsLegacyBudget` and 
`testSinkOnlyFragmentCanStillReserveOneFullBlock` in 
`LocalExecutionPlanContextBudgetTest`; the targeted suite and both SQL 
integration environments pass.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/SharedTsBlockQueue.java:
##########
@@ -265,7 +267,10 @@ public ListenableFuture<Void> add(TsBlock tsBlock) {
       localMemoryManager
           .getQueryPool()
           .registerPlanNodeIdToQueryMemoryMap(
-              localFragmentInstanceId.queryId, fullFragmentInstanceId, 
localPlanNodeId);
+              localFragmentInstanceId.queryId,
+              fullFragmentInstanceId,
+              localPlanNodeId,
+              MemorySide.OUTPUT);

Review Comment:
   Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63 by excluding internal 
pipeline chains from shared budgets. Eligibility is decided after all pipeline 
factories are known, and currently requires exactly one pipeline with no 
scheduling dependency. This is deliberately conservative: any multi-pipeline FI 
falls back as a whole, including its original per-plan-node MemoryPool 
admission, rather than merely changing the handle limit.
   
   Added planner fallback coverage and a saturated two-queue forwarding 
regression using real SharedTsBlockQueues and MemoryPool. Refilling the 
upstream queue no longer prevents publishing the forwarded block in the 
downstream queue, and cleanup returns accounting to zero.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
       String planNodeId,
       long bytesToReserve,
       long maxBytesCanReserve) {
+    FragmentMemoryReservation fragment =
+        getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+    PlanNodeReservation reservation =
+        requireReservation(
+            fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, 
planNodeId);
     long tryUsedBytes = 
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
-    long queryRemainingBytes =
-        maxBytesCanReserve
-            - queryMemoryReservations
-                .get(queryId)
-                .get(fragmentInstanceId)
-                .merge(planNodeId, bytesToReserve, Long::sum);
-    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && 
queryRemainingBytes >= 0;
+    long sideUsedBytes = 
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);

Review Comment:
   Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63. Admission now checks the 
applicable FI-side or independent plan-node quota under the FI accounting lock, 
then uses the global memory block's bounded atomic allocation. Counters are 
updated only after allocation succeeds. Failed attempts publish no speculative 
charges and require no rollback, removing the crossed global/side increments in 
this interleaving.
   
   Free updates accounting under the same FI lock and completes waiting futures 
outside it. The post-publication retry and per-future completion/cancellation 
synchronization remain. Added a contention test with 768/1024 bytes occupied 
and two concurrent 256-byte requests, repeated 100 times: exactly one request 
progresses, freeing it completes the other, and accounting returns to zero.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
       String planNodeId,
       long bytesToReserve,
       long maxBytesCanReserve) {
+    FragmentMemoryReservation fragment =
+        getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+    PlanNodeReservation reservation =
+        requireReservation(
+            fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, 
planNodeId);
     long tryUsedBytes = 
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
-    long queryRemainingBytes =
-        maxBytesCanReserve
-            - queryMemoryReservations
-                .get(queryId)
-                .get(fragmentInstanceId)
-                .merge(planNodeId, bytesToReserve, Long::sum);
-    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && 
queryRemainingBytes >= 0;
+    long sideUsedBytes = 
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);
+    fragment.planNodes.compute(
+        planNodeId,
+        (key, value) -> {
+          value.bytesReserved += bytesToReserve;
+          return value;
+        });
+    return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes()
+        && sideUsedBytes <= maxBytesCanReserve;

Review Comment:
   Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63 by limiting shared budgets 
to eligible Collect/MappingCollect FIs. The complete FI plan is checked up to 
Exchange boundaries; any unsupported multi-input node, including MergeSort or 
Join, disables sharing for the entire FI even when nested above or below 
Collect. Those FIs retain both the original handle quotas and independent 
plan-node admission. MergeSort's consumption algorithm is unchanged.
   
   Added mixed-FI eligibility tests and a saturated-budget regression using the 
real TableMergeSortOperator with simulated inputs backed by MemoryPool. A fills 
its quota and queues prefetch, but B can still reserve its head block and 
MergeSort produces output. All 107 targeted unit tests and all 8 
IoTDBCollectFanInIT cases in both single-node and three-DataNode environments 
passed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to