ColinLeeo commented on code in PR #18869: URL: https://github.com/apache/iotdb/pull/18869#discussion_r4279742022
########## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemorySide.java: ########## @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iotdb.db.queryengine.execution.memory; + +/** The exchange budget charged by a registered plan node. */ +public enum MemorySide { + INPUT, + OUTPUT; + + public long getMaxBytes(long fragmentExchangeMaxBytes) { + long inputMaxBytes = fragmentExchangeMaxBytes / 2; + return this == INPUT ? inputMaxBytes : fragmentExchangeMaxBytes - inputMaxBytes; Review Comment: Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63. Shared budgets now require both INPUT and OUTPUT to hold the configured maximum TsBlock. Otherwise the entire FI retains the original handle quotas and independent plan-node admission; exchange handles start with the original full-F limit until planning selects a policy. Sink-only FIs do not opt in, so the F = B boundary keeps its original output quota. Added `testSideSmallerThanOneBlockKeepsLegacyBudget` and `testSinkOnlyFragmentCanStillReserveOneFullBlock` in `LocalExecutionPlanContextBudgetTest`; the targeted suite and both SQL integration environments pass. ########## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/SharedTsBlockQueue.java: ########## @@ -265,7 +267,10 @@ public ListenableFuture<Void> add(TsBlock tsBlock) { localMemoryManager .getQueryPool() .registerPlanNodeIdToQueryMemoryMap( - localFragmentInstanceId.queryId, fullFragmentInstanceId, localPlanNodeId); + localFragmentInstanceId.queryId, + fullFragmentInstanceId, + localPlanNodeId, + MemorySide.OUTPUT); Review Comment: Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63 by excluding internal pipeline chains from shared budgets. Eligibility is decided after all pipeline factories are known, and currently requires exactly one pipeline with no scheduling dependency. This is deliberately conservative: any multi-pipeline FI falls back as a whole, including its original per-plan-node MemoryPool admission, rather than merely changing the handle limit. Added planner fallback coverage and a saturated two-queue forwarding regression using real SharedTsBlockQueues and MemoryPool. Refilling the upstream queue no longer prevents publishing the forwarded block in the downstream queue, and cleanup returns accounting to zero. ########## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java: ########## @@ -460,22 +536,68 @@ public boolean tryReserve( String planNodeId, long bytesToReserve, long maxBytesCanReserve) { + FragmentMemoryReservation fragment = + getFragmentReservation(queryId, fragmentInstanceId, planNodeId); + PlanNodeReservation reservation = + requireReservation( + fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, planNodeId); long tryUsedBytes = memoryBlock.forceAllocateWithoutLimitation(bytesToReserve); - long queryRemainingBytes = - maxBytesCanReserve - - queryMemoryReservations - .get(queryId) - .get(fragmentInstanceId) - .merge(planNodeId, bytesToReserve, Long::sum); - return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && queryRemainingBytes >= 0; + long sideUsedBytes = fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve); Review Comment: Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63. Admission now checks the applicable FI-side or independent plan-node quota under the FI accounting lock, then uses the global memory block's bounded atomic allocation. Counters are updated only after allocation succeeds. Failed attempts publish no speculative charges and require no rollback, removing the crossed global/side increments in this interleaving. Free updates accounting under the same FI lock and completes waiting futures outside it. The post-publication retry and per-future completion/cancellation synchronization remain. Added a contention test with 768/1024 bytes occupied and two concurrent 256-byte requests, repeated 100 times: exactly one request progresses, freeing it completes the other, and accounting returns to zero. ########## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java: ########## @@ -460,22 +536,68 @@ public boolean tryReserve( String planNodeId, long bytesToReserve, long maxBytesCanReserve) { + FragmentMemoryReservation fragment = + getFragmentReservation(queryId, fragmentInstanceId, planNodeId); + PlanNodeReservation reservation = + requireReservation( + fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId, planNodeId); long tryUsedBytes = memoryBlock.forceAllocateWithoutLimitation(bytesToReserve); - long queryRemainingBytes = - maxBytesCanReserve - - queryMemoryReservations - .get(queryId) - .get(fragmentInstanceId) - .merge(planNodeId, bytesToReserve, Long::sum); - return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() && queryRemainingBytes >= 0; + long sideUsedBytes = fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve); + fragment.planNodes.compute( + planNodeId, + (key, value) -> { + value.bytesReserved += bytesToReserve; + return value; + }); + return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() + && sideUsedBytes <= maxBytesCanReserve; Review Comment: Fixed in d3347c349540121589ebd22ce1cb585a1e0dce63 by limiting shared budgets to eligible Collect/MappingCollect FIs. The complete FI plan is checked up to Exchange boundaries; any unsupported multi-input node, including MergeSort or Join, disables sharing for the entire FI even when nested above or below Collect. Those FIs retain both the original handle quotas and independent plan-node admission. MergeSort's consumption algorithm is unchanged. Added mixed-FI eligibility tests and a saturated-budget regression using the real TableMergeSortOperator with simulated inputs backed by MemoryPool. A fills its quota and queues prefetch, but B can still reserve its head block and MergeSort produces output. All 107 targeted unit tests and all 8 IoTDBCollectFanInIT cases in both single-node and three-DataNode environments passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
