Copilot commented on code in PR #18869:
URL: https://github.com/apache/iotdb/pull/18869#discussion_r4278867643
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/SharedTsBlockQueue.java:
##########
@@ -265,7 +267,10 @@ public ListenableFuture<Void> add(TsBlock tsBlock) {
localMemoryManager
.getQueryPool()
.registerPlanNodeIdToQueryMemoryMap(
- localFragmentInstanceId.queryId, fullFragmentInstanceId,
localPlanNodeId);
+ localFragmentInstanceId.queryId,
+ fullFragmentInstanceId,
+ localPlanNodeId,
+ MemorySide.OUTPUT);
Review Comment:
Dependent pipeline queues now share the same bounded OUTPUT budget, which
can prevent forwarding. An upstream queue can fill the budget and queue another
reservation. When the parent pipeline removes one block, `free()` grants that
upstream request; the parent then cannot reserve space for the removed block in
its downstream local queue. SharedTsBlockQueue publishes that block only after
reservation, while Driver waits on `sink.isFull()` before reading more upstream
data. The downstream queue stays empty and the upstream queue stays full.
Reserve forwarding capacity or isolate dependent pipeline budgets, and add a
saturated two-queue pipeline-chain regression.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
String planNodeId,
long bytesToReserve,
long maxBytesCanReserve) {
+ FragmentMemoryReservation fragment =
+ getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+ PlanNodeReservation reservation =
+ requireReservation(
+ fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId,
planNodeId);
long tryUsedBytes =
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
- long queryRemainingBytes =
- maxBytesCanReserve
- - queryMemoryReservations
- .get(queryId)
- .get(fragmentInstanceId)
- .merge(planNodeId, bytesToReserve, Long::sum);
- return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() &&
queryRemainingBytes >= 0;
+ long sideUsedBytes =
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);
Review Comment:
The separate global and shared-side increments can reject both concurrent
requests even when one fits. With a 1024-byte pool, 768 bytes held elsewhere,
and an empty 256-byte INPUT budget, two 256-byte attempts can interleave as
A-global=1024, B-global=1280, B-side=256, A-side=512. A fails the side check
and B fails the global check. This schedule can recur during their
post-publication retries, leaving both futures pending after rollback restores
256 bytes of capacity. Rollback does not notify waiters, so progress then
depends on an unrelated release. Make admission and rollback atomic against
competing reservations, or guarantee a retry when rollback restores capacity,
and test this deterministic interleaving.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/memory/MemoryPool.java:
##########
@@ -460,22 +536,68 @@ public boolean tryReserve(
String planNodeId,
long bytesToReserve,
long maxBytesCanReserve) {
+ FragmentMemoryReservation fragment =
+ getFragmentReservation(queryId, fragmentInstanceId, planNodeId);
+ PlanNodeReservation reservation =
+ requireReservation(
+ fragment.planNodes.get(planNodeId), queryId, fragmentInstanceId,
planNodeId);
long tryUsedBytes =
memoryBlock.forceAllocateWithoutLimitation(bytesToReserve);
- long queryRemainingBytes =
- maxBytesCanReserve
- - queryMemoryReservations
- .get(queryId)
- .get(fragmentInstanceId)
- .merge(planNodeId, bytesToReserve, Long::sum);
- return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes() &&
queryRemainingBytes >= 0;
+ long sideUsedBytes =
fragment.getUsedBytes(reservation.side).addAndGet(bytesToReserve);
+ fragment.planNodes.compute(
+ planNodeId,
+ (key, value) -> {
+ value.bytesReserved += bytesToReserve;
+ return value;
+ });
+ return tryUsedBytes <= memoryBlock.getTotalMemorySizeInBytes()
+ && sideUsedBytes <= maxBytesCanReserve;
Review Comment:
The shared INPUT limit can deadlock MergeSort under memory pressure. With a
256-byte budget and 64-byte blocks, SourceHandle A can reserve four blocks and
queue its fifth before B requests its first. When MergeSort caches one A block,
`free()` grants A's earlier request and fills the budget again. MergeSort then
skips its cached A input and waits for B, so neither input releases the memory
B needs. The existing prefetch loop and reservation queue permit this schedule;
ready-input Collect does not protect MergeSort. Preserve capacity for required
inputs or make reservation scheduling distinguish missing head blocks from
prefetch, and add a deterministic saturated-budget MergeSort regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]