Copilot commented on code in PR #18747:
URL: https://github.com/apache/iotdb/pull/18747#discussion_r4129094706
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/source/SourceHandle.java:
##########
@@ -305,7 +309,10 @@ private synchronized void trySubmitGetDataBlocksTask() {
long reservedBytes = 0L;
Pair<ListenableFuture<Void>, Boolean> pair = null;
long blockedSize = 0L;
- while (sequenceIdToDataBlockSize.containsKey(endSequenceId)) {
+ // Keep one in-flight block per source handle when it participates in a
shared Collect quota.
+ // Standalone exchanges retain their existing batching behavior.
+ while (sequenceIdToDataBlockSize.containsKey(endSequenceId)
+ && (!usesSharedMemoryReservationGroup || endSequenceId ==
startSequenceId)) {
Review Comment:
The one-block guard resets `startSequenceId` to `nextSequenceId` on every
invocation, so every new `TNewDataBlockEvent` can reserve and fetch another
block before the previous one is consumed. `SinkChannel.send()` emits one such
event per block, allowing a single remote child to accumulate many in-flight
blocks and monopolize the shared Collect quota. Also require `nextSequenceId ==
currSequenceId` before admitting the grouped handle's one block.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java:
##########
@@ -261,16 +270,50 @@ public void addExchangeOperator(ExchangeOperator
exchangeOperator) {
this.exchangeOperatorList.add(exchangeOperator);
}
+ public CollectQuotaGroup enterCollectQuotaGroup(String planNodeId) {
+ CollectQuotaGroup group = new CollectQuotaGroup("collect-fan-in-" +
planNodeId);
+ collectQuotaGroups.add(group);
+ activeCollectQuotaGroups.push(group);
+ return group;
+ }
+
+ public void exitCollectQuotaGroup(CollectQuotaGroup group) {
+ if (activeCollectQuotaGroups.peek() != group) {
+ throw new IllegalStateException();
+ }
+ activeCollectQuotaGroups.pop();
+ }
+
+ public void registerExchangeSourceHandle(ISourceHandle sourceHandle) {
+ if (!activeCollectQuotaGroups.isEmpty()) {
+ activeCollectQuotaGroups.peek().addSourceHandle(sourceHandle);
+ }
+ }
+
public void setMaxBytesOneHandleCanReserve() {
long maxBytesOneHandleCanReserve = getMaxBytesOneHandleCanReserve();
LOGGER.debug(
DataNodeQueryMessages
.MAXBYTESONEHANDLECANRESERVE_FOR_EXCHANGEOPERATOR_IS_ARG_EXCHANGESUMNUM_IS_ARG,
maxBytesOneHandleCanReserve,
exchangeSumNum);
+ java.util.Set<ISourceHandle> groupedSourceHandles =
+ Collections.newSetFromMap(new IdentityHashMap<>());
+ long maxBytesPerFI =
+
IoTDBDescriptor.getInstance().getMemoryConfig().getMaxBytesPerFragmentInstance();
+ for (CollectQuotaGroup group : collectQuotaGroups) {
+ for (ISourceHandle sourceHandle : group.getSourceHandles()) {
+ groupedSourceHandles.add(sourceHandle);
+
sourceHandle.setMemoryReservationGroupId(group.getReservationPlanNodeId());
+ sourceHandle.setMaxBytesCanReserve(maxBytesPerFI);
Review Comment:
Each Collect group gets a distinct reservation key and the full
`maxBytesPerFragmentInstance`. `MemoryPool` enforces the supplied limit per
plan-node key, not across all keys, so a fragment with two Collect groups (or
one group plus ungrouped exchanges) can reserve more than its fragment-instance
data-exchange budget. The full budget must be partitioned across reservation
scopes, or all scopes must additionally share an FI-wide cap.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java:
##########
@@ -2956,6 +2956,12 @@ private static OrderingScheme
constructOrderingSchema(List<Symbol> symbols) {
return new OrderingScheme(symbols, orderings);
}
+ private static boolean isStreamingAggregationEnabled(AggregationNode node) {
+ // Keep an explicit ordering boundary for plans whose logical grouping
property is streamable.
+ // The local operator generator still chooses blocking HashAggregation for
these inputs.
+ return node.isStreamable();
Review Comment:
This helper has no callers, so it does not implement the stated
blocking-aggregation boundary. Physical final aggregations produced by
`split()` retain `preGroupedSymbols`, and `TableOperatorGenerator` therefore
still creates a streaming aggregation above a Collect. Since Collect now
interleaves ready children, equal groups from one child can be separated by
another child and emitted as duplicate groups. Clear the physical aggregation's
pre-grouped symbols when its input may be Collect, while preserving ordering
metadata separately where needed.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/source/SourceHandle.java:
##########
@@ -677,6 +701,7 @@ public void run() {
indexOfUpstreamSinkHandle);
}
if (tsBlockNum == 0) {
+ fail(new IllegalStateException());
Review Comment:
An empty response now fails the fragment with an `IllegalStateException`
whose message is null, hiding the actual exchange failure from diagnostics. Use
the existing localized unexpected-response-size message, as the non-empty
mismatch path already does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]