ColinLeeo opened a new pull request, #18868:
URL: https://github.com/apache/iotdb/pull/18868

   ## Description
   
   Exchange memory is divided among individual handles, and Collect consumes 
children in sequence. With many inputs, this can strand memory in per-handle 
quotas and leave ready inputs waiting behind a blocked child. This change 
shares each fragment instance's exchange budget by direction and lets Collect 
consume a ready input while preserving query correctness after unordered fan-in.
   
   ### Shared exchange budgets
   
   - Split the fragment exchange budget `F` into fixed `INPUT = F / 2` and 
`OUTPUT = F - INPUT` budgets shared by all pipelines in that fragment.
   - Charge remote sources to the receiving fragment's INPUT budget and 
sinks/local shared queues to the producing fragment's OUTPUT budget. Local 
queues are charged once, independent of creation order.
   - Maintain global, fragment-side, and plan-node accounting across 
reservation, rollback, cancellation, and release. Retry a waiting reservation 
after publishing it so a concurrent release cannot miss the request; 
synchronize completion with release/cancellation to prevent duplicate charging.
   
   ### Ready-input Collect and plan properties
   
   - Collect and MappingCollect observe every live direct child, reuse 
pending-future listeners, and keep the selected child stable during output 
backpressure. Operator methods run on the driver thread; completion callbacks 
only report readiness or failure.
   - Refresh readiness after null/empty results, propagate child failures and 
cancellation, close all inputs, and account for the retained memory of all live 
children. MappingCollect uses the actual selected child's mapping.
   - Clear invalid pre-grouping above unordered Collect/Union so aggregation 
uses HashAggregation. Preserve valid streaming aggregation in ordered branches, 
restore partition sorting when required, and prevent sort elimination across 
unordered fan-in.
   - Derive TopK input-order flags from actual physical inputs and preserve 
runtime filter identities. MergeSort/Join are covered by dependency validation 
only; their consumption algorithms and progress mechanisms are outside this 
change. Any-ready consumption for TableTopK and CountMerge is also outside this 
change.
   
   Design and detailed validation commands: [exchange fan-in 
design](https://github.com/ColinLeeo/iotdb/blob/9e54ec5c95cafe7ed2ea5c01779bf894f5498e27/docs/superpowers/specs/2026-10-14-exchange-fanin-budget-design.md).
   
   ### Validation
   
   - 118 targeted unit tests across 14 classes passed, with no failures or 
skipped tests. Coverage includes a deterministic free-before-enqueue race, 
shared-budget contention and cleanup, Collect with 100/1000 children, 
concurrent readiness, failure/close handling, and 
grouping/ordering/runtime-filter properties.
   - Full-reactor English compilation/tests and Chinese `test-compile 
-DskipTests -P with-zh-locale` passed.
   - Rebuilt and installed the branch's node artifacts, then ran all 7 
`IoTDBCollectFanInIT` tests in both `TableSimpleIT` and `TableClusterIT` (three 
DataNodes). Both runs passed. SQL coverage includes aggregation/DISTINCT, 
Union, nested windows, Fill, TopK, tree/table exchange paths, and join 
dependencies.
   - Spotless, Checkstyle, license checks, and `git diff --check` passed. Maven 
validation used offline mode with rebuilt local artifacts.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed, including concurrent reservation, release, 
cancellation, and readiness paths.
   - [x] added documentation for modified behaviors.
   - [x] added comments explaining non-obvious concurrency and ownership 
decisions.
   - [x] added or updated unit tests.
   - [x] added integration tests.
   - [x] been tested in a test IoTDB cluster.
   
   ##### Key changed/added classes
   
   `MemoryPool`, `MemorySide`, `LocalExecutionPlanContext`, exchange handles 
and `SharedTsBlockQueue`; `AnyChildBlocked`, `CollectOperator`, 
`MappingCollectOperator`; `TableDistributedPlanGenerator`, 
`ParallelizeGrouping`, and `SortElimination`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to