wirybeaver opened a new pull request, #19470:
URL: https://github.com/apache/pinot/pull/19470
## Summary
This PR adds the smallest safe foundation for adaptive query execution (AQE)
in Pinot's multi-stage engine: a pure partition-coalescing policy driven by
completed producer partition sizes.
Ballista can re-plan an undispatched consumer after a materialized producer
stage completes. Pinot currently dispatches an eager streaming mailbox graph
whose endpoints already encode worker assignments, so changing stage
parallelism after dispatch would be unsafe. This change therefore implements
only the reusable decision seam and does not mutate a running query.
## Design
```text
Ballista AQE
+--------------------+ +----------------+ +--------------------+
| completed producer | --> | shuffle stats | --> | lazy consumer |
| materialized data | | and replanning | | stage dispatch |
+--------------------+ +----------------+ +--------------------+
Pinot MSE today
+----------------+ eager whole-graph dispatch +------------------+
| physical plan | ---------------------------------> | streaming mailbox|
+----------------+ | topology |
+------------------+
|
immutable after dispatch
X
runtime stage resizing
Safe first Pinot AQE seam
ordered completed-partition byte sizes + target size
|
v
+------------------------------+
| AdaptivePartitionCoalescer |
| deterministic contiguous |
| immutable partition ranges |
+------------------------------+
|
v
future undispatched consumer-stage assignment
```
`AdaptivePartitionCoalescer` walks partitions from left to right and:
- coalesces adjacent small groups using Ballista's production thresholds;
- preserves source order and assigns each partition exactly once;
- keeps oversized source partitions indivisible;
- returns the original assignment when parallelism cannot decrease;
- uses exact intermediate byte totals before threshold comparison;
- returns immutable half-open partition ranges.
The policy is intentionally independent of mailbox, RPC, planner,
worker-assignment, and live-dispatch classes. Applying it to queries remains
deferred until Pinot supports staged dispatch and materialized or late-bound
exchange data.
## Test Plan
- `./mvnw -pl pinot-query-runtime -am -Dtest=AdaptivePartitionCoalescerTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `./mvnw spotless:apply -pl pinot-query-runtime`
- `./mvnw checkstyle:check -pl pinot-query-runtime`
- `./mvnw license:format -pl pinot-query-runtime`
- `./mvnw license:check -pl pinot-query-runtime`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
