wirybeaver opened a new pull request, #19472:
URL: https://github.com/apache/pinot/pull/19472
## Summary
Adds Phase 2 of the AQE prerequisite stack: opt-in staged dispatch with
unchanged worker assignments.
Depends on #19471.
## Design
```text
materialized dependency
producer group A ----------------------------+
| all unique workers complete |
| all streams emit ServerDone v
+----------------------------------> consumer group B
|
root group contains 0?
|
run reducer locally
streaming / pipeline-breaker edge:
stage X ===== contracted dispatch group ===== stage Y
```
The broker builds a deterministic dispatch-group DAG. Materialized edges
form completion barriers; streaming-connected stages are dispatched together. A
group transitions only after every expected `(stageId, workerId)` reports
success and every stream closes. Duplicate or unexpected reports cannot release
the barrier.
## Invariants
- default OFF via `stagedDispatch=true`
- one absolute deadline across all waves
- assignments, stage IDs, and worker IDs remain immutable
- a failed worker or stream prevents downstream dispatch
- stage 0 participates in dependency ordering but executes locally
- no retries or running-stage replacement
## Test Plan
- graph/session/observer/dispatcher tests: 110 passed
- Spotless, Checkstyle, and license checks passed for `pinot-common`,
`pinot-spi`, and `pinot-query-runtime`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
