wirybeaver opened a new issue, #19485:
URL: https://github.com/apache/pinot/issues/19485
## Goal
Build an incremental Adaptive Query Runner PoC for Pinot's multi-stage query
engine. The work first proves a runtime-informed execution decision at an
existing pipeline breaker, then adds the execution seams required to adapt an
undispatched downstream stage after its producer completes.
## Roadmap
- [ ] **AQE-PR-1 — Minimal executable AQE rule:** #19470
- Skip probe-side leaf execution when a completed dynamic-broadcast SEMI
JOIN build side is empty.
- Demonstrates a real runtime decision without changing the stage graph or
worker routing.
- [ ] **AQE-PR-2 — Materialized exchange:** #19471
- Add opt-in, consumer-independent materialized HASH partitions.
- Publish exact output handles and row/byte statistics after producer
completion.
- [ ] **AQE-PR-3 — Staged dispatch:** #19472
- Dispatch stages in dependency order.
- Hold downstream stages behind all-worker completion barriers while
preserving the original assignments.
- [ ] **AQE-PR-4 — Late-bound routing:** #19483
- Validate the completed producer's exact partition-handle coverage.
- Bind materialized inputs to downstream workers immediately before
serialization and dispatch.
## Execution model
```text
producer stage completes
|
v
materialized partitions + runtime statistics
|
v
completion barrier releases downstream stage
|
v
late-bind partition handles to consumer workers
|
v
dispatch the ready consumer stage
```
## PoC boundaries
- Opt-in only; existing execution remains the default.
- No join reordering.
- No dynamic stage insertion or removal.
- No cancellation or replacement of running stages.
- No retries or stage-attempt fencing in this PoC.
- AQE-PR-2 through AQE-PR-4 retain the planned worker counts; they establish
the seams needed for a later deterministic `M -> K` partition-coalescing rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
