wirybeaver opened a new pull request, #19483:
URL: https://github.com/apache/pinot/pull/19483
## Summary
Adds Phase 3 of the AQE prerequisite stack: late-bound routing for
materialized exchange inputs.
Depends on #19471 and #19472. This PR targets `master`, so its GitHub diff
is cumulative; the Phase 3-specific commit is `e6b5888d84` on top of the
staged-dispatch branch.
- carries materialized input handles through worker metadata and plan
serialization
- validates exact producer-worker × logical-partition coverage before routing
- binds completed producer outputs to downstream workers immediately before
dispatch
- makes materialized receive operators read the assigned handles while
retaining the static fallback
- keeps stage IDs, worker counts, and worker placement unchanged
## Design
```text
completed producer workers
|
v
P x M materialized handles
|
v
exact coverage validation
|
+--> logical partition 0 --> consumer worker 0
+--> logical partition 1 --> consumer worker 1
+--> logical partition M - 1 --> consumer worker M - 1
|
v
bind ready consumer metadata
|
v
serialize and dispatch stage
|
v
assigned-handle file readers
```
For each materialized producer/consumer boundary, the broker waits for the
producer stage to complete, validates a dense and duplicate-free `P x M` handle
set, then assigns every producer's logical partition `p` to consumer worker
`p`. Binding happens before the dispatch group is marked dispatched and before
its worker requests are serialized.
The router preserves each consumer worker's placement, mailbox, and custom
metadata. It rejects request/stage mismatches, invalid producer or partition
IDs, duplicate handles, incomplete coverage, non-dense consumer worker IDs, and
rebinding.
## Scope
This phase establishes the late-binding seam only. It does not yet change
downstream parallelism, reorder joins, add or remove stages, retry stages, or
cancel running stages.
## Test Plan
- `QueryPlanSerDeUtilsTest`: 1 passed
- `MaterializedPartitionRouterTest`, `MaterializedMailboxOperatorTest`,
`QueryDispatcherTest`: 85 passed
- Spotless, Checkstyle, and license checks passed for `pinot-common`,
`pinot-query-planner`, and `pinot-query-runtime`
- `git diff --check` passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
