wirybeaver opened a new pull request, #19471:
URL: https://github.com/apache/pinot/pull/19471
## Summary
Adds the first executable AQE prerequisite: an opt-in static materialized
HASH exchange. Existing worker counts and placement remain unchanged.
- marks one deterministic eligible HASH exchange when
`materializedExchange=true`
- writes producer-local, consumer-independent logical partition files,
including empty partitions
- reports exact row/byte handles through `OpChainComplete`
- fetches local or remote partitions through a backpressure-aware gRPC stream
- cleans output on consumption, cancellation, failure, and shutdown
- validates complete producer/worker/partition handle coverage at the broker
## Design
```text
producer rows
|
v
existing HashExchange
|
+--> logical p0 file --+
+--> logical p1 file --+--> atomic publish --> OpChainComplete handles
+--> logical pN file --+ |
v
consumer worker p <--- pulls logical p from every producer worker
|
v
existing downstream operators
```
The materialized identity is `(requestId, producerStageId, producerWorkerId,
logicalPartitionId)` and deliberately excludes consumer identity. This PR does
not resize stages or delay dispatch; it establishes the data boundary required
by later staged-dispatch and late-binding phases.
## Safety
- default OFF
- only unsorted, non-prepartitioned, single-consumer HASH exchanges are
eligible
- atomic file publication; incomplete output is never visible
- cleanup/read registration is serialized against cancellation and shutdown
- gRPC output honors readiness and closes abandoned readers
## Test Plan
- `PinotDispatchPlannerTest,PlanNodeSerDeTest`: 140 passed
- materialized mailbox/store/gRPC/operator/session/dispatcher tests: 109
passed
- Spotless, Checkstyle, and license checks passed for `pinot-common`,
`pinot-spi`, `pinot-query-planner`, and `pinot-query-runtime`
## Follow-ups
1. staged dispatch (dependent PR)
2. late-bound materialized routing (dependent PR)
3. runtime partition coalescing and production hardening
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
