siddharthteotia opened a new pull request, #18551:
URL: https://github.com/apache/pinot/pull/18551
### Improvement Summary
- Adds `PinotJoinCommuteRule` to the MSE logical planner.
- When a user writes `dim JOIN fact`, the rule swaps the inputs so the dim
ends up on the right — fixing both the broadcast-direction and the build-side
memory pressure that the syntactic-default planner gets wrong today
### Problem
- In MSE, `PinotJoinExchangeNodeInsertRule` decides distribution **purely
syntactically**:
- For equi-joins the right side defaults to `BROADCAST`, and the runtime
always builds the hash table from the right (`BaseJoinOperator`).
- So `dim JOIN fact` broadcasts and builds the *fact* table — exactly
backwards.
BEFORE (user-written) AFTER (rule fires)
JOIN JOIN
/ \ / \
dim_tbl fact_tbl fact_tbl dim_tbl
(small) (BIG) (BIG) (small)
Build side: fact (BIG) ❌ Build side: dim (small) ✅
Broadcast: fact (BIG) ❌ Broadcast: dim (small) ✅
The right side is **both** the broadcast side **and** the build side, so one
reordering the JOIN fixes both inefficiencies.
### Solution
- `dim INNER/LEFT/RIGHT/FULL JOIN fact` — auto-commuted so dim ends up on
the right.
- **`dim JOIN fact` with `/*+ join_strategy='lookup' */`** — previously a
runtime error (`LookupJoinOperator` requires dim on the right); now
auto-corrected. Lookup join no longer requires users to remember table order
when writing the query.
- `ResourceBasedQueriesTest` now propagates `isDimTable=true` to the
planner, so dim-aware rules can be exercised end-to-end (was previously a
silent gap).
### Long-term direction
- This rule is intentionally narrow and stats-free — the decision making
code uses the dim-table flag rather than cardinality estimates.
- A full cost-based optimizer will generalize this work (size-based
auto-commute beyond dim/fact, multi-way join reordering...).
- When that lands, the dim-flag predicate will be subsumed by stats-driven
costing, while the surrounding infrastructure (hint-bypass logic, swap
mechanics, replication-aware signals) remains useful.
- Comparable to how Calcite ships both JoinCommuteRule and
LoptOptimizeJoinRule — narrow rules coexist with cost-based rules.
### Testing
- Added PinotJoinCommuteRuleTest — isolated HepPlanner, every predicate
branch including idempotency and the lookup auto-correct case.
- Added Plan-shape tests (PinotJoinCommutePlanTest) — real QueryEnvironment,
asserts rule fires, doesn't over-fire, and respects the off-switch.
- Runtime query correctness via ResourceBasedQueriesTest) — H2-compared
INNER / LEFT / RIGHT / FULL OUTER, filters
- Ran Full pinot-query-planner and ResourceBasedQueriesTest - pass with zero
failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
