Zoltán Borók-Nagy created IMPALA-15664:
------------------------------------------
Summary: Calcite: invert joins whose probe side runs on fewer
nodes than the build side (TPC-DS q14_2 single-instance join)
Key: IMPALA-15664
URL: https://issues.apache.org/jira/browse/IMPALA-15664
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
For Calcite plans, Planner.invertJoins() skips the cost-based join inversion
(IMPALA-15196), because Calcite has already ordered the joins. But Calcite's
cost model does not know that the probe side sets a join's parallelism.
In TPC-DS q14_2 at SF1000 (PLANNER=CALCITE, CTE_THRESHOLD=1, MT_DOP=8, 8
executors), joins 55 and 85 probe with item, a one-file table read by a single
fragment instance, and build on 20.9M store_sales rows. The join and the
aggregation above it then run on one thread: about 8 s on the critical path.
*Prototype:* a join of a Calcite plan is inverted if its probe side runs on
fewer nodes than its build side and isInvertedJoinCheaper() says the inverted
join is cheaper. q14_2 went from 16.1 s to 13.7 s; the 104-query total from
336.0 s to 333.3 s.
IMPALA-15196 proposes avoiding invertJoins() for Calcite plans altogether; this
case shows that a parallelism-driven inversion is still needed (IMPALA-5612
added that for the original planner).
Prototype: https://gerrit.cloudera.org/#/c/25090/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
