Zoltán Borók-Nagy created IMPALA-15686:
------------------------------------------
Summary: Calcite planner: planning is slower than the original
planner on 101 of 104 TPC-DS queries, also without repeated subtrees
Key: IMPALA-15686
URL: https://issues.apache.org/jira/browse/IMPALA-15686
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
With PLANNER=CALCITE, planning ("Planning finished" in the query timeline)
takes longer than with the original planner, a fixed cost that hurts short
queries most.
*Data* (TPC-DS SF10000, 12 executors, master 9a6a6b1432, MT_DOP=8,
CTE_THRESHOLD default -1, best of 3, all metadata cached; q14_1, 16.0 s vs 2.9
s, is IMPALA-15523 and excluded):
* 103 queries: 18.8 s vs 11.9 s, median 128 vs 84 ms. The 62 queries under 10 s
with Calcite: median 85 vs 57 ms.
* The Calcite steps (RelNode conversion to "Created final Impala convert plan")
take 7.5 s, more than the 6.8 s difference: "post join" pass 3.2 s, "pre join"
pass 2.4 s, join ordering 1.0 s, rest 0.8 s. All other planning steps: 11.3 s
vs 11.9 s.
* 9 of the 10 queries with the most optimizer time (q64 789 ms, q5 515 ms, q80
513 ms, ...) inline a WITH clause several times or have EXISTS subqueries
(IMPALA-15523).
* Queries without WITH clause or subquery are slower too (27 of 28; 2.46 s vs
1.69 s): q13 315 vs 74 ms, q48 171 vs 54 ms, q85 175 vs 74 ms.
* At SF100, with a DAG-mode prototype for IMPALA-15523, CTE_THRESHOLD=1 and
other pending changes, Calcite planning still averaged 103 ms on 91 short
queries vs 57 ms with the original planner.
*Cause* (master c580d36e61): CalciteOptimizer runs the same TOP_DOWN HepPlanner
program (22-23 rules) before and after join ordering
(CalciteOptimizer.java:174-189, :228-270), each run wrapped in two
RelFieldTrimmer passes (:233, :270). The profile has no per-rule breakdown.
*Proposal:*
* Profile planning per rule off-cluster (e.g. TpcdsCpuCostPlannerTest),
starting with q13, q48 and q85.
* Report rule firings and time per program in the profile.
* Limit the post-join pass to rules that join ordering can enable, or skip it
when join ordering changed nothing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
