Zoltán Borók-Nagy created IMPALA-15659:
------------------------------------------
Summary: Calcite: do not materialize large CTE suggestions without
aggregation that have few consumers (TPC-DS q14_2, q75)
Key: IMPALA-15659
URL: https://issues.apache.org/jira/browse/IMPALA-15659
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
With CTE_THRESHOLD > 0, CalciteOptimizer.runCTEProgram() materializes every
suggested subtree that is referenced at least CTE_THRESHOLD times, whatever its
size. For a large scan/filter/join subtree, buffering the rows and reading them
back costs more than computing them again.
TPC-DS q14_2 at SF1000 materializes 1.65B, 856M and 432M rows of sales x
date_dim, and each CTE is read back twice.
*Prototype:* a suggestion without an Aggregate that is estimated at more than
100M rows and has at most 2 consumers is no longer materialized. Measured at
SF1000 on 8 executors (MT_DOP=8, PLANNER=CALCITE, CTE_THRESHOLD=1):
* q14_2: 13.8 s -> 8.0 s;
* q75: 10.8 s -> 8.9 s.
Without the consumer limit, q14_1 got 11 s slower: its CTEs feed 15 consumers
each and are worth materializing at that scale (but see IMPALA-?).
Both limits are constants in the prototype. A cost-based decision would be
better: rows x consumers x per-row recompute cost against the buffer cost, plus
the buffer size against the memory limit until the buffer can spill
(IMPALA-15313).
Prototype: https://gerrit.cloudera.org/#/c/25092/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
