Zoltán Borók-Nagy created IMPALA-15660:
------------------------------------------
Summary: Calcite: a CTE shared by consumers with different
selective filters loses partition pruning (TPC-DS q59 2.2x slower)
Key: IMPALA-15660
URL: https://issues.apache.org/jira/browse/IMPALA-15660
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
TPC-DS q59 uses wss (store_sales x date_dim, grouped by d_week_seq and
ss_store_sk) twice. Each use is joined to date_dim with a different, disjoint
year range. With CTE_THRESHOLD=1, wss becomes one CTE producer with 2 consumers:
* The runtime filters derived from each consumer's date range target the CTE
CONSUMER, which applies them only after the CTE is computed.
* The producer's store_sales scan only gets filters from the unfiltered
date_dim build, so it reads all 8.25B rows at SF3000.
* With CTE_THRESHOLD=0, each inlined copy gets a partition filter on
ss_sold_date_sk and reads about 1.7B rows.
q59 with PLANNER=CALCITE and MT_DOP=8:
* SF3000, 8 executors: 8.5 s with CTE_THRESHOLD=1, 3.8 s with CTE_THRESHOLD=0,
with identical results;
* SF10000, 12 executors: 18.4 s vs 7.5 s.
IMPALA-15652 transfers the OR of the consumers' date filters into the producer.
It keeps one shared producer and brought q59 from 22.6 to 13.5 s at SF10000 in
a separate measurement, but inlining remains faster for q59. This ticket covers
only the inlining. The size limit in IMPALA-15659 does not apply, because wss
has an aggregate.
*Proposal:* in runCTEProgram(), inline a CTE whose consumers apply different
selective filters: each consumer sits on the probe side of a selective inner
join (estimated reduction below about 0.5) on a column that maps, through group
keys or projections, to a base column inside the CTE. When all consumers need
the same rows, as in q64, the shared producer with the transfer from
IMPALA-15652 is better. A cost-based choice between the two could follow.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
