Zoltán Borók-Nagy created IMPALA-15652:
------------------------------------------
Summary: Calcite: transfer a dimension filter that every consumer
of an aggregating CTE applies into the CTE producer (q64, q59)
Key: IMPALA-15652
URL: https://issues.apache.org/jira/browse/IMPALA-15652
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
A CTE producer computes all its input even when every consumer joins it with a
selectively filtered dimension. External runtime filters cannot reach producer
scans (blockFiltersForCteProducer(), RuntimeFilterGenerator.java:1440-1449,
master 15cc045e2d), and runCTEProgram() (CalciteOptimizer.java:372) ignores
consumer filters.
TPC-DS SF10000, MT_DOP=8, PLANNER=CALCITE, CTE_THRESHOLD=1:
* q64: both cs_ui consumers keep 345 of 402K items; the producer scans all
14.4B catalog_sales rows.
* q59: wss consumers need 2 of 5 years; the producer scans all 27.5B
store_sales rows.
*Proposal:*
* If every consumer i inner-joins group key k to table D filtered by Pi,
runCTEProgram() adds a semi-join:
{noformat}
Aggregate(group by k, ...) over SemiJoin(X, Filter(P1 OR ... OR Pn) over
Scan(D), X.k = d.c)
{noformat}
It drops whole groups, keeping aggregates and HAVING unchanged, and its runtime
filter reaches producer scans.
* Gate (prototype): the OR keeps at most 5% of D and 1M rows; the aggregation
input at least 1M rows.
* Estimate ranges: range Sargs get no estimate
(FilterSelectivityEstimator.java:329-331) and count as 0.1, so q59 fails the
gate. The prototype estimates integer and DATE ranges as values in range / NDV.
If IMPALA-14867 estimates ranges from NDV or low/high stats, this becomes
unnecessary.
*Measured* with a prototype (TPC-DS SF10000, 12 executors, best of 3, identical
results): q64 64 -> 28 s; q59 22.6 -> 13.5 s from adding the range estimate to
the transfer. No other TPC-DS plan changed.
IMPALA-? proposes inlining and, second, this transfer. They are complementary:
the shared producer suits q64, but inlining is faster on q59 (18.4 vs 7.5 s
with CTE_THRESHOLD=1 vs 0, separate session).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
