Zoltán Borók-Nagy created IMPALA-15674:
------------------------------------------
Summary: IS NULL / IS NOT NULL on a CTE consumer or inline view
slot gets the default selectivity 0.1 (TPC-DS q17, q25 broadcast 40M-149M-row
builds)
Key: IMPALA-15674
URL: https://issues.apache.org/jira/browse/IMPALA-15674
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
IsNullPredicate.computeSelectivity() (master 1a26e1f778) reads the null count
of a single-column predicate's slot only if the slot belongs to a table. A slot
of a CTE consumer or an inline view has none, so the default selectivity 0.1
applies. The code has a TODO about exactly this:
{code:java}
// TODO: increase this to make sure we don't end up favoring broadcast joins
// due to underestimated cardinalities?
{code}
With PLANNER=CALCITE and CTE_THRESHOLD=1, and the IMPALA-15489 change that
derives IS NOT NULL filters after the CTE rewrite
(https://gerrit.cloudera.org/#/c/25030/, applied in these runs), d_date_sk IS
NOT NULL ends up above a date_dim CTE consumer. In TPC-DS q17 at SF1000 that
cut the date_dim estimate from 274 to 27 rows, so a 40M-row build was costed as
3.6M, and the joins above it broadcast 40M- and 149M-row builds in q17 and q25.
On master the derivation runs before the CTE rewrite, so the predicate stays
inside the CTE on the base table. This needs fixing before or together with
IMPALA-15489; the problem also applies to IS NULL and IS NOT NULL on any inline
view slot.
*Prototype:* the predicate follows single-SlotRef source exprs down to a table
slot and uses its null count; a slot whose stats show no NULLs gets selectivity
1.0 for IS NOT NULL. NULLs added by outer joins inside the view are ignored. It
changes 19 plans.
Measured at SF1000 on 8 executors (MT_DOP=8, PLANNER=CALCITE, CTE_THRESHOLD=1):
the 104-query total went from 345.9 s to 336.0 s on top of IMPALA-15673 and
IMPALA-15631. q17 and q25 no longer broadcast; q17 went from 7.7 s to 1.8 s
with all broadcast fixes.
At SF10000 (12 executors, best of 3), a second prototype that also estimates IS
NOT NULL on aggregate outputs from null fractions instead of 0.1 makes q4's
55M-row join build partitioned instead of broadcast: q4 72.5 -> 36.0 s.
Prototype: https://gerrit.cloudera.org/#/c/25088/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
