shrivasshankar opened a new pull request, #27148:
URL: https://github.com/apache/datafusion/pull/27148
## Which issue does this PR close?
- Closes #27146.
- Re-lands #25790, which was reverted in #27145 because of #27108.
## Rationale for this change
#25790 let filters on decimal columns use interval analysis instead of the
default 20% selectivity. It caused a big TPC-H Q18 regression (#27108), so it
was reverted.
The problem was that Q18's `HAVING sum(l_quantity) > 300` filters an
aggregate
output, which has no min/max stats. Without bounds, interval analysis assumes
every row passes, so the filter estimate went from 20% to 100%. The subquery
then looked bigger than the joined orders, and the semi join built its hash
table on the wrong side (LeftSemi instead of RightSemi).
## What changes are included in this PR?
- The first commit is #25790 unchanged.
- The second commit is the fix: `FilterExec` only uses interval analysis for
a
decimal predicate when every decimal column in it has both min and max
stats. Otherwise it keeps the old estimate, as before #25790. Other types
are not affected.
## What is the testing strategy for this PR?
- `test_filter_statistics_decimal_without_bounds` checks that a decimal
column
with no stats, or only a min, keeps the default selectivity, on its own and
together with a bounded integer column.
- `filter_statistics_decimal.slt` now has a Q18 plan check on small
TPC-H-shaped parquet tables. It checks that the HAVING filter is estimated
at 20% and that the semi join is RightSemi. Without the fix, both checks
fail (100% and LeftSemi).
- EXPLAIN plans for all 22 TPC-H queries on SF1 and SF10 parquet data are
identical to main. With #25790 alone, only Q18 changes.
TPC-H Q18, SF1:
| | HAVING filter estimate | Semi join | Peak memory |
| ------- | ---------------------: | --------- | ----------: |
| #25790 | 6.0M (100%) | LeftSemi | ~1.25 GB |
| main | 1.2M (20%) | RightSemi | ~0.4 GB |
| This PR | 1.2M (20%) | RightSemi | ~0.4 GB |
The Q6-style query from #25613 keeps the improvement: 111,291 estimated rows
(actual 114,160), vs 1,200,243 on main.
Worth a closer look in review:
- Integer columns without min/max stats still get a selectivity of 1.0, as
they did before #25790. I kept this PR to the decimal case so no existing
plans change. Falling back to the default for all types could be a
follow-up.
- If a predicate mixes a bounded column with a decimal column that has no
stats, the whole predicate uses the old estimate, again matching the
behavior before #25790.
## Are there any user-facing changes?
Same as #25790: better row estimates for filters on decimal columns that have
min/max stats, and a new public function, `check_statistics_support`. Decimal
filters without stats keep the default selectivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
