Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23595
to look at the new patch set (#10).
Change subject: IMPALA-14510: Disable ineffective runtime filters more
aggressively
......................................................................
IMPALA-14510: Disable ineffective runtime filters more aggressively
HdfsScanner::GetNext() does a check for runtime filter effectiveness
every 16 row batches. When runtime filters are evaluated at the row
level, a highly selective runtime filter may eliminate so many rows
that GetNext() is rarely returning a row batch. When an ineffective
runtime filter is paired with this highly selective filter, the
ineffective filter may never get disabled and could be evaluated on
millions of rows with no benefit. For example, on a run of TPC-DS Q13
at scale, we see this in the profile:
- Rows processed: 47.46M (47458959)
- Rows rejected: 0 (0)
- Rows total: 47.46M (47458959)
This adds an additional check for runtime filter effectiveness that
runs every 65k unfiltered rows. This means that ineffective filters
get disabled even if they are paired with a highly selective filter.
This disables some ineffective filters on both TPC-H and TPC-DS
without a noticeable overhead. Runtime filters are disabled
independently for each file, so there may still be some room for
further improvement by remembering bad filters across scan range
boundaries.
Testing:
- Ran TPC-H / TPC-DS benchmarks
Change-Id: Iab48fe610b7784ea34c0df729e4e5207712efedd
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
3 files changed, 35 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/23595/10
--
To view, visit http://gerrit.cloudera.org:8080/23595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iab48fe610b7784ea34c0df729e4e5207712efedd
Gerrit-Change-Number: 23595
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
