Zoltán Borók-Nagy created IMPALA-15648:
------------------------------------------
Summary: Parquet and ORC scanners evaluate runtime filters on
partition columns for every row
Key: IMPALA-15648
URL: https://issues.apache.org/jira/browse/IMPALA-15648
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
Parquet and ORC scanners add every runtime filter of the scan to their
row-level filters (HdfsParquetScanner::Open(), HdfsOrcScanner::Open()), and
HdfsScanner::EvalRuntimeFilters(), interpreted and codegen'd, probes each of
them for every row (master 9c8bcc9585). This includes filters whose target only
reads partition columns. Such a filter has the same result for all rows of a
partition, and HdfsScanNodeBase::PartitionPassesFilters() already applies it to
the partition values before each file, split and row batch, so the per-row
probes only repeat that check.
*Proposal:* for non-Iceberg tables, mark the filters whose target expression
only reads partition key slots in HdfsScanPlanNode::Init() and skip them in the
interpreted and codegen'd EvalRuntimeFilters(). Their row counters in the
profile then stay 0. Iceberg tables are excluded because their partition values
come from the data files.
*Measured* on TPC-DS SF100 with 5 executors (MT_DOP=0): a prototype that does
this and also checks filter effectiveness by the number of rows evaluated
instead of every 16 output batches (see IMPALA-14510) was 4.2% faster in total
(1.8% without q72 and q78), geomean 0.94. The two parts were not measured
separately.
Prototype: https://gerrit.cloudera.org/#/c/25202/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
