Hello Quanlong Huang, Aman Sinha, Jason Fehr, Xuebin Su, Joe McDonnell, Michael
Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24558
to look at the new patch set (#6).
Change subject: IMPALA-15019: Avoid materializing non-projected filtered
partition column in Calcite planner.
......................................................................
IMPALA-15019: Avoid materializing non-projected filtered partition column in
Calcite planner.
This commit is a small optimization to the Calcite planner that already exists
in the original planner. For a query like:
select id from functional_parquet.alltypes where year = 2010
The year column is partitioned. Because of this, there is no reason to
materialize
the column in the scan node. The columns where year is not 2010 are contained
within the partitioned directory and are pruned out. Since the year column is
not
selected, it does not need to show up anywhere as the data is selected.
To enable this, a "filteredOnlyInputRefs" field was added to the context that is
passed down through the RelNodes at physical creation time. On the Filter
RelNode,
it adds any input ref field in the filter condition that is not projected out
through its parent Project RelNode.
In order for this to work, it is worth mentioning that it relies on the fact
that
the only way a Project can exist between a Filter and a TableScan (i.e. the
Filter-Project-TableScan hierarchy) is when the RelFieldTrimmer adds the Project
that essentially "trims" the fields coming out of the scan immediately after the
scan occurs. Any Project that manipulates fields will be on top of the Filter
via
the FilterProjectTranspose rule, so the Project underneath the Filter will only
contain non-manipulated RexInputRefs as its project fields.
IMPALA-14815 is a duplicate of this Jira, and some tests have been fixed in the
stats-extrapolation.test file.
Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882
---
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaFilterRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/query_test/test_hbo.py
8 files changed, 136 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/24558/6
--
To view, visit http://gerrit.cloudera.org:8080/24558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882
Gerrit-Change-Number: 24558
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Xuebin Su <[email protected]>
