Hello Quanlong Huang, Aman Sinha, Jason Fehr, Xuebin Su, Joe McDonnell, Michael
Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24558
to look at the new patch set (#7).
Change subject: IMPALA-15019: Avoid materializing non-projected filtered
partition column.
......................................................................
IMPALA-15019: Avoid materializing non-projected filtered partition column.
This commit is a small optimization to the Calcite planner that already
exists in the original planner. For a query like:
select id from functional_parquet.alltypes where year = 2010
The year column is partitioned. Because of this, there is no reason to
materialize the column in the scan node. The columns where year is not
2010 are contained within the partitioned directory and are pruned out.
Since the year column is not selected, it does not need to show up
anywhere as the data is selected.
To enable this, a "filteredOnlyInputRefs" field was added to the context
that is passed down through the RelNodes at physical creation time. On
the Filter RelNode, it adds any input ref field in the filter condition
that is not projected out through its parent Project RelNode.
In order for this to work, it is worth mentioning that it relies on the
fact that the only way a Project can exist between a Filter and a
TableScan (i.e. the Filter-Project-TableScan hierarchy) is when the
RelFieldTrimmer adds the Project that essentially "trims" the fields
coming out of the scan immediately after the scan occurs. Any Project
that manipulates fields will be on top of the Filter via the
FilterProjectTranspose rule, so the Project underneath the Filter will
only contain non-manipulated RexInputRefs as its project fields.
IMPALA-14815 is a duplicate of this Jira, and some tests have been fixed
in the stats-extrapolation.test file.
Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882
---
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaFilterRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/query_test/test_hbo.py
8 files changed, 138 insertions(+), 97 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/24558/7
--
To view, visit http://gerrit.cloudera.org:8080/24558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882
Gerrit-Change-Number: 24558
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Xuebin Su <[email protected]>
