Steve Carlin has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/24558 )
Change subject: IMPALA-15019: Avoid materializing non-projected filtered partition column in Calcite planner. ...................................................................... IMPALA-15019: Avoid materializing non-projected filtered partition column in Calcite planner. This commit is a small optimization to the Calcite planner that already exists in the original planner. For a query like: select id from functional_parquet.alltypes where year = 2010 The year column is partitioned. Because of this, there is no reason to materialize the column in the scan node. The columns where year is not 2010 are contained within the partitioned directory and are pruned out. Since the year column is not selected, it does not need to show up anywhere as the data is selected. To enable this, a "filteredOnlyInputRefs" field was added to the context that is passed down through the RelNodes at physical creation time. On the Filter RelNode, it adds any input ref field in the filter condition that is not projected out through its parent Project RelNode. In order for this to work, it is worth mentioning that it relies on the fact that the only way a Project can exist between a Filter and a TableScan (i.e. the Filter-Project-TableScan hierarchy) is when the RelFieldTrimmer adds the Project that essentially "trims" the fields coming out of the scan immediately after the scan occurs. Any Project that manipulates fields will be on top of the Filter via the FilterProjectTranspose rule, so the Project underneath the Filter will only contain non-manipulated RexInputRefs as its project fields. IMPALA-14815 is a duplicate of this Jira, and some tests have been fixed in the stats-extrapolation.test file. Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882 --- M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaFilterRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/query_test/test_hbo.py 8 files changed, 136 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/24558/5 -- To view, visit http://gerrit.cloudera.org:8080/24558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I902dc625963321aff57e5c05af3d1137c06aa882 Gerrit-Change-Number: 24558 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin <[email protected]>
