Zach Amsden has uploaded a new patch set (#12). Change subject: IMPALA-5003: Constant propagation in scan nodes and inline views ......................................................................
IMPALA-5003: Constant propagation in scan nodes and inline views When conjuncts are pushed into table refs and inline views, they can be considered for constant progagation within that node. In certain cases, we might end up with a FALSE conditional and now we can convert ScanNodes to EmptySet nodes when that occurs. I also added an inequality collation phase which is now partially tested and will combine conjuncts such as a < k1, a < k2 into a < min(k1, k2), as well as detect equivalence from a >= k, a <= k, and determine conflicting bounds requirements to be false. This could be expanded to do analysis against other slotrefs in the future, but this should probably be saved for another diff. Testing: Expanded the test cases for the planner to achieve constant propagation. Added Kudu, datasource, Hdfs and HBase tests to validate we can create EmptySetNodes. Query: explain select * from functional_hbase.widetable_250_cols a where a.int_col1 > 1 and a.int_col1 <= 20 and a.int_col1 < 50 and a.int_col1 > 2 +------------------------------------------------------------------- | Explain String +------------------------------------------------------------------- | Estimated Per-Host Requirements: Memory=1.00GB VCores=1 | PLAN-ROOT SINK | | | 01:EXCHANGE [UNPARTITIONED] | | | 00:SCAN HBASE [functional_hbase.widetable_250_cols a] | predicates: a.int_col1 <= 20, a.int_col1 > 2 +------------------------------------------------------------------- Fetched 10 row(s) in 0.08s Right now I'm seeing some suspicious test output, specifically the SCANRANGELOCATIONS seem to be arbitrarily modified: section SCANRANGELOCATIONS of query: select * from functional_hbase.alltypesagg where bigint_col is not null and bool_col = true Actual does not match expected result: HBASE KEYRANGE port=16201 <unbounded>:1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ HBASE KEYRANGE port=16202 1:5 HBASE KEYRANGE port=16202 9:<unbounded> HBASE KEYRANGE port=16203 5:9 NODE 0: Expected: HBASE KEYRANGE port=16201 <unbounded>:3 HBASE KEYRANGE port=16202 3:7 HBASE KEYRANGE port=16203 7:<unbounded> NODE 0: Change-Id: I79750a8edb945effee2a519fa3b8192b77042cb4 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/ValueRange.java M fe/src/main/java/org/apache/impala/rewrite/ExprRewriter.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/conjunct-ordering.test A testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test 21 files changed, 640 insertions(+), 128 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/6389/12 -- To view, visit http://gerrit.cloudera.org:8080/6389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79750a8edb945effee2a519fa3b8192b77042cb4 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Zach Amsden <zams...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Marcel Kornacker <mar...@cloudera.com> Gerrit-Reviewer: Zach Amsden <zams...@cloudera.com>