[Impala-ASF-CR] IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5506/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 17 Mar 2020 16:23:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15462 )
Change subject: IMPALA-9183: Convert certain disjunctive predicates to
conjunctive normal form
..
Patch Set 1:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/15462/1/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/15462/1/common/thrift/ImpalaInternalService.thrift@413
PS1, Line 413:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@486
PS1, Line 486: rules.add(new
ConvertToCNFRule(queryCtx.getClient_request().getQuery_options().getMax_cnf_exprs(),
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java
File fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java:
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@121
PS1, Line 121: Expr lhs1 = (CompoundPredicate)
CompoundPredicate.createDisjunctivePredicate(disjuncts);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@125
PS1, Line 125: Expr rhs1 = (CompoundPredicate)
CompoundPredicate.createDisjunctivePredicate(disjuncts);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@126
PS1, Line 126: Predicate newPredicate = (CompoundPredicate)
CompoundPredicate.createConjunction(lhs1, rhs1);
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@136
PS1, Line 136: Expr lhs1 = (CompoundPredicate)
CompoundPredicate.createDisjunctivePredicate(disjuncts);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@140
PS1, Line 140: Expr rhs1 = (CompoundPredicate)
CompoundPredicate.createDisjunctivePredicate(disjuncts);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@141
PS1, Line 141: Predicate newPredicate = (CompoundPredicate)
CompoundPredicate.createConjunction(lhs1, rhs1);
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java@165
PS1, Line 165: Predicate newPredicate = (CompoundPredicate)
CompoundPredicate.createConjunction(lhs1, rhs1);
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java:
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java@817
PS1, Line 817: RewritesOk("(int_col > 10 AND float_col < 5.0) OR (int_col <
20 AND float_col > 15.0)", rule,
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/15462/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java@819
PS1, Line 819: "AND int_col > 10 OR float_col > 15.0 AND
int_col > 10 OR int_col < 20");
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/15462
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072
Gerrit-Change-Number: 15462
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 17 Mar 2020 15:37:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form
Aman Sinha has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15462 Change subject: IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form .. IMPALA-9183: Convert certain disjunctive predicates to conjunctive normal form Added an expression rewrite rule to convert a disjunctive predicate to conjunctive normal form (CNF). Converting to CNF enables multi-table predicates that were only evaluated by a Join operator to be converted into either single-table conjuncts that are eligible for predicate pushdown to the scan operator or other multi-table conjuncts that are eligible to be pushed to a Join below. This helps improve performance for such queries. Since converting to CNF expands the number of expressions, we place a limit on the maximum number of CNF exprs (each AND is counted as 1 CNF expr) that are considered. Once the MAX_CNF_EXPRS limit (default is 100) is exceeded, whatever expression was supplied to the rule is returned without further transformation. A setting of -1 or 0 allows unlimited number of CNF exprs to be created upto int32 max. Another option ENABLE_CNF_REWRITES enables or disables the entire rewrite. This is False by default until we have done more thorough functional testing. Examples of rewrites: original: (a AND b) OR c rewritten: (a OR c) AND (b OR c) original: (a AND b) OR (c AND d) rewritten: (a OR c) AND (a OR d) AND (b OR c) AND (b OR d) original: NOT(a OR b) rewritten: NOT(a) AND NOT(b) Testing: - Added new unit tests with variations of disjunctive predicates and verified their Explain plans - Manually tested the result correctness on impala shell by running these queries with ENABLE_CNF_REWRITES enabled and disabled - Preliminary performance testing of TPC-DS q13 on a 10TB scale factor shows almost 5x improvement: Original baseline: 47.5 sec With this patch and CNF rewrite enabled: 9.4 sec Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java A fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test 10 files changed, 522 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15462/1 -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha
