Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15278 )
Change subject: IMPALA-7784: Use unescaped string in partition pruning + fix duplicatedly unescaping strings ...................................................................... Patch Set 3: (5 comments) Sorry to be late on this! Updated the patch based on the discussions. http://gerrit.cloudera.org:8080/#/c/15278/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15278/3//COMMIT_MSG@17 PS3, Line 17: Tests > HdfsPartitionTest seems to also include related tests. I feel it hard to add tests in HdfsPartitionTest. There are only two tests in it. One verifies the symmetric, transitive, reflexive and anti-symmetric of partition comparison. The other one verifies that FileDescriptor#cloneWithNewHostIndex() works with new host indices. The last one is unrelated. For the first one, if we add a StringLiteral with needsUnescaping_=true, the tests can still pass even without this fix. I tend to skip this if you are ok too. http://gerrit.cloudera.org:8080/#/c/15278/3//COMMIT_MSG@18 PS3, Line 18: - Add tests for partition pruning on unescaped strings. > This may be out of scope, but interop tests with Hive could be also useful, Good point! I created IMPALA-9459 for this. http://gerrit.cloudera.org:8080/#/c/15278/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java: http://gerrit.cloudera.org:8080/#/c/15278/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@42 PS3, Line 42: private final boolean needsUnescaping_; > optional: I think that the ideal solution would be to have an enum with the Good point. I leave a TODO for this since it takes some time to sort out all the sources. http://gerrit.cloudera.org:8080/#/c/15278/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@71 PS3, Line 71: // TODO: should we consider needsUnescaping_ in hashing? > I know that HashMap<Expr> is avoided for exactly this reason. The expressio Yeah, I add this TODO since I think this hash funciton is not used. Thanks for the explanation! Removed this TODO. http://gerrit.cloudera.org:8080/#/c/15278/3/testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test File testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test: http://gerrit.cloudera.org:8080/#/c/15278/3/testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test@25 PS3, Line 25: select * from functional.part_strings_with_quotes where p = "\"" > Can you add conjunct with an expression? e.g. AND p=concat("", '"'). Done -- To view, visit http://gerrit.cloudera.org:8080/15278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8070f16a74f9aeade294504f2834abb8b3b38f Gerrit-Change-Number: 15278 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Thu, 05 Mar 2020 02:49:28 +0000 Gerrit-HasComments: Yes
