[Impala-ASF-CR] IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17038 ) Change subject: IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8104/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fc758d3c1e75c7066936d590aec8bff8d2b00b0 Gerrit-Change-Number: 17038 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 09 Feb 2021 13:05:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17038 ) Change subject: IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException .. Patch Set 2: (7 comments) Thanks for the comments! http://gerrit.cloudera.org:8080/#/c/17038/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17038/1//COMMIT_MSG@7 PS1, Line 7: IMPAL > nit: IMPALA Oops, thanks. Done. http://gerrit.cloudera.org:8080/#/c/17038/1//COMMIT_MSG@25 PS1, Line 25: when doing star expansion > I think we also need to ignore them when resolving slot refs. These cases a Good catch. Added fix and tests in PS2. http://gerrit.cloudera.org:8080/#/c/17038/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/17038/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@130 PS1, Line 130: SelectStmt(SelectList selectList, : FromClause fromClause, : Expr wherePred > Maybe the current name is ok. We may refactor the AcidRewriter and reuse it Made this a property of TableRef, the name is 'isHidden()'. http://gerrit.cloudera.org:8080/#/c/17038/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@130 PS1, Line 130: SelectStmt(SelectList selectList, : FromClause fromClause, : Expr wherePred > I am not too familiar with the full ACID rewrites, but my guess is that the You are right, but since then Quanlong mentioned that we will get hidden table refs for other reasons as well. http://gerrit.cloudera.org:8080/#/c/17038/1/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@133 PS1, Line 133: ngPredicate, Lis > optional: I think that it would be slightly better to store this informatio Done http://gerrit.cloudera.org:8080/#/c/17038/1/testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test File testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test: http://gerrit.cloudera.org:8080/#/c/17038/1/testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test@317 PS1, Line 317: > +1 Sure. http://gerrit.cloudera.org:8080/#/c/17038/1/testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test@317 PS1, Line 317: > Can you add a bit more complex query? The idea is to include the same hidde This query actually revealed another bug, opened IMPALA-10493 for it. But if we put put the JOIN condition to the WHERE clause instead of the ON clause the query works fine. -- To view, visit http://gerrit.cloudera.org:8080/17038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fc758d3c1e75c7066936d590aec8bff8d2b00b0 Gerrit-Change-Number: 17038 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 09 Feb 2021 12:51:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17038 to look at the new patch set (#2). Change subject: IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException .. IMPALA-10482: Select-star query on unrelative collection column of transactional table hits IllegalStateException SELECT * query on unrelative collection column of transactional ORC table will hit IllegalStateException. The AcidRewriter will rewrite queries like "select item from my_complex_orc.int_array" to "select item from my_complex_orc t, t.int_array" This cause troubles in star expansion. Because the original query "select * from my_complex_orc.int_array" is analyzed as "select item from my_complex_orc.int_array" But the rewritten query "select * from my_complex_orc t, t.int_array" is analyzed as "select id, item from my_complex_orc t, t.int_array". The workaround is to keep track of the automatically added table refs during query rewrite. So when we analyze the rewritten query we can ignore these auxiliary table refs when doing star expansion. Testing: * added e2e tests Change-Id: I8fc758d3c1e75c7066936d590aec8bff8d2b00b0 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test 7 files changed, 191 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17038/2 -- To view, visit http://gerrit.cloudera.org:8080/17038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fc758d3c1e75c7066936d590aec8bff8d2b00b0 Gerrit-Change-Number: 17038 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
