[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog This fixed most of the remaining Kudu tests which relied on CTAS. Now only a few Kudu tests fail: FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added The above 4 fail because they are asserting something about the caching behavior of the old catalog implementation. FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table These fail due to attempting to load non-existent tables referred to by a DELETE statement. Need to investigate these further, but not related to CTAS. Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Reviewed-on: http://gerrit.cloudera.org:8080/10913 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java 9 files changed, 135 insertions(+), 28 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 6: Code-Review+2 Adding +2 since Vuk said LGTM pending Csaba's +1 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 19:27:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 05:22:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2860/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 25 Jul 2018 02:07:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/ -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 22:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 19:51:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 5: Code-Review+1 lgtm, but I do not feel the confidence to give +2 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 16:39:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/18/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 24 Jul 2018 00:37:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Hello Tianyi Wang, Csaba Ringhofer, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10913 to look at the new patch set (#4). Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog This fixed most of the remaining Kudu tests which relied on CTAS. Now only a few Kudu tests fail: FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added The above 4 fail because they are asserting something about the caching behavior of the old catalog implementation. FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table These fail due to attempting to load non-existent tables referred to by a DELETE statement. Need to investigate these further, but not related to CTAS. Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java 9 files changed, 135 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/4 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/18/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 23 Jul 2018 23:48:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? > ok, so just for future proofing. yep. improved the TODO to be more clear what I meant. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 23 Jul 2018 23:01:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? > nope, this would only show up temporarily in the analyzer during analysis o ok, so just for future proofing. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 01:10:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? > makes sense... I'd prefer the flag to guard. nope, this would only show up temporarily in the analyzer during analysis of a CREATE TABLE AS SELECT statement -- it never gets inserted into any Db and never sticks around outside the scope of a single CTAS -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 00:21:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? > yea, the target table is sort of an odd temporary catalog entry which doesn makes sense... I'd prefer the flag to guard. so this can end up in a "show tables" and from there, such a load could be requested? if that's the case, seems like a hidden table scheme for tmp could make sense. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 00:19:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? > what would be loaded... this is a new table if I've understood it correctly yea, the target table is sort of an odd temporary catalog entry which doesn't actually have an entry in the HMS. I was afraid that, if someone someone uses this method to create an instance, and then ends up calling getPartitions(), we'll end up making an HMS API call to get partitions for a table and throw an error like TableDoesNotExistException. Was thinking of either a new subclass or adding some kind of boolean isCtasTarget_ to assert against to make sure we don't do something like that. I'll elaborate. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 00:11:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 3: Code-Review+1 (2 comments) lgtm. I'll let Csaba have another look. http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@188 PS3, Line 188: the nit: remove second 'the' http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114 PS3, Line 114: load them? what would be loaded... this is a new table if I've understood it correctly? -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 18 Jul 2018 23:54:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Hello Tianyi Wang, Csaba Ringhofer, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10913 to look at the new patch set (#3). Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog This fixed most of the remaining Kudu tests which relied on CTAS. Now only a few Kudu tests fail: FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added The above 4 fail because they are asserting something about the caching behavior of the old catalog implementation. FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table These fail due to attempting to load non-existent tables referred to by a DELETE statement. Need to investigate these further, but not related to CTAS. Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java 10 files changed, 134 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/3 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java@176
PS2, Line 176: public FeKuduTable
createKuduCtasTarget(org.apache.hadoop.hive.metastore.api.Table msTbl,
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57
PS2, Line 57: import
org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result;
> sorry, Eclipse went nuts on autocomplete here apparently. Will remove in ne
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@211
PS2, Line 211: public TTableDescriptor toThriftDescriptor(int tableId,
Set referencedPartitions) {
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@83
PS2, Line 83: List partitionBy =
KuduTable.loadPartitionByParams(
> nit: this could be one line
Done
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 18 Jul 2018 00:59:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 ) Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57 PS2, Line 57: import org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result; > Where do we use this? I did not find it in the patch. sorry, Eclipse went nuts on autocomplete here apparently. Will remove in next rev. -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 00:25:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java@176
PS2, Line 176: public FeKuduTable
createKuduCtasTarget(org.apache.hadoop.hive.metastore.api.Table msTbl,
nit: long line
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57
PS2, Line 57: import
org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result;
Where do we use this? I did not find it in the patch.
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@211
PS2, Line 211: public TTableDescriptor toThriftDescriptor(int tableId,
Set referencedPartitions) {
nit: long line
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@83
PS2, Line 83: List partitionBy =
KuduTable.loadPartitionByParams(
nit: this could be one line
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 12 Jul 2018 16:28:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Hello Tianyi Wang, Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10913 to review the following change. Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog .. IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog This fixed most of the remaining Kudu tests which relied on CTAS. Now only a few Kudu tests fail: FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added The above 4 fail because they are asserting something about the caching behavior of the old catalog implementation. FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table These fail due to attempting to load non-existent tables referred to by a DELETE statement. Need to investigate these further, but not related to CTAS. Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java 10 files changed, 137 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/1 -- To view, visit http://gerrit.cloudera.org:8080/10913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46 Gerrit-Change-Number: 10913 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
