[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16019 ) Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. Patch Set 4: (3 comments) Patch 4 addresses review issues. http://gerrit.cloudera.org:8080/#/c/16019/3/testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test File testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test: http://gerrit.cloudera.org:8080/#/c/16019/3/testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test@8 PS3, Line 8: > Can you remove the tpch. prefix? It shouldn't be necessary if this is calle Done http://gerrit.cloudera.org:8080/#/c/16019/3/testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test@10 PS3, Line 10: > It'd be good to include a results section here to verify that it produces t Done http://gerrit.cloudera.org:8080/#/c/16019/3/tests/query_test/test_aggregation.py File tests/query_test/test_aggregation.py: http://gerrit.cloudera.org:8080/#/c/16019/3/tests/query_test/test_aggregation.py@379 PS3, Line 379: > This fits better in TestTPCAggregationQueries below (l419) Done -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 02 Jun 2020 18:50:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Hello Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16019 to look at the new patch set (#4). Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad In streaming-aggregation-node.cc , when replicate_input_ is true and num_aggs > 1, it will call AddBatchStreaming several times(more than 1), each time, the out_batch will be used. If a row is not cached, the value will be saved in the out_batch, and out_batch's row count will be increased. The row_count did not set back to 0 when next while loop. Therefore in out_batch, it is possible that not all the tuples are non-null. (For example the rows added when agg_idx = 1, only tuple with 1 not null; the rows added when when agg_idx = 2, only tuple with 2 not null). But in grouping-aggregation-ir.cc, the serialize out code is start from very beginning of out_batch for a agg_idx, it has good chance to hit null tuple. Fix the issue by only serialize the tuples being added by current function call. Tests: Manual tests Unit tests Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 --- M be/src/exec/grouping-aggregator-ir.cc A testdata/workloads/tpch/queries/min-multiple-distinct-aggs.test M tests/query_test/test_aggregation.py 3 files changed, 24 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16019/4 -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16019 ) Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. Patch Set 3: The new patch(patch set 3) passed all core tests -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 02 Jun 2020 10:25:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Hello Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16019 to look at the new patch set (#3). Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad In streaming-aggregation-node.cc , when replicate_input_ is true and num_aggs > 1, it will call AddBatchStreaming several times(more than 1), each time, the out_batch will be used. If a row is not cached, the value will be saved in the out_batch, and out_batch's row count will be increased. The row_count did not set back to 0 when next while loop. Therefore in out_batch, it is possible that not all the tuples are non-null. (For example the rows added when agg_idx = 1, only tuple with 1 not null; the rows added when when agg_idx = 2, only tuple with 2 not null). But in grouping-aggregation-ir.cc, the serialize out code is start from very beginning of out_batch for a agg_idx, it has good chance to hit null tuple. Fix the issue by only serialize the tuples being added by current function call. Tests: Manual tests Unit tests Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 --- M be/src/exec/grouping-aggregator-ir.cc A testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test M tests/query_test/test_aggregation.py 3 files changed, 20 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16019/3 -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16019 ) Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. Patch Set 2: (4 comments) upload the second patch to address review issues. http://gerrit.cloudera.org:8080/#/c/16019/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16019/1//COMMIT_MSG@7 PS1, Line 7: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad > nit: make the first line more concise so it fits on a single line. Done http://gerrit.cloudera.org:8080/#/c/16019/1/be/src/exec/grouping-aggregator-ir.cc File be/src/exec/grouping-aggregator-ir.cc: http://gerrit.cloudera.org:8080/#/c/16019/1/be/src/exec/grouping-aggregator-ir.cc@160 PS1, Line 160: out_batch_sta > nit: out_batch_start Done http://gerrit.cloudera.org:8080/#/c/16019/1/testdata/data/local_parquet_tbl/part-0-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet File testdata/data/local_parquet_tbl/part-0-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet: PS1: > Checking in this amount of binary data is an issue for a number of reasons, Remove the parquet files. Figure out a query which make tpch.lineitem can reproduce the issue. It saves us a lot of space. http://gerrit.cloudera.org:8080/#/c/16019/1/tests/query_test/test_aggregation.py File tests/query_test/test_aggregation.py: http://gerrit.cloudera.org:8080/#/c/16019/1/tests/query_test/test_aggregation.py@380 PS1, Line 380: def test_min_multiple_distinct(self, vector, unique_database): > We should avoid this restriction - other tests achieve similar things witho removed. Use tpch.lineitem which need not load extra data. -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 02 Jun 2020 02:30:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad
Hello Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16019 to look at the new patch set (#2). Change subject: IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad .. IMPALA-9809: Multi-aggregation query on particular dataset crashes impalad In streaming-aggregation-node.cc , when replicate_input_ is true and num_aggs > 1, it will call AddBatchStreaming several times(more than 1), each time, the out_batch will be used. If a row is not cached, the value will be saved in the out_batch, and out_batch's row count will be increased. The row_count did not set back to 0 when next while loop. Therefore in out_batch, it is possible that not all the tuples are non-null. (For example the rows added when agg_idx = 1, only tuple with 1 not null; the rows added when when agg_idx = 2, only tuple with 2 not null). But in grouping-aggregation-ir.cc, the serialize out code is start from very beginning of out_batch for a agg_idx, it has good chance to hit null tuple. Fix the issue by only serialize the tuples being added by current function call. Tests: Manual tests Unit tests Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 --- M be/src/exec/grouping-aggregator-ir.cc A testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test M tests/query_test/test_aggregation.py 3 files changed, 21 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16019/2 -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9809: A query with multi-aggregation functions on particular dataset crashes impala daemon
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16019 Change subject: IMPALA-9809: A query with multi-aggregation functions on particular dataset crashes impala daemon .. IMPALA-9809: A query with multi-aggregation functions on particular dataset crashes impala daemon In streaming-aggregation-node.cc , when replicate_input_ is true and num_aggs > 1, it will call AddBatchStreaming several times(more than 1), each time, the out_batch will be used. If a row is not cached, the value will be saved in the out_batch, and out_batch's row count will be increased. The row_count did not set back to 0 when next while loop. Therefore in out_batch, it is possible that not all the tuples are non-null. (For example the rows added when agg_idx = 1, only tuple with 1 not null; the rows added when when agg_idx = 2, only tuple with 2 not null). But in grouping-aggregation-ir.cc, the serialize out code is start from very beginning of out_batch for a agg_idx, it has good chance to hit null tuple. Fix the issue by only serialize the tuples being added by current function call. Tests: Manual tests Unit tests Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 --- M be/src/exec/grouping-aggregator-ir.cc A testdata/data/local_parquet_tbl/_SUCCESS A testdata/data/local_parquet_tbl/part-0-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-1-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-2-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-3-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-4-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-5-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-6-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-7-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-8-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-9-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-00010-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/data/local_parquet_tbl/part-00011-fafc2cd0-f5c8-4fbb-ac3f-717447d67af8-c000.snappy.parquet A testdata/workloads/functional-query/queries/QueryTest/min-multiple-distinct-aggs.test M tests/query_test/test_aggregation.py 16 files changed, 30 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16019/1 -- To view, visit http://gerrit.cloudera.org:8080/16019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I06d73171cdc40bdbb15960573030ac7fc94a7e16 Gerrit-Change-Number: 16019 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 8: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 8 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 06 Nov 2019 05:18:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 3: Code-Review+1 (1 comment) Except for one small issue, the change looks good http://gerrit.cloudera.org:8080/#/c/14600/3/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/14600/3/tests/webserver/test_web_pages.py@378 PS3, Line 378: high_filecount_tbls = response_json["longest_loading_tables"] nit: Could you change the variable name? -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 01 Nov 2019 11:44:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/14600/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14600/2//COMMIT_MSG@14 PS2, Line 14: Testing: We usually add end to end test for it. http://gerrit.cloudera.org:8080/#/c/14600/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/14600/2/fe/src/main/java/org/apache/impala/catalog/Table.java@102 PS2, Line 102: // Time spend on table metadata loading. (ns) For the value is median loading time, you may need to add comments to state that. -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 31 Oct 2019 13:30:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: https://issues.apache.org/jira/browse/IMPALA-9049 https://issues.apache.org/jira/browse/IMPALA-9050 The related flaky tests jiras -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 19:07:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 3: The build error seems not related: beeswax/impala_beeswax.py:519: in __do_rpc 02:40:17 ] raise ImpalaBeeswaxException(self.__build_error_message(b), b) 02:40:17 ] E ImpalaBeeswaxException: ImpalaBeeswaxException: 02:40:17 ] EINNER EXCEPTION: 02:40:17 ] EMESSAGE: InconsistentMetadataFetchException: Fetching DATABASE failed. Could not find TCatalogObject(type:DATABASE, catalog_version:0, db:TDatabase(db_name:test_alter_database_set_owner_1e264a1)) 02:40:17 ] raise ImpalaBeeswaxException(self.__build_error_message(b), b) 02:40:17 ] E ImpalaBeeswaxException: ImpalaBeeswaxException: 02:40:17 ] EINNER EXCEPTION: 02:40:17 ] EMESSAGE: Unable to open Kudu table: Network error: failed to read from TLS socket (remote: 172.18.0.1:7051): Cannot send after transport endpoint shutdown (error 108) -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Mon, 14 Oct 2019 19:21:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 2: (1 comment) Patch set 2 add the comments. Thanks for the review. http://gerrit.cloudera.org:8080/#/c/14403/1/testdata/workloads/functional-query/queries/QueryTest/alias.test File testdata/workloads/functional-query/queries/QueryTest/alias.test: http://gerrit.cloudera.org:8080/#/c/14403/1/testdata/workloads/functional-query/queries/QueryTest/alias.test@89 PS1, Line 89: # literal alias > It would be good to include the JIRA here for future reference. Maybe also Done -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 11 Oct 2019 12:12:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Hello Anurag Mantripragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14403 to look at the new patch set (#2). Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly When the optimizer rewrites conjunct exprs with constant values, a new expr may cause AnalysisException. In this case, the conjuncts should use the original expr, not the intermediate expr produced by propagateConstants. Fixed optimizeConjuncts to handle this scenario properly. Tests: Add unit test for alias. Ran exhaustive tests. Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M testdata/workloads/functional-query/queries/QueryTest/alias.test 2 files changed, 19 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14403/2 -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14403 Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly When the optimizer rewrites conjunct exprs with constant values, a new expr may cause AnalysisException. In this case, the conjuncts should use the original expr, not the intermediate expr produced by propagateConstants. Fixed optimizeConjuncts to handle this scenario properly. Tests: Add unit test for alias. Ran exhaustive tests. Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M testdata/workloads/functional-query/queries/QueryTest/alias.test 2 files changed, 17 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14403/1 -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-9022: Fixed test query profile storage load time filesystem
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14387 ) Change subject: IMPALA-9022: Fixed test_query_profile_storage_load_time_filesystem .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14387/1/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14387/1/tests/query_test/test_observability.py@674 PS1, Line 674: def test_query_profile_storage_load_time_filesystem(self, unique_database, > I a better fix is to give this an @execute_serially annotation. I looked at The test is written with unique_database, it should not be bothered by other tests. -- To view, visit http://gerrit.cloudera.org:8080/14387 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee1afec6f2b706bc28b270aad731a138662490a Gerrit-Change-Number: 14387 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 08 Oct 2019 20:40:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9022: Fixed test query profile storage load time filesystem
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14387 Change subject: IMPALA-9022: Fixed test_query_profile_storage_load_time_filesystem .. IMPALA-9022: Fixed test_query_profile_storage_load_time_filesystem Skip part of the test which can be affected by the random behavor of Catalog V2. The major purpose of the test is to verify storage load time can be in query profile when metadata loading happens, which is not affected by the change. Change-Id: I6ee1afec6f2b706bc28b270aad731a138662490a --- M tests/query_test/test_observability.py 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/14387/1 -- To view, visit http://gerrit.cloudera.org:8080/14387 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6ee1afec6f2b706bc28b270aad731a138662490a Gerrit-Change-Number: 14387 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 6: (4 comments) Rebase and fix the tests. http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@676 PS3, Line 676: > So does 'Probes:.*\((\d+)\)', positive look-behinds are useful when you nee Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@677 PS3, Line 677: u > Done Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@688 PS3, Line 688: ry needs load > Done Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@691 PS3, Line 691: l > Done Done -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 24 Sep 2019 17:28:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Hello Bharath Vissapragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14234 to look at the new patch set (#6). Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ... Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (178 Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 8 files changed, 108 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/6 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 5: (1 comment) > Patch Set 3: > > (2 comments) http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@676 PS3, Line 676: '(?<=Probes: )\d+(\.\d+)?' > The point I am trying to make here is that your regexes are overly complica I feel my approach is more strict, it requires the number must directly follow "Probes: ". But your way is fine too, for our profile should not have ill-formatted data. I will fix my tests. -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 24 Sep 2019 14:48:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h File be/src/exec/hash-table.h: http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@651 PS3, Line 651: > It looks like the comment you added was "This method should only be called I do not know the code well, but from what I read, When spill or serializestreamforspill happens, there is not enough memory for hashtable, so the partition is spilled and hashtable is closed. Under this condition, this hashtable is not used by the query at all, and the stats should be all 0(related to operations) or not relevant. So we should not add them to the query profile. And some other Close happens under error conditions when creating hashtable which should be added either. -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 24 Sep 2019 14:43:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 5: (2 comments) patch 5 fixed style issues. http://gerrit.cloudera.org:8080/#/c/14234/4/be/src/exec/hash-table.h File be/src/exec/hash-table.h: http://gerrit.cloudera.org:8080/#/c/14234/4/be/src/exec/hash-table.h@650 PS4, Line 650: arent_profile > nit: parent_profile Done http://gerrit.cloudera.org:8080/#/c/14234/4/be/src/exec/hash-table.h@650 PS4, Line 650: RuntimeProfile* parent_profile); > nit: the formatting convention puts the open parenthesis on the previous li Done -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Mon, 23 Sep 2019 20:40:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Hello Bharath Vissapragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14234 to look at the new patch set (#5). Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ... Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (178 Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 8 files changed, 108 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/5 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 3: (6 comments) Patch 4 addresses review issues. http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h File be/src/exec/hash-table.h: http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@643 PS3, Line 643: // Create profile and counters for HashTable stats and add to parent profile. : /// Returns a HashTableStatsProfile object. > would be good to document that this actually makes a child profile called " Done http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@645 PS3, Line 645: CreateAndAddToProfile > I'd suggest returning a unique_ptr, and using an unique_ptr to store it - t Done http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@651 PS3, Line 651: StatsCountersAdd > Not sure I follow, once the HashTable is closed, the internal counters are But not all the close of hashtable should trigger the counters adding. I add more comments to explain the related case. http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@677 PS3, Line 677: a > Done Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@688 PS3, Line 688: execute_query > Done Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@691 PS3, Line 691: > Done Done -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 20 Sep 2019 13:24:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Hello Bharath Vissapragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14234 to look at the new patch set (#4). Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ... Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (178 Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 8 files changed, 108 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/4 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 3: > Patch Set 3: > > (2 comments) Give you an example for integer only case: Hash Table: - HashCollisions: 0 (0) - Probes: 2 (2) - Resizes: 0 (0) - Travel: 0 (0) -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 19 Sep 2019 20:01:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 3: (5 comments) > Patch Set 3: > > (2 comments) . I need the number only to change to float, so I need the return to be 2.52 in your case. . When the Probe is 0 or 18, there is no dot. http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h File be/src/exec/hash-table.h: http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@532 PS3, Line 532: HashTableStatsProfile > what exactly is this struct suppose to encapsulate? would be good to add so Done http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@534 PS3, Line 534: RuntimeProfile *hashtable_profile = nullptr; > nit: RuntimeProfile* hasbtable_profile = nullptr; Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@677 PS3, Line 677: a > nit: an Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@688 PS3, Line 688: execute_query > should assert that the query was actually successful: Done http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@691 PS3, Line 691: > nit: extra space Done -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 19 Sep 2019 19:51:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14234 ) Change subject: IMPALA-7637: Add more hash table stats to profile .. Patch Set 3: (2 comments) Answer some questions while I fix the review issues. http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h File be/src/exec/hash-table.h: http://gerrit.cloudera.org:8080/#/c/14234/3/be/src/exec/hash-table.h@651 PS3, Line 651: StatsCountersAdd > I think this kinda makes sense - we know the number of buckets earlier than It is accumulated counters, it can not be added in the close method. Some close called from spill, for example, we did not add this counter at close for collisions number in the legacy code. http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/14234/3/tests/query_test/test_observability.py@676 PS3, Line 676: '(?<=Probes: )\d+(\.\d+)?' > is the positive lookahead in this regex necessary? would the following rege I need to get the number only. For example 12 or 12.3. Your search seems will get back whole string "Probes: ...12.3" and will get nothing for integer case. -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 19 Sep 2019 13:06:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Hello Bharath Vissapragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14234 to look at the new patch set (#3). Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ... Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (178 Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 8 files changed, 97 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/3 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14234 )
Change subject: IMPALA-7637: Add more hash table stats to profile
..
Patch Set 2:
(3 comments)
New patch refactoring code to address the review issues, add support for group
aggregation profiling, add more check for tests and re-ran exhaustive tests
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/hash-table.h
File be/src/exec/hash-table.h:
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/hash-table.h@741
PS1, Line 741: /// Returns an iterator at the beginning of the hash table.
Advancing this iterator
> nit: we usually use lower_case for trivial accessor functions (this is allo
Agree, number of buckets follows the pattern. I removed these new methods for
they are no longer needed.
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc
File be/src/exec/partitioned-hash-join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc@595
PS1, Line 595: }
> Found the place in grouping-aggregator, I will put the similar code in a co
Done
http://gerrit.cloudera.org:8080/#/c/14234/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/14234/1/tests/query_test/test_observability.py@675
PS1, Line 675: assert "Resizes:" in runtime_profile
: nprobes = re.search('(?<=Probes: )\d+(\.\d+)?',
runtime_profile)
: # Probes and travel can be 0. The number can be a integer or
float with K
: assert float(nprobes.group(0)) >= 0
: ntravel = re.search('(?<=Travel: )\d
> That's a good point. I think it would be easy with the way the code is stru
Add assert on values, I just found , at least in aggregation, every counter
value can be 0.
Also, add the test for group by queries.
--
To view, visit http://gerrit.cloudera.org:8080/14234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87
Gerrit-Change-Number: 14234
Gerrit-PatchSet: 2
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Wed, 18 Sep 2019 13:05:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Hello Bharath Vissapragada, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14234 to look at the new patch set (#2). Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ... Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (178 Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 8 files changed, 91 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/2 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13786 to look at the new patch set (#6). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: * Ran queries that can trigger all of, none of or some of the related tables loading. * Check query profile for each query. * Check catalog metrics for each table. * Add unit tests to test_observability.py * Ran all core tests. Sample output: Profile for Catalog V1: (storage-load-time is the added property and it is part of Metadata load in Query Compilation): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Profile for Catalog V2: (StorageLoad.Time is the added property and it is in CatalogFetch): Frontend: - CatalogFetch.ColumnStats.Misses: 1 - CatalogFetch.ColumnStats.Requests: 1 - CatalogFetch.ColumnStats.Time: 0 - CatalogFetch.Config.Misses: 1 - CatalogFetch.Config.Requests: 1 - CatalogFetch.Config.Time: 3ms - CatalogFetch.DatabaseList.Hits: 1 - CatalogFetch.DatabaseList.Requests: 1 - CatalogFetch.DatabaseList.Time: 0 - CatalogFetch.PartitionLists.Misses: 1 - CatalogFetch.PartitionLists.Requests: 1 - CatalogFetch.PartitionLists.Time: 4ms - CatalogFetch.Partitions.Hits: 2 - CatalogFetch.Partitions.Misses: 1 - CatalogFetch.Partitions.Requests: 3 - CatalogFetch.Partitions.Time: 1ms - CatalogFetch.RPCs.Bytes: 1.01 KB (1036) - CatalogFetch.RPCs.Requests: 4 - CatalogFetch.RPCs.Time: 93ms - CatalogFetch.StorageLoad.Time: 68ms - CatalogFetch.TableNames.Hits: 2 - CatalogFetch.TableNames.Requests: 2 - CatalogFetch.TableNames.Time: 0 - CatalogFetch.Tables.Misses: 1 - CatalogFetch.Tables.Requests: 1 - CatalogFetch.Tables.Time: 91ms Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/query_test/test_observability.py 9 files changed, 170 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13786/6 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 6: patch 6 fix the test: because of the name change of the property, an old test was failed. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 17 Sep 2019 13:44:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14234 )
Change subject: IMPALA-7637: Add more hash table stats to profile
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc
File be/src/exec/partitioned-hash-join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc@595
PS1, Line 595: void PhjBuilder::AddHashTableStatsToProfile(RuntimeProfile
*profile) {
> The counters cannot be in the HashTable class. For each PhjBuilder object,
Found the place in grouping-aggregator, I will put the similar code in a common
method, it seems only HashTable possible.
--
To view, visit http://gerrit.cloudera.org:8080/14234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87
Gerrit-Change-Number: 14234
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Mon, 16 Sep 2019 21:48:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14234 )
Change subject: IMPALA-7637: Add more hash table stats to profile
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc
File be/src/exec/partitioned-hash-join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/14234/1/be/src/exec/partitioned-hash-join-builder.cc@595
PS1, Line 595: void PhjBuilder::AddHashTableStatsToProfile(RuntimeProfile
*profile) {
> The HashTable class is also used in the hash aggregation (grouping-aggregat
The counters cannot be in the HashTable class. For each PhjBuilder object, it
can have more than one hash table. For counters and profile object are all in
PhjBuilder, it is more natural the method is in the class. I will do more
research on where to profile grouping-aggregator.
--
To view, visit http://gerrit.cloudera.org:8080/14234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87
Gerrit-Change-Number: 14234
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Mon, 16 Sep 2019 20:03:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7637: Add more hash table stats to profile
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14234 Change subject: IMPALA-7637: Add more hash table stats to profile .. IMPALA-7637: Add more hash table stats to profile Add hash table counters(probes, travel and resizes) to profile. Put hash table stats into the child profile "hash table". Tests: Add new test test_query_profle_hashtable. Ran exhaustive tests. Profile Sample: Hash Join Builder (join_node_id=2): ExecOption: Codegen Disabled: disabled due to optimization hints Runtime filters: 1 of 1 Runtime Filter Published - BuildRowsPartitionTime: 157.960us - BuildRowsPartitioned: 100 (100) - HashTablesBuildTime: 298.817us - LargestPartitionPercent: 7 (7) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 17.12 KB (17536) - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) Hash Table: - HashBuckets: 256 (256) - HashCollisions: 0 (0) - Probes: 2.52K (2520) - Resizes: 0 (0) - Travel: 1.79K (1787) Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 --- M be/src/exec/hash-table.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M tests/query_test/test_observability.py 4 files changed, 48 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14234/1 -- To view, visit http://gerrit.cloudera.org:8080/14234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87 Gerrit-Change-Number: 14234 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 3: Re-format the commit message. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 11 Sep 2019 14:03:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13786 to look at the new patch set (#4). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: * Ran queries that can trigger all of, none of or some of the related tables loading. * Check query profile for each query. * Check catalog metrics for each table. * Add unit tests to test_observability.py * Ran all core tests. Sample output: Profile for Catalog V1: (storage-load-time is the added property and it is part of Metadata load in Query Compilation): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Profile for Catalog V2: (StorageLoad.Time is the added property and it is in CatalogFetch): Frontend: - CatalogFetch.ColumnStats.Misses: 1 - CatalogFetch.ColumnStats.Requests: 1 - CatalogFetch.ColumnStats.Time: 0 - CatalogFetch.Config.Misses: 1 - CatalogFetch.Config.Requests: 1 - CatalogFetch.Config.Time: 3ms - CatalogFetch.DatabaseList.Hits: 1 - CatalogFetch.DatabaseList.Requests: 1 - CatalogFetch.DatabaseList.Time: 0 - CatalogFetch.PartitionLists.Misses: 1 - CatalogFetch.PartitionLists.Requests: 1 - CatalogFetch.PartitionLists.Time: 4ms - CatalogFetch.Partitions.Hits: 2 - CatalogFetch.Partitions.Misses: 1 - CatalogFetch.Partitions.Requests: 3 - CatalogFetch.Partitions.Time: 1ms - CatalogFetch.RPCs.Bytes: 1.01 KB (1036) - CatalogFetch.RPCs.Requests: 4 - CatalogFetch.RPCs.Time: 93ms - CatalogFetch.StorageLoad.Time: 68ms - CatalogFetch.TableNames.Hits: 2 - CatalogFetch.TableNames.Requests: 2 - CatalogFetch.TableNames.Time: 0 - CatalogFetch.Tables.Misses: 1 - CatalogFetch.Tables.Requests: 1 - CatalogFetch.Tables.Time: 91ms Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/query_test/test_observability.py 9 files changed, 169 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13786/4 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13786 to look at the new patch set (#3). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: * Ran queries that can trigger all of, none of or some of the related tables loading. * Check query profile for each query. * Check catalog metrics for each table. * Add unit tests to test_observability.py * Ran all core tests. Sample output: Profile for Catalog V1: (storage-load-time is the added property and it is part of Metadata load in Query Compilation): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Profile for Catalog V2: (StorageLoad.Time is the added property and it is in CatalogFetch): Frontend: - CatalogFetch.ColumnStats.Misses: 1 - CatalogFetch.ColumnStats.Requests: 1 - CatalogFetch.ColumnStats.Time: 0 - CatalogFetch.Config.Misses: 1 - CatalogFetch.Config.Requests: 1 - CatalogFetch.Config.Time: 3ms - CatalogFetch.DatabaseList.Hits: 1 - CatalogFetch.DatabaseList.Requests: 1 - CatalogFetch.DatabaseList.Time: 0 - CatalogFetch.PartitionLists.Misses: 1 - CatalogFetch.PartitionLists.Requests: 1 - CatalogFetch.PartitionLists.Time: 4ms - CatalogFetch.Partitions.Hits: 2 - CatalogFetch.Partitions.Misses: 1 - CatalogFetch.Partitions.Requests: 3 - CatalogFetch.Partitions.Time: 1ms - CatalogFetch.RPCs.Bytes: 1.01 KB (1036) - CatalogFetch.RPCs.Requests: 4 - CatalogFetch.RPCs.Time: 93ms - CatalogFetch.StorageLoad.Time: 68ms - CatalogFetch.TableNames.Hits: 2 - CatalogFetch.TableNames.Requests: 2 - CatalogFetch.TableNames.Time: 0 - CatalogFetch.Tables.Misses: 1 - CatalogFetch.Tables.Requests: 1 - CatalogFetch.Tables.Time: 91ms Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/query_test/test_observability.py 9 files changed, 169 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13786/3 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 2: (8 comments) Submit patch 3 to address the code review http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG@18 PS2, Line 18: Testing: Ran queries that can trigger all of, none of or : some of the related tables loading. : Check query profile for each query. : Check catalog metrics for each table. : Add unit tests to test_observability.py : Ran all core tests. > nit: if these are a list of testing activities done, i generally like to fo Done http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG@27 PS2, Line 27: After ran a hbase query (Metadata load finished is divided into several lines > nit: line too long Done http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG@28 PS2, Line 28: because of limitation of commit message): Query Compilation: 4s401ms > nit: the profile snippet should be on a newline Done http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG@35 PS2, Line 35: Profile for Catalog V2: > would be good to explain how storage-load-time is represented in the v1 pro Done http://gerrit.cloudera.org:8080/#/c/13786/2//COMMIT_MSG@46 PS2, Line 46: storage-load-time > nit: why is this formatted differently from everything else - e.g. would I will change to CatalogFetch.StorageLoad.Time which consistent with V1 and has three levels. http://gerrit.cloudera.org:8080/#/c/13786/2/common/thrift/CatalogObjects.thrift File common/thrift/CatalogObjects.thrift: http://gerrit.cloudera.org:8080/#/c/13786/2/common/thrift/CatalogObjects.thrift@496 PS2, Line 496: 15: optional i64 storage_metadata_load_time > nit: change to storage_metadata_load_time_ns so we know it is in nanosecond Done http://gerrit.cloudera.org:8080/#/c/13786/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13786/2/tests/query_test/test_observability.py@23 PS2, Line 23: SkipIfCatalogV2 > is this needed? Done http://gerrit.cloudera.org:8080/#/c/13786/2/tests/query_test/test_observability.py@684 PS2, Line 684: def test_query_profile_storage_load_time(self): > nit: might be cleaner to split this into two tests, one for kudu and one fo I think they can put together: They are similar and do not belong to filesystem loading. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Mon, 09 Sep 2019 16:15:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 2: submit patch set 2 to include profile support for catalog V2 Ran exhaustive test: Only one unrelated failure: query_test/test_result_spooling.py (test_full_queue_large_fetch) -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 06 Sep 2019 12:47:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13786 to look at the new patch set (#2). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: Profile for Catalog V1: (storage-load-time is the added property): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Profile for Catalog V2: Frontend: - CatalogFetch.ColumnStats.Misses: 1 - CatalogFetch.ColumnStats.Requests: 1 - CatalogFetch.ColumnStats.Time: 16ms - CatalogFetch.Config.Misses: 1 - CatalogFetch.Config.Requests: 1 - CatalogFetch.Config.Time: 24ms - CatalogFetch.DatabaseList.Hits: 1 - CatalogFetch.DatabaseList.Requests: 1 - CatalogFetch.DatabaseList.Time: 0 - CatalogFetch.Metadata.storage-load-time: 218ms - CatalogFetch.PartitionLists.Misses: 1 - CatalogFetch.PartitionLists.Requests: 1 Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/query_test/test_observability.py 9 files changed, 160 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13786/2 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8889: Fix error messages for unsupported operations on acid tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14133 ) Change subject: IMPALA-8889: Fix error messages for unsupported operations on acid tables .. Patch Set 3: (1 comment) patch 3 add the space. http://gerrit.cloudera.org:8080/#/c/14133/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/14133/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@242 PS2, Line 242: operationStr, table.getFullName())); > nit: missing space Done -- To view, visit http://gerrit.cloudera.org:8080/14133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe Gerrit-Change-Number: 14133 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Aug 2019 16:24:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8889: Fix error messages for unsupported operations on acid tables
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14133 to look at the new patch set (#3). Change subject: IMPALA-8889: Fix error messages for unsupported operations on acid tables .. IMPALA-8889: Fix error messages for unsupported operations on acid tables Provides up-to-date error messages for acid tables. Makes minor code change for ensureTableWriteSupported after HIVEMANAGEDINSERTWRITE is enabled. Tests: Fixed and tested AnalyerTest Fixed acid-negative test Ran all core tests for Hive 3 Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test 7 files changed, 19 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14133/3 -- To view, visit http://gerrit.cloudera.org:8080/14133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe Gerrit-Change-Number: 14133 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8889: Fix error messages for unsupported operations on acid tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14133 ) Change subject: IMPALA-8889: Fix error messages for unsupported operations on acid tables .. Patch Set 2: (2 comments) Submit patch set 2 to fix the commit message. http://gerrit.cloudera.org:8080/#/c/14133/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14133/1//COMMIT_MSG@9 PS1, Line 9: up-to-da > typo? Done http://gerrit.cloudera.org:8080/#/c/14133/1//COMMIT_MSG@10 PS1, Line 10: Makes minor code change for ensureTableWriteSupported after > I do not understand what this means. Can you add more detail? This is talking about https://gerrit.cloudera.org/c/14133/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java#294 I will add more information. -- To view, visit http://gerrit.cloudera.org:8080/14133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe Gerrit-Change-Number: 14133 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Aug 2019 14:50:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8889: Fix error messages for unsupported operations on acid tables
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14133 to look at the new patch set (#2). Change subject: IMPALA-8889: Fix error messages for unsupported operations on acid tables .. IMPALA-8889: Fix error messages for unsupported operations on acid tables Provides up-to-date error messages for acid tables. Makes minor code change for ensureTableWriteSupported after HIVEMANAGEDINSERTWRITE is enabled. Tests: Fixed and tested AnalyerTest Fixed acid-negative test Ran all core tests for Hive 3 Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test 7 files changed, 19 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14133/2 -- To view, visit http://gerrit.cloudera.org:8080/14133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe Gerrit-Change-Number: 14133 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8889: Fix error messages for unsupported operations on acid tables
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14133 Change subject: IMPALA-8889: Fix error messages for unsupported operations on acid tables .. IMPALA-8889: Fix error messages for unsupported operations on acid tables Provides uptodate error messages for acid tables. Makes minor code change after HIVEMANAGEDINSERTWRITE is enabled. Tests: Fixed and tested AnalyerTest Fixed acid-negative test Ran all core tests for Hive 3 Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test 7 files changed, 19 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14133/1 -- To view, visit http://gerrit.cloudera.org:8080/14133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I732bf651405c9ed75d1843390050b786720e3ffe Gerrit-Change-Number: 14133 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7027: fix StringLiteral.localEquals()
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14096 ) Change subject: IMPALA-7027: fix StringLiteral.localEquals() .. Patch Set 1: Ran all exhaustive tests: https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5623/ -- To view, visit http://gerrit.cloudera.org:8080/14096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 Gerrit-Change-Number: 14096 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 21 Aug 2019 13:56:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7027: fix StringLiteral.localEquals()
Yongzhi Chen has uploaded a new patch set (#2) to the change originally created by Tim Armstrong. ( http://gerrit.cloudera.org:8080/14096 ) Change subject: IMPALA-7027: fix StringLiteral.localEquals() .. IMPALA-7027: fix StringLiteral.localEquals() String literals with the same value but different types are not interchangeable. localEquals() should check the type, similar to a cast expression. Testing: - Added String literals tests. - Passed exhaustive tests. Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 --- M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java A testdata/workloads/functional-query/queries/QueryTest/string-literals.test M tests/query_test/test_chars.py 3 files changed, 46 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/14096/2 -- To view, visit http://gerrit.cloudera.org:8080/14096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 Gerrit-Change-Number: 14096 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] DRAFT: IMPALA-8836: Support COMPUTE STATS on insert only ACID tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14066 ) Change subject: DRAFT: IMPALA-8836: Support COMPUTE STATS on insert only ACID tables .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14066 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c06b4678c1ff75c5aa1586a78afea563e64057f Gerrit-Change-Number: 14066 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 15 Aug 2019 19:22:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8793: Implement TRUNCATE for insert-only ACID tables
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14071 )
Change subject: IMPALA-8793: Implement TRUNCATE for insert-only ACID tables
..
Patch Set 1:
(3 comments)
A couple of my comments.
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
File fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java:
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java@71
PS1, Line 71: analyzer.ensureTableNotFullAcid(table_);
I think this line can be removed, the check on line 70 already blocks the
FullAcid table.
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/catalog/Catalog.java@704
PS1, Line 704: if (txnId == 0L) transactionKeepalive_.addLock(lockId,
ctx);
Is there a corresponding remove lock part?
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14071/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1854
PS1, Line 1854: for (FeFsPartition part: partitions) {
Could you comment a little more on why a transactional table does not remove
files? Can that cause file leaks?
--
To view, visit http://gerrit.cloudera.org:8080/14071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic749b7f27da157e1c0ebf9b7e9b6ee09afad122a
Gerrit-Change-Number: 14071
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Thu, 15 Aug 2019 18:49:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] DRAFT: IMPALA-8836: Support COMPUTE STATS on insert only ACID tables
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14066 )
Change subject: DRAFT: IMPALA-8836: Support COMPUTE STATS on insert only ACID
tables
..
Patch Set 2:
(3 comments)
It generally looks good, just small issues.
http://gerrit.cloudera.org:8080/#/c/14066/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14066/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1017
PS2, Line 1017: msTbl.putToParameters("COLUMN_STATS_ACCURATE", "false");
Just remove the property from parameters. "false" is not a valid value. The
value is in json string style.
http://gerrit.cloudera.org:8080/#/c/14066/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3809
PS2, Line 3809: tblTxn = MetastoreShim.createTblTransaction(
In order not to leak resources, Is that possible transaction ID is not valid,
you create a new transaction? Should you handle commit the transaction if it
owned in case? Or precondition here to make sure the id always valid?
http://gerrit.cloudera.org:8080/#/c/14066/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/14066/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1673
PS2, Line 1673: long transactionId =
MetastoreShim.openTransaction(hmsClient);
Why remove the name here?
--
To view, visit http://gerrit.cloudera.org:8080/14066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c06b4678c1ff75c5aa1586a78afea563e64057f
Gerrit-Change-Number: 14066
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Thu, 15 Aug 2019 17:06:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14037 ) Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. Patch Set 6: (1 comment) Because partition's column_stats_accurate property is never shown in "show create table" or show partitions, I added Hive query select count(*) in the tests to show the patch works by showing hive does not return wrong values for count after impala insert. http://gerrit.cloudera.org:8080/#/c/14037/6/testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test File testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test: http://gerrit.cloudera.org:8080/#/c/14037/6/testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test@45 PS6, Line 45: > Again, no checks about the stats for the partitioned table. the same reason -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Aug 2019 14:19:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14037 ) Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. Patch Set 6: Because partition's column_stats_accurate property is never shown in "show create table" or show partitions, I added Hive query select count(*) in the tests to show the patch works by showing hive does not return wrong values for count after impala insert. -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Aug 2019 14:17:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/14037 )
Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties
..
Patch Set 5:
(2 comments)
submit patch set 6
http://gerrit.cloudera.org:8080/#/c/14037/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14037/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3762
PS3, Line 3762: table.getDb().getName(), table.getName());
: }
: }
> You could get the write id of INSERT the same way as we get the transaction
I got the conclusion of the same writeid by testing as the following:
This list the write ID in partitions table after 1. Hive did compute stats for
the whole table (analyze table insertonly_part_colstats compute statistics for
columns;) . 2. Impala insert a row with column stats accurate remove.
3. Hive compute stats for the whole table again. You can see the write ID in
partitions increased only 1 each time which means no waste of write id number.
And it consists with hive's insert statement. See the last select on
partitions(it is after hive insert a row to the 2010-01-01 partiton:
HMS_home_yongzhi_Impala_cdp=> select * from "PARTITIONS" where "TBL_ID"=3274;
PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
WRITE_ID
-+-+--+---+---++--
15991 | 1565289739 |0 | ds=2010-01-01 | 19250 | 3274 |
18
15992 | 1565289739 |0 | ds=2010-01-02 | 19251 | 3274 |
18
(2 rows)
HMS_home_yongzhi_Impala_cdp=> select * from "PARTITIONS" where "TBL_ID"=3274;
PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
WRITE_ID
-+-+--+---+---++--
15991 | 1565289739 |0 | ds=2010-01-01 | 19250 | 3274 |
19
15992 | 1565289739 |0 | ds=2010-01-02 | 19251 | 3274 |
18
(2 rows)
HMS_home_yongzhi_Impala_cdp=> select * from "PARTITION_PARAMS" where
"PART_ID"=15991;
PART_ID | PARAM_KEY | PARAM_VALUE
-+---+-
15991 | transient_lastDdlTime | 1565633500
15991 | numFiles | 4
15991 | totalSize | 8
15991 | numRows | 4
15991 | rawDataSize | 4
(5 rows)
HMS_home_yongzhi_Impala_cdp=> select * from "TBLS" where
"TBL_NAME"='insertonly_part_colstats';
HMS_home_yongzhi_Impala_cdp=> select * from "PARTITIONS" where "TBL_ID"=3274;
PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
WRITE_ID
-+-+--+---+---++--
15991 | 1565289739 |0 | ds=2010-01-01 | 19250 | 3274 |
20
15992 | 1565289739 |0 | ds=2010-01-02 | 19251 | 3274 |
20
(2 rows)
HMS_home_yongzhi_Impala_cdp=> select * from "PARTITIONS" where "TBL_ID"=3274;
PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
WRITE_ID
-+-+--+---+---++--
15991 | 1565289739 |0 | ds=2010-01-01 | 19250 | 3274 |
21
15992 | 1565289739 |0 | ds=2010-01-02 | 19251 | 3274 |
20
(2 rows)
http://gerrit.cloudera.org:8080/#/c/14037/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14037/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3718
PS5, Line 3718: if (update.isSetTransaction_id()) {
: transactionId = update.getTransaction_id();
: }
> nit: fits single line
Done
--
To view, visit http://gerrit.cloudera.org:8080/14037
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb
Gerrit-Change-Number: 14037
Gerrit-PatchSet: 5
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 12 Aug 2019 18:43:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#6). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties Hive depends on property COLUMN_STATS_ACCURATE to tell if the stored statistics accurate. After Impala inserts data, it does not set statistics values up-to-date(for example numRows). Impala should unset COLUMN_STATS_ACCURATE to tell Hive the stored stats are no longer accurate. The patch impletes: After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implements the stats changes above for both acid/non-acid tables. Tests: Manual tests. Run core tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 340 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/6 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14037 ) Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. Patch Set 5: (2 comments) Submit patch 5 http://gerrit.cloudera.org:8080/#/c/14037/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/14037/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@110 PS3, Line 110: : /** > This signature should be changed in the Hive2 MetastoreShim Done http://gerrit.cloudera.org:8080/#/c/14037/3/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/14037/3/tests/query_test/test_acid.py@104 PS3, Line 104: pIfABFS.hive > This sounds weird, it would be good to investigate the cause, but I am ok w I will do more research later, I have a feeling that hive client and impala clients not using the same thread (at least sometimes), they cannot see each other's changes sometimes because one calls before another one really finishes. -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Sat, 10 Aug 2019 20:08:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#5). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties Hive depends on property COLUMN_STATS_ACCURATE to tell if the stored statistics accurate. After Impala inserts data, it does not set statistics values up-to-date(for example numRows). Impala should unset COLUMN_STATS_ACCURATE to tell Hive the stored stats are no longer accurate. The patch impletes: After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implements the stats changes above for both acid/non-acid tables. Tests: Manual tests. Run core tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 342 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/5 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#4). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties Hive depends on property COLUMN_STATS_ACCURATE to tell if the stored statistics accurate. After Impala inserts data, it does not set statistics values up-to-date(for example numRows). Impala should unset COLUMN_STATS_ACCURATE to tell Hive the stored stats are no longer accurate. The patch impletes: After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implements the stats changes above for both acid/non-acid tables. Tests: Manual tests. Run core tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 340 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/4 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#3). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implemented the stats change for both acid/non-acid tables. Tests: Manual tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 338 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/3 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#2). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid table. Implemented for both acid/non-acid tables. Tests: Manual tests. Add ee test to test interop with Hive for acid table. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 332 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/2 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14037 Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid table. Implemented for both acid/non-acid tables. Tests: Manual tests. Add ee test to test interop with Hive for acid table. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 326 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/1 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: From yesterdays FE meeting, Vihang prefer following fix: --- a/tests/common/impala_test_suite.py +++ b/tests/common/impala_test_suite.py @@ -81,6 +81,7 @@ from tests.util.thrift_util import create_transport # Imports required for Hive Metastore Client from hive_metastore import ThriftHiveMetastore +from hive_metastore import ttypes from thrift.protocol import TBinaryProtocol # Initializing the logger before conditional imports, since we will need it @@ -122,7 +123,6 @@ SET_PATTERN = re.compile( METRICS_URL = 'http://localhost:25000/metrics?json' GROUP_NAME = grp.getgrgid(pwd.getpwnam(getuser()).pw_gid).gr_name - # Base class for Impala tests. All impala test cases should inherit from this class class ImpalaTestSuite(BaseTestSuite): @classmethod @@ -806,7 +806,14 @@ class ImpalaTestSuite(BaseTestSuite): table = self.hive_client.get_table(db_name, table_name) assert table is not None self.hive_client.drop_table(db_name, table_name, True) -self.hive_client.create_table(table) +if HIVE_MAJOR_VERSION > 2: + capabilities = ["EXTREAD", "EXTWRITE","HIVEBUCKET2", "HIVEMANAGEDINSERTREAD", + "HIVEMANAGEDINSERTWRITE", "HIVEMQT", "HIVEFULLACIDREAD", "HIVEFULLACIDWRITE"] + createTblRequest = ttypes.CreateTableRequest(table, + processorCapabilities=capabilities, processorIdentifier="ImpalatestPython") + self.hive_client.create_table_req(createTblRequest) +else: + self.hive_client.create_table(table) -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 16:29:58 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-8637: Implement transaction handling and locking for ACID queries
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: WIP IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13968/2/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/13968/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1683 PS2, Line 1683: Do we have commitTransaction calls somewhere that actively call the transactionKeepalive_.deleteTransaction(txnId); method(instead of using keepalive to detect and remove)? http://gerrit.cloudera.org:8080/#/c/13968/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1726 PS2, Line 1726: lockComponent.setType(LockType.EXCLUSIVE); Does hive have code to handle deadlock prevention? -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 01 Aug 2019 22:42:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: I created a jira for the local catalog issue: https://issues.apache.org/jira/browse/IMPALA-8822 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 01 Aug 2019 16:40:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8751: Skip failing Kudu - HMS integration tests with Hive 3
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13854 ) Change subject: IMPALA-8751: Skip failing Kudu - HMS integration tests with Hive 3 .. Patch Set 1: Code-Review+1 To skip these kudu tests is fine to me. -- To view, visit http://gerrit.cloudera.org:8080/13854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic19b8d93eaa6e8ec886c5704578563fb0871f941 Gerrit-Change-Number: 13854 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Jul 2019 18:14:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13765 ) Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. Patch Set 5: submit patch set 5 with better tests -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 16 Jul 2019 06:31:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Hello Bharath Vissapragada, Quanlong Huang, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13765 to look at the new patch set (#5). Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, the planner makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation(IsNullPredicate is not supported for complex types), AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Ran all core tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test M tests/query_test/test_nested_types.py 3 files changed, 63 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/5 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. Patch Set 12: (1 comment) > Patch Set 11: Code-Review+2 > > (1 comment) > > Lgtm! Thanks for applying the changes! Zoltan and Csaba, Thanks for the review! http://gerrit.cloudera.org:8080/#/c/13558/11/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java: http://gerrit.cloudera.org:8080/#/c/13558/11/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@46 PS11, Line 46: // Key for config option read from hive-site.xml > nit: leftover blank line from previous PS Done -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 12 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 15 Jul 2019 18:24:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#12). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. When upgrade tables from non-acid to acid, the default capabilities should be changed too. Currently, use the workaround by explicitly setting tblproperties OBJCAPABILITIES with the acid properties. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 21 files changed, 418 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/12 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 12 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. Patch Set 11: (2 comments) Submit patch 11 with: 1. Use Impala build number by calling BackendConfig. 2. Add ToDo for alter table (nonacid to acid) table properties changes. 3. Rebase and re-run Hive 2 and Hive 3 core Hive 2 all passed: https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5404/testReport/ Hive 3 7 failures, no regressions: https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5402/testReport/ http://gerrit.cloudera.org:8080/#/c/13558/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13558/10//COMMIT_MSG@26 PS10, Line 26: When upgrade tables from non-acid to acid, the default > Can you add a TODO about ugpraded tables? My impression is that creating a Done http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java: http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@47 PS7, Line 47: // Key for config option read from hive-site.xml : private static final String HIVE_METASTORE_CNXN_DELAY_MS_CONF = : "impala.catalog.metastore.cnxn.creation.delay.ms"; : private static final int DEFAULT_HIVE_METASTORE_CNXN_DELAY_MS_CONF = 0; : / > Here to make sure the client capabilities are set before the pool start to new patch moves the code out of the static block. -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 13 Jul 2019 10:55:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#11). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. When upgrade tables from non-acid to acid, the default capabilities should be changed too. Currently, use the workaround by explicitly setting tblproperties OBJCAPABILITIES with the acid properties. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 22 files changed, 419 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/11 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13558 )
Change subject: IMPALA-8593: Support table capabilities handling with Hive 3
..
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@92
PS7, Line 92: eUtils.valida
> Yeah, sorry about that, I thought MAJOR_VERSION is only used to generate th
I will change back.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@532
PS7, Line 532:
> sorry, hive.
I will change to hive.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@270
PS7, Line 270: break;
: case READ:
: case ANY:
: default:
> Yeah, but here we want to check if the table has READ access type, right?
Here is simplfied. Assume read, read/write and write all have read permission.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@47
PS7, Line 47: static {
: if (MetastoreShim.getMajorVersion() > 2) {
: MetastoreShim.setHiveClientCapabilities();
: }
: }
> But if you only add it to MetastoreShim 3, then it should be fine, right?
Here to make sure the client capabilities are set before the pool start to
create hms client.
I tried place BackendConfig inside MetastoreClient creation around line 99, but
it is still with null value.
--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 10
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 12 Jul 2019 18:23:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. Patch Set 10: (1 comment) Add one missing comment. http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@198 PS7, Line 198: Operation typ > nit: please add comment Done -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 12 Jul 2019 16:04:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#10). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 405 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/10 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13558 )
Change subject: IMPALA-8593: Support table capabilities handling with Hive 3
..
Patch Set 9:
(10 comments)
Patch set 9 addresses review issues and adds more tests.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@78
PS7, Line 78: private static final String EXTWRITE = "EXTWRITE";
: private static final String EXTREAD = "EXTREAD";
> nit: unused
Done
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@83
PS7, Line 83: private static final String HIVEMANAGEDINSERTREAD =
"HIVEMANAGEDINSERTREAD";
: private static final String HIVEMANAGEDINSERTWRITE =
"HIVEMANAGEDINS
> nit: unused
these are some possible capabilities we may support.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@87
PS7, Line 87: private static final String HIVEMQT = "HIVEMQT";
> nit: unused
We may need it in the future.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@92
PS7, Line 92: eUtils.valida
> nit: Technically this class doesn't have a version number, so maybe we shou
Done
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@470
PS7, Line 470: HIVESQL,
> nit: maybe add TODO for HIVEMANAGEDINSERTWRITE once IMPALA-8636 goes in.
Done
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@532
PS7, Line 532:
> nit: impala
Done
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@130
PS7, Line 130: public static final byte ACCESSTYPE_READ = (byte)2;
: public static final byte ACCESSTYPE_WRITE = (byte)4;
: public static final byte ACCESSTYPE_READWRITE = (byte
> nit: maybe we could use the same naming as in Hive, i.e. ACCESSTYPE_READONL
Done
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@270
PS7, Line 270: case READ:
: case ANY:
: default:
: ensureTableSupported(table);
> nit: ensureTableSupported() also succeeds if the table has ACCESSTYPE_WRITE
write only belong to the operation WRITE. In the code ensureTableSupported,
require_write(write only) is already included.
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java:
http://gerrit.cloudera.org:8080/#/c/13558/7/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@47
PS7, Line 47: static {
: if (MetastoreShim.getMajorVersion() > 2) {
: MetastoreShim.setHiveClientCapabilities();
: }
: }
> nit: Can you place this to MetastoreShim 3 only?
The setHiveClientCapabilities is only implemented in MetasoreShim 3, in Shim2
the method just throws Unsupported exception. add the if statement to avoid
thrown unsupported Exception in Hive 2
http://gerrit.cloudera.org:8080/#/c/13558/7/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/13558/7/tests/metadata/test_ddl.py@669
PS7, Line 669: if HIVE_MAJOR_VERSION > 2:
: assert properties['OBJCAPABILITIE
> We could also check OBJCAPABILITIES here, as it should have a fix values.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 9
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 12 Jul 2019 14:49:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#9). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 404 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/9 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#8). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 404 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/8 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 8 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8681: Fix null pointer exception in ValidWriteIdLists generation
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13845 ) Change subject: IMPALA-8681: Fix null pointer exception in ValidWriteIdLists generation .. Patch Set 1: Code-Review+1 The change looks good. -- To view, visit http://gerrit.cloudera.org:8080/13845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9d1dcf9b7496c4a80e9af82cadad8682085d849 Gerrit-Change-Number: 13845 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 11 Jul 2019 16:42:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. Patch Set 7: (3 comments) Patch set 7 addresses review issues. http://gerrit.cloudera.org:8080/#/c/13558/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13558/6//COMMIT_MSG@7 PS6, Line 7: Support table capabilities handling with Hive > I think that bucketed tables are now a secondary feature compared to capabi Done http://gerrit.cloudera.org:8080/#/c/13558/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13558/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2210 PS6, Line 2210: he table cannot be found after created. > This means that HMS doesn't return Kudu tables where OBJCAPABILITIES is set Done http://gerrit.cloudera.org:8080/#/c/13558/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2211 PS6, Line 2211: ed jira > typo: adding Done -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 7 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 11 Jul 2019 14:50:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Support table capabilities handling with Hive 3
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#7). Change subject: IMPALA-8593: Support table capabilities handling with Hive 3 .. IMPALA-8593: Support table capabilities handling with Hive 3 This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 393 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/7 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 7 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. Patch Set 6: (2 comments) Patch set 6 addressed the review issues and was rebased to the latest hive bump. Run cdp all core tests, the failures are the same as before the change to upstream master. Both have 12 failures: My private cdp build: https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5393/testReport/ upstream without the patch: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch-cdp-hive/23/testReport/ So no regressions. http://gerrit.cloudera.org:8080/#/c/13558/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13558/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2208 PS5, Line 2208: } e > What should happen if it is a full acid tables? If it shouldn't be possible Done http://gerrit.cloudera.org:8080/#/c/13558/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2210 PS5, Line 2210: 1. The property is not stor > A comment could be added about the lack of capabilities for Kudu tables. Done -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 11 Jul 2019 04:00:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#6). Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. IMPALA-8593: Prohibit write operations for bucketed tables This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 392 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/6 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13786/1/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13786/1/tests/query_test/test_observability.py@589 PS1, Line 589: @SkipIfCatalogV2.hms_event_polling_enabled() > we should really try to get this to work in Catalog V2, as that will eventu It works with Catalog V2, but when catalog V2 + hms event polling, running a query triggers no metadata loading at all. Storage loading is part of the metastore loading, we can only verify it when metastore loading existing in profile. This test is just creating a test env to make sure this loading can really happen. The skip is to avoid flaky tests. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 09 Jul 2019 16:59:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13558 )
Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..
Patch Set 5:
Ran Hive 2 and Hive 3 core tests. For hive 3 tests, there are 4 failures and
are the same failures as upstream master :
Test Result (4 failures / -7)
custom_cluster.test_hive_parquet_codec_interop.TestParquetInterop.test_hive_impala_interop[protocol:
beeswax | exec_option: {'sync_ddl': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
parquet/none]
generate_junitxml.buildall.run-custom-cluster-tests
org.apache.impala.catalog.events.MetastoreEventsProcessorTest.testInsertEventOnRemovedTable
org.apache.impala.catalog.events.MetastoreEventsProcessorTest.testInsertEvents
https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5367/#showFailuresLink
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch-cdp-hive/20/#showFailuresLink
So there is no regression.
Because of recent cdp bumping from 107 to 195, Impala has to add capabilities
when creating tables. Most impala created table have OBJCAPABILITIES except for
kudu tables (seems that kudu managed table does not store extra table
properties). Therefore, kudu table will not check capabilities for now.
--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 5
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 08 Jul 2019 13:48:11 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#5). Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. IMPALA-8593: Prohibit write operations for bucketed tables This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Set default capabilities for tables. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests(Hive 2 and Hive 3) ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Enable capabilities checking for Kudu tables. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py 19 files changed, 390 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/5 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: (1 comment) The comment for skipping is added. http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py@55 PS2, Line 55: # that Catalog loads tables by itself, the q > Can you add this information in a comment? Done -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 14:34:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Hello Zoltan Borok-Nagy, Todd Lipcon, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13736 to look at the new patch set (#4). Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. IMPALA-8681: Only show ValidWriteIdLists for Acid tables Lists ValidWriteIds for transactional tables in profile. If a query does not trigger any transactional table loading, the query profile will not have the "Loaded ValidWriteIdLists" timeline. Tests: Manual tests. Fixed StmtMetadataLoaderTest. Added acid_profile test Sample output: Query Compilation: 3s525ms - Metadata load started: 37.369ms (37.369ms) - Metadata load finished. loaded-tables=1/1 ... - Loaded ValidWriteIdLists for transactional tables: functional.insert_only_transactional_table:0:9223372036854775807:: : 3s312ms (551.463us) - Analysis finished: 3s370ms (58.110ms) ... Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 --- M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java A testdata/workloads/functional-query/queries/QueryTest/acid-profile.test M tests/common/skip.py M tests/query_test/test_acid.py 5 files changed, 46 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13736/4 -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. Patch Set 4: Hive will treat all acid table as managed and all managed are needed to be transactional. Impala created tables will be external tables. The changing and discussion is still going on. This patch is just first step, we can make more fixes later. Following are related docs for HMS translation layer: Hive: https://docs.google.com/document/d/1ZmtMlKvnSz-EFGuCuYfPVA-2kfgHOWenL8sW47vkhnw/edit# Impala: https://docs.google.com/document/d/1lgcD_ZqIFVydLdRPQhkYBdFu2C0mH1A8VnXpcSs9psA/edit#heading=h.l9b51051hhma Three engines: https://docs.google.com/document/d/1fPIn1WW-tRRSmM-aGEVl0DI-TByIswo5HrLdgSPcw9s/edit?ts=5d1baa75 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 12:14:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13765 ) Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. Patch Set 3: (5 comments) Answer Sahil's question: IsNull Check may be necessary for other places, but use IsNullPredicate in requiresNullWrapping is not required. We can workaround it. Attach patch 4 with fixed tests. Run all core tests. http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG@16 PS3, Line 16: cannot both be null and non-null. But during its > This line is a bit confusing. Can you clarify what you mean here? Although we do not have the predict to tell if a value is null, we know that a valid should be either null or nonnull and cannot be both. http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG@17 PS3, Line 17: checking whether the wrapper needed, because of code > what do you mean by "because of code limitation"? We do not have IsNullPredicate for complex type, it is mentioned in the latter part of the message. I will make a change to add it here too. http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test File testdata/workloads/functional-query/queries/QueryTest/complex_joins.test: http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test@5 PS3, Line 5: create table array_tbl (col1 string, col2 array ) stored as parquet; > databases such as "functional_parquet" already have tables such as "complex This is the test case that the customer provided. And our functional table does not have data. http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test@35 PS3, Line 35: select t.col1, rank() over(order by t.col1) as rnk > is the analytical function necessary to re-produce the error? otherwise I t Yes, it is the customer's case, I must make sure the tests passed with/without data. Why create two: The testing framework parallels the tests which make test flaky if switching between impala and hive. http://gerrit.cloudera.org:8080/#/c/13765/3/tests/query_test/test_complex.py File tests/query_test/test_complex.py: http://gerrit.cloudera.org:8080/#/c/13765/3/tests/query_test/test_complex.py@23 PS3, Line 23: class TestComplex(ImpalaTestSuite): > You can just add this test to test_nested_types.py instead of creating a ne I will move -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 03 Jul 2019 09:11:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13765 to look at the new patch set (#4). Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, the planner makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation(IsNullPredicate is not supported for complex types), AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Ran all core tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test M tests/query_test/test_nested_types.py 3 files changed, 51 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/4 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: The original review is in https://gerrit.cloudera.org/c/13738/ -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 02 Jul 2019 17:04:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13786 Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: Profile:(storage-load-time is the added property): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/common/skip.py M tests/query_test/test_observability.py 8 files changed, 144 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13786/1 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13558 )
Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..
Patch Set 4:
(14 comments)
Patch set 4 addresses the issue and passed core tests for Hive2
http://gerrit.cloudera.org:8080/#/c/13558/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13558/3//COMMIT_MSG@9
PS3, Line 9:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@60
PS3, Line 60: import org.apache.impala.common.Imp
> I didn't find where we use this.
removed
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@64
PS3, Line 64: import org.apache.impala.service.MetadataOp;
> This is only mentioned in comment, but is not used.
removed
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@438
PS3, Line 438:* - insert-only Acid table read
:* - virtual view read
:* - materialized view read
:*/
> Can you make a bullet list from this?
Done
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@445
PS3, Line 445: ame = I
> Unused variable.
Done
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@451
PS3, Line 451: // Use MAJOR_VERSION for now
: // TODO: Change to impala build version when the functions
are fixed.
: String impalaId = St
> I do not understand the intention here.
Explain why I am using major version not impala version for impala id: The
function available in Impala does not work for catalog front end and causes for
troubles. For the issue does not affect the feature(any name will be fine),
comment here, we can change the name after the related functions fixed.
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@454
PS3, Line 454: String[] capabilities = new String[] {
: EXTWRITE, // External tab
> nit: one line
Done
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@466
PS3, Line 466: };
> nit: missing "is"
I think the "is" can be omitted, but I will add it anyway.
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@300
PS3, Line 300: Check if the table type is supported
> nit: "Check if the table type is supported" would be better.
Done
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@311
PS3, Line 311: // the operations are not supported, we will generate
error messages
> nit: missing "are"
This can be omitted too.
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@319
PS3, Line 319: ensureTableNotTransactional(ta
> Does this make sense in the MetastoreShim.getMajorVersion() == 2 case?
It is just a direct translation of the original code where
ensureTableNotFullAcid(table) used, for version 2, need check if it is
transactional. I will change to ensureTableNotTransactional(table)
And this kind of check is needed for transactional table use table properties,
it can go into version 2 DB.
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@365
PS3, Line 365: analyzer.ensureTableNotTransactional(table_);
> I also mentioned in AnalyzerTest.java that prohibiting COMPUTE STATS is str
Done, change back to the original implementation. I thought we cannot compute
accurate stats for the tables that we do not know how to write.
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java:
http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@897
PS3, Line 897: "is a bucketed table. Only read operations are supported
on such tables
> It is strange for me that DROP STATS is allowed, while COMPUTE STATS is not
Fixed.
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Sahil Takiar, Todd Lipcon, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13558 to look at the new patch set (#4). Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. IMPALA-8593: Prohibit write operations for bucketed tables This patch adds a method to check if a table bucketed. For Hive 3, integrates with HMS translation layer for capabilities checks. Implements methods ensureTableWriteSupported and ensureTableReadSupported. Tests: Added unit tests to ParserTest and AnalyzerTest. Added bucketed tables which are required by IMPALA-8439. Ran core tests. ToDo: Integrate checking bucketed tables capabilities and creating error messages with HMS translation after Hive provides the required functions. Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv 17 files changed, 346 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/4 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13765 to look at the new patch set (#3). Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, it makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation, AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test A tests/query_test/test_complex.py 3 files changed, 84 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/3 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13765 to look at the new patch set (#2). Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, it makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation, AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test A tests/query_test/test_complex.py 3 files changed, 83 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/2 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 3: (1 comment) patch 3 addresses the comment issue. http://gerrit.cloudera.org:8080/#/c/13736/2/testdata/workloads/functional-query/queries/QueryTest/acid-profile.test File testdata/workloads/functional-query/queries/QueryTest/acid-profile.test: http://gerrit.cloudera.org:8080/#/c/13736/2/testdata/workloads/functional-query/queries/QueryTest/acid-profile.test@16 PS2, Line 16: # Verify that ValidWriteIdLists is in the profile > I will change Done -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Jun 2019 21:54:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Hello Zoltan Borok-Nagy, Todd Lipcon, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13736 to look at the new patch set (#3). Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. IMPALA-8681: Only show ValidWriteIdLists for Acid tables Lists ValidWriteIds for transactional tables in profile. If a query does not trigger any transactional table loading, the query profile will not have the "Loaded ValidWriteIdLists" timeline. Tests: Manual tests. Fixed StmtMetadataLoaderTest. Added acid_profile test Sample output: Query Compilation: 3s525ms - Metadata load started: 37.369ms (37.369ms) - Metadata load finished. loaded-tables=1/1 ... - Loaded ValidWriteIdLists for transactional tables: functional.insert_only_transactional_table:0:9223372036854775807:: : 3s312ms (551.463us) - Analysis finished: 3s370ms (58.110ms) ... Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 --- M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java A testdata/workloads/functional-query/queries/QueryTest/acid-profile.test M tests/common/skip.py M tests/query_test/test_acid.py 5 files changed, 42 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13736/3 -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13765 Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, it makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation, AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test A tests/query_test/test_complex.py 3 files changed, 85 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/1 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
