[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 18: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5095/ -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 18 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 04:48:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. IMPALA-9028: impala-shell should not try to reconnect if quitting When the impala-shell is disconnected, it will try to reconnect for any command that a user runs (as part of ImpalaShell's precmd()). This doesn't make sense when the user is trying to quit the shell (i.e. by typing 'quit' or 'exit' or hitting Ctrl-D). This skips the attempt to reconnect when quitting the shell. Testing: - Added test in test_shell_interactive.py - Verified by hand Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Reviewed-on: http://gerrit.cloudera.org:8080/14391 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 23 insertions(+), 1 deletion(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8974: Create kudu table failed when catalogd connected to mysql directly instead of use HMS
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Create kudu table failed when catalogd connected to mysql directly instead of use HMS .. Patch Set 1: Thanks for your review, Tim. This bug is easy to reproduce in the dev environment, and I will try to add a regression test to invalidate this patch. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 1 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 16 Oct 2019 03:17:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14307 )
Change subject: IMPALA-7506: support global INVALIDATE METADATA in local
catalog mode
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1104
PS5, Line 1104: if (ctx.collectFullUpdates || tbl.getCatalogVersion() <=
ctx.toVersion) {
> I've tried tracking the min catalog versions in local-catalog mode coordina
For your concerns, I think it's ok since we already have the logic to force the
table being propogated if it has been skipped more than
MAX_NUM_SKIPPED_TOPIC_UPDATES(2) times. Why can't we do the same thing just
after global invalidate metadata?
If there are too many concurrent DDLs, global invalidate metadata may also hang
for a while in the existing implementation. Because the coordinator is waiting
for catalogd to propogate the skipped table, and catalogd is trying to
propogate it after skips it 2 times but waiting for acquiring the table lock
that is hold by the concurrent DDLs. (same as IMPALA-6671)
--
To view, visit http://gerrit.cloudera.org:8080/14307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Gerrit-Change-Number: 14307
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 16 Oct 2019 02:41:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly When the optimizer rewrites conjunct exprs with constant values, a new expr may cause AnalysisException. In this case, the conjuncts should use the original expr, not the intermediate expr produced by propagateConstants. Fixed optimizeConjuncts to handle this scenario properly. Tests: Add unit test for alias. Ran exhaustive tests. Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Reviewed-on: http://gerrit.cloudera.org:8080/14403 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M testdata/workloads/functional-query/queries/QueryTest/alias.test 2 files changed, 19 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 16 Oct 2019 02:04:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4804/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 01:51:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052: Events processor should skip blacklisted database and table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052: Events processor should skip blacklisted database and table .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4805/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Oct 2019 01:50:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 16 Oct 2019 01:46:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14458 Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. IMPALA-9053: DDLs should generate lineage graphs. DDLs like 'create table' should generate minimal lineage graphs so that consumers like Atlas can use information like 'queryText' to establish lineages. This change adds a call to the computeLineageGraph() method during analysis phase of createTable which populates the graph with basic information like queryText. If it is a CTAS, this graph is enhanced in the "insert" phase with dependencies. Testing: Add an EE test to verify lineage information and also to check it is flushed to disk properly. Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e --- M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/custom_cluster/test_lineage.py 4 files changed, 61 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/14458/1 -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-9052: Events processor should skip blacklisted database and table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052: Events processor should skip blacklisted database and table .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5096/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Oct 2019 01:09:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052: Events processor should skip blacklisted database and table
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052: Events processor should skip blacklisted database and table .. IMPALA-9052: Events processor should skip blacklisted database and table IMPALA-8797 introduced a configuration which can blacklist certain database and tables so that catalog never loads them. This is useful to skip the Hive's internal book-keeping tables which are unreadable by Impala (like Information schema and sys tables). However, if there are events generated on such tables, the events processor goes into error state because the database does not exist. This patch adds support for ignoring blacklisted database and tables in the events processor. Testing: 1. Added a new test case which makes sure that events processor is not in error state after receiving events on blacklisted tables and dbs. 2. Ran MetastoreEventsProcessorTest Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M tests/custom_cluster/test_event_processing.py 2 files changed, 42 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14457/3 -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 16 Oct 2019 00:38:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5095/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 18 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 00:33:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 18 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 00:33:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. Patch Set 2: Code-Review+2 (4 comments) http://gerrit.cloudera.org:8080/#/c/14457/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14457/2//COMMIT_MSG@7 PS2, Line 7: nit: I think we don't need space here http://gerrit.cloudera.org:8080/#/c/14457/2//COMMIT_MSG@9 PS2, Line 9: IMPALA-8797 introduced a configuration which can blacklist certain database and tables so nit: lines exceeding 72 characters. Though not part of our code style, it's more friendly for text wrapping in console tools. http://gerrit.cloudera.org:8080/#/c/14457/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/14457/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@214 PS2, Line 214: Iterator it = metastoreEvents.iterator(); nit: non-used variable http://gerrit.cloudera.org:8080/#/c/14457/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@227 PS2, Line 227: + "blacklisted database or table.")); nit: should we log the db and table for better debuggability? -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 15 Oct 2019 23:56:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8920: [DOCS] Documented the query option for disabling HBase row estimation
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14394 ) Change subject: IMPALA-8920: [DOCS] Documented the query option for disabling HBase row estimation .. IMPALA-8920: [DOCS] Documented the query option for disabling HBase row estimation Change-Id: Id131b66a3457ef6cbc326a0f3ed99de2a3950c3e Reviewed-on: http://gerrit.cloudera.org:8080/14394 Tested-by: Impala Public Jenkins Reviewed-by: Quanlong Huang --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_disable_hbase_num_rows_estimate.xml D docs/topics/impala_disable_outermost_topn.xml 4 files changed, 90 insertions(+), 49 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id131b66a3457ef6cbc326a0f3ed99de2a3950c3e Gerrit-Change-Number: 14394 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 17 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 23:51:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4803/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 15 Oct 2019 23:36:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. Patch Set 2: Code-Review+1 Looks good to me. Thanks for the change. -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 15 Oct 2019 23:41:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. Patch Set 3: Verified+1 I'm going to manually verify since the precommit tests won't use these scripts, and it already compiled and passed rat checks in the gerrit-code-review-checks job -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 23:34:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4802/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 15 Oct 2019 23:34:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. IMPALA-9034: fix distcc+ccache According to the ccache manpage, the correct way to compose it with distcc is to use CCACHE_PREFIX. I think this explains why local ccache wasn't working for me. I updated the distcc wrapper scripts to do this instead and confirmed that it works - after doing a clean and rebuilding the same branch, it is much faster and "ccache -s" shows a bunch of cache hit. ccache on the distcc server side still works. Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Reviewed-on: http://gerrit.cloudera.org:8080/14408 Reviewed-by: Andrew Sherman Tested-by: Tim Armstrong --- M be/CMakeLists.txt M bin/distcc/distcc.sh M bin/distcc/distcc_env.sh 3 files changed, 45 insertions(+), 38 deletions(-) Approvals: Andrew Sherman: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8920: [DOCS] Documented the query option for disabling HBase row estimation
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14394 ) Change subject: IMPALA-8920: [DOCS] Documented the query option for disabling HBase row estimation .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id131b66a3457ef6cbc326a0f3ed99de2a3950c3e Gerrit-Change-Number: 14394 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 23:23:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/14433 )
Change subject: IMPALA-7504 ParseKerberosPrincipal() should use
krb5_parse_name() instead
..
Patch Set 2:
(10 comments)
Thanks for a first cut at this.
I'm still not 100% convinced that this is the right thing to do. but whatever
happens it is a good training exercise.
I think the general questions are:
What is the performance impact of this change. Is this code on the main path of
a query?
Is the new function better at dealing with bad input than the old function?
http://gerrit.cloudera.org:8080/#/c/14433/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14433/2//COMMIT_MSG@8
PS2, Line 8:
If you look at other Impala commit message you will see that they usually try
to explain what is being done in the commit in some detail. I always imagine I
am explaining what I am doing to some other developer. This is a place to
discuss why as well as what you are doing, and any tradeoffs that were
considered.
You should also add a section explaining what testing you did
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc
File be/src/util/auth-util.cc:
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@90
PS2, Line 90: krb5_context g_krb5_ctx;
Maybe 'krb5_ctx' is a clearer name?
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@91
PS2, Line 91: CHECK_EQ(krb5_init_context(_krb5_ctx), 0);
Do you need unused_realm or to call krb5_get_default_realm?
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@93
PS2, Line 93: CHECK_EQ(krb5_get_default_realm(g_krb5_ctx, _realm), 0);
What is the performance impact of these extra allocation/deallocations?
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@97
PS2, Line 97: krb5_error_code code;
You could join with the next line:
krb5_error_code code = ...
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@99
PS2, Line 99: if(code != 0) {
Is there a test that generates TErrorCode::BAD_PRINCIPAL_FORMAT?
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@100
PS2, Line 100: return Status(TErrorCode::BAD_PRINCIPAL_FORMAT, principal);
Maybe need to call
krb5_free_context()
before returning
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@105
PS2, Line 105: char* p = data->data;
I think you can say
*service_name = data->data;
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@111
PS2, Line 111: for(int j = 0; j < data->length; j++) {
*hostname = data->data;
http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@114
PS2, Line 114: krb5_free_principal(g_krb5_ctx, princ);
Maybe need to call
krb5_free_context()
before returning
--
To view, visit http://gerrit.cloudera.org:8080/14433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24
Gerrit-Change-Number: 14433
Gerrit-PatchSet: 2
Gerrit-Owner: Xiaomeng Zhang
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Tue, 15 Oct 2019 23:16:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14457 ) Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. IMPALA-9052 : Events processor should skip blacklisted database and tables IMPALA-8797 introduced a configuration which can blacklist certain database and tables so that catalog never loads them. This is useful to skip the Hive's internal book-keeping tables which are unreadable by Impala (like Information schema and sys tables). However, if there are events generated on such tables, the events processor goes into error state because the database does not exist. This patch adds support for ignoring blacklisted database and tables in the events processor. Testing: 1. Added a new test case which makes sure that events processor is not in error state after receiving events on blacklisted tables and dbs. 2. Ran MetastoreEventsProcessorTest Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M tests/custom_cluster/test_event_processing.py 2 files changed, 40 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14457/2 -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9052 : Events processor should skip blacklisted database and tables
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14457 Change subject: IMPALA-9052 : Events processor should skip blacklisted database and tables .. IMPALA-9052 : Events processor should skip blacklisted database and tables IMPALA-8797 introduced a configuration which can blacklist certain database and tables so that catalog never loads them. This is useful to skip the Hive's internal book-keeping tables which are unreadable by Impala (like Information schema and sys tables). However, if there are events generated on such tables, the events processor goes into error state because the database does not exist. This patch adds support for ignoring blacklisted database and tables in the events processor. Testing: 1. Added a new test case which makes sure that events processor is not in error state after receiving events on blacklisted tables and dbs. 2. Ran MetastoreEventsProcessorTest Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M tests/custom_cluster/test_event_processing.py 2 files changed, 39 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14457/1 -- To view, visit http://gerrit.cloudera.org:8080/14457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic5a53b722e6225e9cad9954f447f821c2c677e60 Gerrit-Change-Number: 14457 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. Patch Set 3: Code-Review+2 (1 comment) Thanks for fixing the lock-up http://gerrit.cloudera.org:8080/#/c/14408/3/bin/distcc/distcc.sh File bin/distcc/distcc.sh: http://gerrit.cloudera.org:8080/#/c/14408/3/bin/distcc/distcc.sh@41 PS3, Line 41: echo "Expected IMPALA_DISTCC_ENV_VERSION=$EXPECTED_VERSION, re-source distcc_env.sh" Nice -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 22:30:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4801/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 22:08:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 21:44:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: This is the dockerhub outage causing image pull to fail, I will retry -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 21:43:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4800/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 21:50:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@75 PS6, Line 75: !KuduTable.isExternalTable(msTbl) Should this be checking for whether the table is synchronized? Are impala users able to create external tables today with the purge option? If so, do we need to update CatalogOpExecutor::createKuduTable() to also handle synchronized tables? -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 21:49:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5094/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 21:44:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5093/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 21:26:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 21:25:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14391 to look at the new patch set (#3). Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. IMPALA-9028: impala-shell should not try to reconnect if quitting When the impala-shell is disconnected, it will try to reconnect for any command that a user runs (as part of ImpalaShell's precmd()). This doesn't make sense when the user is trying to quit the shell (i.e. by typing 'quit' or 'exit' or hitting Ctrl-D). This skips the attempt to reconnect when quitting the shell. Testing: - Added test in test_shell_interactive.py - Verified by hand Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 23 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/14391/3 -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14391/2/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/14391/2/tests/shell/test_shell_interactive.py@243 PS2, Line 243: result = run_impala_shell_interactive(vector, 'quit;', shell_args=['-ifoo'], > Maybe add a case with a different command that does result in a "reconnect" Good point, added a case that does reconnect -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 21:22:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14391/2/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/14391/2/tests/shell/test_shell_interactive.py@243 PS2, Line 243: result = run_impala_shell_interactive(vector, 'quit;', shell_args=['-ifoo'], Maybe add a case with a different command that does result in a "reconnect"? -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 21:07:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4799/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 21:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5091/ -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 20:59:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: Added test and rebased -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 20:20:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14391 ) Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5092/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 15 Oct 2019 20:20:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9028: impala-shell should not try to reconnect if quitting
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14391 to look at the new patch set (#2). Change subject: IMPALA-9028: impala-shell should not try to reconnect if quitting .. IMPALA-9028: impala-shell should not try to reconnect if quitting When the impala-shell is disconnected, it will try to reconnect for any command that a user runs (as part of ImpalaShell's precmd()). This doesn't make sense when the user is trying to quit the shell (i.e. by typing 'quit' or 'exit' or hitting Ctrl-D). This skips the attempt to reconnect when quitting the shell. Testing: - Added test in test_shell_interactive.py - Verified by hand Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/14391/2 -- To view, visit http://gerrit.cloudera.org:8080/14391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6a76bc515db609498fa8772e9f0b0c547b82c09e Gerrit-Change-Number: 14391 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 5: > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/4797/ : > Initial code review checks failed. See linked job for details on > the failure. The build failure seems to be due to some maven issue. [INFO] BUILD FAILURE [ERROR] Plugin org.apache.maven.plugins:maven-surefire-plugin:2.12.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-surefire-plugin:jar:2.12.4: Could not transfer artifact org.apache.maven.plugins:maven-surefire-plugin:pom:2.12.4 from/to central (https://repo.maven.apache.org/maven2): /home/ubuntu/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom.part (No such file or directory) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 19:47:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: https://issues.apache.org/jira/browse/IMPALA-9049 https://issues.apache.org/jira/browse/IMPALA-9050 The related flaky tests jiras -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 19:07:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9003: [DOCS] Document per-directory limits for scratch disk usage
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14389 ) Change subject: IMPALA-9003: [DOCS] Document per-directory limits for scratch disk usage .. IMPALA-9003: [DOCS] Document per-directory limits for scratch disk usage - Removed the known issue for terabyte unit (IMPALA-8829). Change-Id: Id76e7650fb726b59883ecb112c2f86a32ba89d9b Reviewed-on: http://gerrit.cloudera.org:8080/14389 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M docs/shared/impala_common.xml M docs/topics/impala_disk_space.xml M docs/topics/impala_known_issues.xml 3 files changed, 203 insertions(+), 77 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id76e7650fb726b59883ecb112c2f86a32ba89d9b Gerrit-Change-Number: 14389 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4796/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 18:10:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4797/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 18:11:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4795/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 17:51:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14344 to look at the new patch set (#11). Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. IMPALA-8997: auto fallback to mt_dop=0 Add a temporary --mt_dop_auto_fallback to allow a graceful transition to using mt_dop for workloads. When this flag is set, DML queries and joins that would otherwise fail with an error when run with mt_dop > 0 fall back to running with mt_dop = 0. This means that a user can set mt_dop for their queries and it will only take effect when supported. The behaviour generally does not change when this flag is not set, with a couple of exceptions: * I made mt_dop automatic for compute stats on all file formats * mt_dop is allowed for single node plans with inserts. The quirky validatePlan() logic previously disallowed this but allowed joins in single node plans. The checks added by this patch can be removed safely once mt_dop is supported by default for all queries. This includes some cleanup: * isDmlStmt() was stale and incorrectly implemented. * Various TreeNode methods did not return instances of subclasses of the requested class, which was strange. This fix is required to make 'contains(JoinNode.class)' work correctly. I checked the callsites of the fixed functions and none of them would be affected by this change because they specified a terminal class without any subclasses. I didn't actually use this fix in the end (I had to write a custom tree traversal in hasUnsupportedMtDopJoin()), but figured I would leave the improvement in here. Testing: Add some basic functional tests ensuring that the fallback takes effect. Run basic join and insert tests with this flag enabled. Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-auto-fallback.test M tests/custom_cluster/test_mt_dop.py M tests/query_test/test_cancellation.py 16 files changed, 177 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/14344/11 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 6 files changed, 82 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14397/5 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8742: Switch to ScanRange::bytes to read() instead of len()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14348 ) Change subject: IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c Gerrit-Change-Number: 14348 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 15 Oct 2019 17:20:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14397 )
Change subject: IMPALA-9030: Handle translated external Kudu tables
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14397/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
File
fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java:
http://gerrit.cloudera.org:8080/#/c/14397/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@120
PS2, Line 120: isSynchronizedTable(tabl
> synchronized comes from whether a table is kept in-snyc with HMS in Kudu HM
I agree the naming could be useful. I synced up with Hao offline to understand
what it means when we say synchronized Kudu tables. I updated the change to
rename this method to synchronized table.
http://gerrit.cloudera.org:8080/#/c/14397/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:
http://gerrit.cloudera.org:8080/#/c/14397/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@142
PS3, Line 142: isExternalTable(msTbl) && !Boolean
:
.parseBoolean(msTbl.getParameters().get(TBL_PROP_EXTERNAL_TABLE_PURGE));
> This should be reversed now, since a synchronized table is either managed o
yeah thats right. I had noticed it but forgot to publish the patchset which
makes the appropriate changes.
http://gerrit.cloudera.org:8080/#/c/14397/3/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@151
PS3, Line 151: public static boolean
isExternalTable(org.apache.hadoop.hive.metastore.api.Table msTbl) {
> line too long (91 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/14397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36
Gerrit-Change-Number: 14397
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Greg Solovyev
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 15 Oct 2019 17:26:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8974: Create kudu table failed when catalogd connected to mysql directly instead of use HMS
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Create kudu table failed when catalogd connected to mysql directly instead of use HMS .. Patch Set 1: Is it possible to reproduce this in a dev environment? Would be good to include a regression test we we don't break it again. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 1 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 17:21:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. Patch Set 2: It's enough of a change you should take a quick look again -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 17:07:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14408 )
Change subject: IMPALA-9034: fix distcc+ccache
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14408/2/bin/distcc/distcc_env.sh
File bin/distcc/distcc_env.sh:
http://gerrit.cloudera.org:8080/#/c/14408/2/bin/distcc/distcc_env.sh@103
PS2, Line 103:
IMPALA_COMPILER_CONFIG_FILE="$IMPALA_HOME/.impala_compiler_opts_v${IMPALA_DISTCC_ENV_VERSION}"
line too long (94 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14408
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d
Gerrit-Change-Number: 14408
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 15 Oct 2019 17:06:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Hello Andrew Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14408 to look at the new patch set (#2). Change subject: IMPALA-9034: fix distcc+ccache .. IMPALA-9034: fix distcc+ccache According to the ccache manpage, the correct way to compose it with distcc is to use CCACHE_PREFIX. I think this explains why local ccache wasn't working for me. I updated the distcc wrapper scripts to do this instead and confirmed that it works - after doing a clean and rebuilding the same branch, it is much faster and "ccache -s" shows a bunch of cache hit. ccache on the distcc server side still works. Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d --- M be/CMakeLists.txt M bin/distcc/distcc.sh M bin/distcc/distcc_env.sh 3 files changed, 45 insertions(+), 38 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/14408/2 -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9034: fix distcc+ccache
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14408 ) Change subject: IMPALA-9034: fix distcc+ccache .. Patch Set 2: I tweaked this to fail the build when distcc_env.sh has not been re-sourced. I also locked up my computer so it seems worth avoiding this :) -- To view, visit http://gerrit.cloudera.org:8080/14408 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0b080709bd765056b9296d3deb805038fc01e5d Gerrit-Change-Number: 14408 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 15 Oct 2019 17:06:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: Can you file JIRAs for the flaky tests (or point to the current ones) if you haven't already. -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 16:59:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 8: Code-Review+1 Lgtm once Csaba's comments are resolved -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 15 Oct 2019 16:44:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5091/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 16:42:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9025: Handle AnalysisException caused by expr rewrites properly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14403 ) Change subject: IMPALA-9025: Handle AnalysisException caused by expr_rewrites properly .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic57bf3f4cdabfe9c5bb304d735bfbf1c0ca7a274 Gerrit-Change-Number: 14403 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 15 Oct 2019 16:42:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 6:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-parser.cc
File be/src/runtime/datetime-iso-sql-format-parser.cc:
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-parser.cc@225
PS6, Line 225: switch (**format) {
: case 'b': return '\b';
: case 'n': return '\n';
: case 'r': return '\r';
: case 't': return '\t';
: }
Please check what escape sequences are supported by ANSI SQL standard.
(same as datetime-parser-common.cc:226)
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc
File be/src/runtime/datetime-iso-sql-format-tokenizer.cc:
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@109
PS6, Line 109: *current_pos != nullptr
nit: current_pos != nullptr && *current_pos != nullptr
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@112
PS6, Line 112: *current_pos < str_end
nit: str_begin <= *current_pos && *current_pos < str_end
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@240
PS6, Line 240: strncmp(current_pos, "\\\"", 2) == 0)
This is safe only if 'current_pos' is zero-terminated, but I don't think it is.
Probably you should check first that there are at least 2 characters between
current_pos and end_pos.
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@273
PS6, Line 273: strncmp(*current_pos, "\\\"", 2)
same as L240.
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@277
PS6, Line 277: strncmp(*current_pos, "\\\"", 4)
same as L240.
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@282
PS6, Line 282: strncmp(*current_pos, "", 2)
same as L240.
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-parser-common.cc
File be/src/runtime/datetime-parser-common.cc:
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-parser-common.cc@201
PS6, Line 201: continue
nit: can be removed, here and below.
http://gerrit.cloudera.org:8080/#/c/14291/6/be/src/runtime/datetime-parser-common.cc@202
PS6, Line 202:} else if (!tok.is_double_escaped && strncmp(text_it, "\\\"",
2) == 0) {
: result.append("\"");
: ++text_it;
: continue;
: } else if (strncmp(text_it, "", 2) == 0) {
: result.append("\\");
: ++text_it;
: continue;
: } else if (strncmp(text_it, "\\b", 2) == 0) {
: result.append("\b");
: ++text_it;
: continue;
: } else if (strncmp(text_it, "\\n", 2) == 0) {
: result.append("\n");
: ++text_it;
: continue;
: } else if (strncmp(text_it, "\\r", 2) == 0) {
: result.append("\r");
: ++text_it;
: continue;
: } else if (strncmp(text_it, "\\t", 2) == 0) {
: result.append("\t");
: ++text_it;
: continue;
: }
Are these the only escape sequences supported by ANSI SQL?
C, for instance supports a bunch of others too:
https://en.wikipedia.org/wiki/Escape_sequences_in_C
http://gerrit.cloudera.org:8080/#/c/14291/6/tests/query_test/test_cast_with_format.py
File tests/query_test/test_cast_with_format.py:
http://gerrit.cloudera.org:8080/#/c/14291/6/tests/query_test/test_cast_with_format.py@663
PS6, Line 663:# Format part is surrounded by double quotes so the quotes
indicating the start and
: # end of the text token has to be escaped.
I did some testing around when the format string is surrounded by ' and the
text token is surrounded by \"
Date to string conversion:
>> select cast(date"1985-12-08" as string format '-MM-DD \"X\"');
returns: 1985-12-08 X
I'm not sure if this should work or return an error instead.
also if only the opening " to the text token is escaped:
>> select cast(date"1985-12-08" as string format '-MM-DD \"X"');
returns 1985-12-08
This should probably return an error.
On the other hand when doing string to date conversion:
>> select cast("1985-12-08 X" as date format '-MM-DD \"X\"');
returns: 1985-12-08
>> select cast("1985-12-08 X" as date format '-MM-DD \"X"');
returns an error.
>> select cast("1985-12-08" as date format '-MM-DD \"X"');
[Impala-ASF-CR] WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4794/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 15 Oct 2019 12:20:55 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/1 Change subject: WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs .. WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/common/TransactionKeepalive.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/stress/test_acid_stress.py 6 files changed, 186 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/1/1 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14400 ) Change subject: IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17e2b7bf9e36c54627276a6812b459912156cc3c Gerrit-Change-Number: 14400 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 11:32:36 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/1 )
Change subject: WIP: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
..
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/1/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/1/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@723
PS1, Line 723: LOG.info("Txn id (" + Long.toString(txnId) + ") lock "
+ Long.toString(lockId) + " is busy, waiting " +
line too long (113 > 90)
http://gerrit.cloudera.org:8080/#/c/1/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/1/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1917
PS1, Line 1917: throw new InternalException(String.format("Error
truncating table %s due to lock " +
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/1/1/tests/common/impala_test_suite.py
File tests/common/impala_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/1/1/tests/common/impala_test_suite.py@219
PS1, Line 219: #
flake8: E265 block comment should start with '# '
http://gerrit.cloudera.org:8080/#/c/1/1/tests/stress/test_acid_stress.py
File tests/stress/test_acid_stress.py:
http://gerrit.cloudera.org:8080/#/c/1/1/tests/stress/test_acid_stress.py@47
PS1, Line 47: def run_tasks(tasks):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/1/1/tests/stress/test_acid_stress.py@191
PS1, Line 191: n
flake8: E713 test for membership should be 'not in'
http://gerrit.cloudera.org:8080/#/c/1/1/tests/stress/test_acid_stress.py@244
PS1, Line 244: x
flake8: E501 line too long (113 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/1/1/tests/stress/test_acid_stress.py@245
PS1, Line 245: e
flake8: E501 line too long (128 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/1
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d
Gerrit-Change-Number: 1
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 15 Oct 2019 11:40:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14400 ) Change subject: IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES .. IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES If authorization is enabled, SHOW TABLES statements or GET_TABLES requests in HS2 protocol (used in HUE and JDBC drivers) will only return tables that the user has ANY privileges on them. If the user don't have any privileges on a table, we need 8 privilege checks (ALL, INSERT, SELECT, ALTER, CREATE, DROP, OWNER, REFRESH) to get this conclusion. It takes time in Sentry to check these one by one if there are thousands of tables. Unfortunately, there are no batch API for these checks. This introduces a performance regression after we supported fine-grained privileges, since before that we just check 3 privileges (ALL, INSERT, SELECT). In practice, SELECT privilege is the minimal privilege set. It's wired to grant INSERT or other privileges to a resource without SELECT privilege. We can simplify the process to only check on SELECT privilege if users make sure that SELECT privilege is the minimal privilege set in their environment. This patch adds a flag(SIMPLIFY_CHECK_ON_SHOW_TABLES) to bypass checking other privileges in SHOW TABLE statements. Testing in a database with 40k tables and granting the user SELECT privilege on only 6 tables. When using Sentry, the SHOW TABLES statement takes 5s. With the SIMPLIFY_CHECK_ON_SHOW_TABLES enabled, time reduces to 1.2s. No performance gain is observed when using Ranger since Ranger is fast enough. Tests: - Add custom cluster test for the flag in test_authorization.py for both Sentry and Ranger. - Run CORE tests Change-Id: I17e2b7bf9e36c54627276a6812b459912156cc3c Reviewed-on: http://gerrit.cloudera.org:8080/14400 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/authorization/test_authorization.py 7 files changed, 108 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17e2b7bf9e36c54627276a6812b459912156cc3c Gerrit-Change-Number: 14400 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9008: Serialize Maven invocations to deflake query-event-hook-api
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14423 ) Change subject: IMPALA-9008: Serialize Maven invocations to deflake query-event-hook-api .. Patch Set 2: Verified+1 Code-Review+2 Only the commit message was changed since the successful verification, so I submit this without running tests again. -- To view, visit http://gerrit.cloudera.org:8080/14423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ice0dfda5ee0ef60c9c208f05b0952555770fc090 Gerrit-Change-Number: 14423 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 15 Oct 2019 10:58:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9008: Serialize Maven invocations to deflake query-event-hook-api
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14423 ) Change subject: IMPALA-9008: Serialize Maven invocations to deflake query-event-hook-api .. IMPALA-9008: Serialize Maven invocations to deflake query-event-hook-api The previous commit for this Jira serialized the compilation of shaded-deps and ext-data-source to fix flakiness. It looks like this led to a new similar issue related to the parallel compilation of ext-data-source and query-event-hook-api. Similarly to the shaded-deps issue, I try to fix this by adding a fictional dependency (query-event-hook-api -> yarn-extras) to serialize the build. Full Maven dependency graph: fe -> query-event-hook-api -> yarn-extras -> ext-data-source -> shaded-deps-> impala-parent The ordering of query-event-hook-apit, yarn-extras, ext-data-source, and shaded-deps are arbitrary. Change-Id: Ice0dfda5ee0ef60c9c208f05b0952555770fc090 Reviewed-on: http://gerrit.cloudera.org:8080/14423 Reviewed-by: Joe McDonnell Reviewed-by: Csaba Ringhofer Tested-by: Csaba Ringhofer --- M query-event-hook-api/CMakeLists.txt 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Joe McDonnell: Looks good to me, approved Csaba Ringhofer: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ice0dfda5ee0ef60c9c208f05b0952555770fc090 Gerrit-Change-Number: 14423 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14400 ) Change subject: IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17e2b7bf9e36c54627276a6812b459912156cc3c Gerrit-Change-Number: 14400 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 07:12:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14400 ) Change subject: IMPALA-9002: Add query option to only check SELECT privilege in SHOW TABLES .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5090/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17e2b7bf9e36c54627276a6812b459912156cc3c Gerrit-Change-Number: 14400 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 15 Oct 2019 07:13:01 + Gerrit-HasComments: No
