[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Alice Fan has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2326 PS7, Line 2326: Returns a list of exprs such that for every i-th expr in that list, it is the first :* widest compatible expression encountered among all > sorry for changing the suggestions for the comments again and again, the te Sure. Thank you! http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2332 PS7, Line 2332: if (exprLists == null || exprLists.size() == 0) return null; > I think having an if statement for exprLists.size() == 1 wont hurt. Just re Done http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@161 PS7, Line 161: // Contains a list of exprs such that for every i-th expr in that list, it is the first : // widest compatible expressio > Contains a list of exprs such that for every i-th expr in that list, it is Done -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 8 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Thu, 09 May 2019 05:51:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Hello Bharath Vissapragada, Paul Rogers, xiaom...@cloudera.com, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13050 to look at the new patch set (#8). Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. IMPALA-966: Attribute type error to the right expression in an insert statement Currently, if an insert statement contains multiple expressions that are incompatible with the column type, the error message returned attributes the error to the wrong expression. This patch makes sure the right expression is blamed. If there are multiple incompatible type values for the target column, then the error is attributed to the first widest (highest precision) incompatible type expression. Testing: - Added tests to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 6 files changed, 84 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13050/8 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 8 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] expr-test: use gtest parameterization
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13289 to look at the new patch set (#2). Change subject: expr-test: use gtest parameterization .. expr-test: use gtest parameterization Instead of running the tests three times with different flags from main(), this uses gtest's parameterization feature to accomplish the same. The advantage here is that we end up with different test names for each of the runs. Additionally, this moves the setup code into a proper setup method so that executing expr-test --gtest_list_tests doesn't waste time starting a cluster. This is prep work towards adding multi-threaded test execution for long-running tests. expr-test seems to currently be one of the worst offenders. Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b --- M be/src/exprs/expr-test.cc 1 file changed, 135 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/13289/2 -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Allow running backend tests sharded and in parallel
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13290 to look at the new patch set (#2). Change subject: Allow running backend tests sharded and in parallel .. Allow running backend tests sharded and in parallel This improves the CMake functions for adding a test, adding a NUM_SHARDS parameter and management of the RUN_SERIAL propery. See the newly added documentation for the ADD_BE_TEST() cmake function for details. This patch enables the new functionality for 'expr-test' which is one of the slowest backend tests. To test this, I enabled parallel runs of expr-test and tested that 'ctest -j 8 -R expr-test' runs the shards in parallel, and that different subsets of test cases run in each shard. This command executed in 230 secs vs 871secs serially (so should save about 10 minutes of test time) Along the way I also split up the longest-running test case in expr-test so that it would shard more easily. I also implemented the same functionality for unified tests and used it in a few of the longer-running ones which I could verify were single-threaded and shouldn't have any cross-test conflicts. Change-Id: I0d44475e3f5a45c806f00d5003eeadf59683b009 --- M be/CMakeLists.txt M be/src/catalog/CMakeLists.txt M be/src/codegen/CMakeLists.txt M be/src/common/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/parquet/CMakeLists.txt M be/src/experiments/CMakeLists.txt M be/src/exprs/CMakeLists.txt M be/src/exprs/expr-test.cc M be/src/rpc/CMakeLists.txt M be/src/runtime/CMakeLists.txt M be/src/runtime/bufferpool/CMakeLists.txt M be/src/runtime/io/CMakeLists.txt M be/src/scheduling/CMakeLists.txt M be/src/service/CMakeLists.txt M be/src/statestore/CMakeLists.txt M be/src/util/CMakeLists.txt M bin/run-backend-tests.sh 18 files changed, 213 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13290/2 -- To view, visit http://gerrit.cloudera.org:8080/13290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d44475e3f5a45c806f00d5003eeadf59683b009 Gerrit-Change-Number: 13290 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13275 ) Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4198/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 09 May 2019 03:45:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7031: Add additional info to query canceled from http endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12926 ) Change subject: IMPALA-7031: Add additional info to query canceled from http endpoint .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3144/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12926 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56983d40e0542bc734ec5a66c339b5131b7b56c8 Gerrit-Change-Number: 12926 Gerrit-PatchSet: 10 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 09 May 2019 03:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7031: Add additional info to query canceled from http endpoint
Alice Fan has posted comments on this change. (
http://gerrit.cloudera.org:8080/12926 )
Change subject: IMPALA-7031: Add additional info to query canceled from http
endpoint
..
Patch Set 9:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-beeswax-server.cc@201
PS9, Line 201: ImpalaServer::
> nit: you can just call the method without this scope resolution.
Done
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-server.h@605
PS9, Line 605: // When a query is unregistered, its QueryStateRecord will be
added into query log
: // during the archive stage. This method is used to identify
whether the query is
: // unregistered and return a proper error message. The method
is used for unknown
: // query, which is a query lost its ClientRequestState.
> Returns the appropriate error message for an invalid query id. Also checks
Done
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-server.h@609
PS9, Line 609: getErrMsgForUnknownQuery
> nit: how about getErrMsgForInvalidQueryId
sure.
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12926/9/be/src/service/impala-server.cc@2589
PS9, Line 2589: Query is already unregistered
> I think we should still say invalid query handle in this case too, because
Done
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/hs2/test_hs2.py
File tests/hs2/test_hs2.py:
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/hs2/test_hs2.py@335
PS9, Line 335:
: assert "Query is already unregistered" or "Invalid query
handle" \
: in get_operation_status_resp.status.errorMessage
> ("a" or "b" in str) is the wrong conditional, please see the following link
Thanks!
Removed changes here. As error msg now will include 'Invalid query handle' in
anyway.
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/query_test/test_cancellation.py
File tests/query_test/test_cancellation.py:
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/query_test/test_cancellation.py@194
PS9, Line 194: ('Invalid query handle' in str(thread.fetch_results_error) or
: 'Query is already unregistered' in
str(thread.fetch_results_error)
: and not join_before_close)
> this becomes A || B & C, which is interpreted as A || (B & C)
Woops. Thanks for point this out!
I removed the "Query is already unregistered" from here, as it is an extra info
for "Invalid query handle"
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/webserver/test_web_pages.py@546
PS9, Line 546:
> add @pytest.mark.execute_serially decorator to this test. You should make s
Thanks!
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/webserver/test_web_pages.py@548
PS9, Line 548: """When a query is successfully canceled, client should be
notified that the
: query is already unregistered."""
> nit:how about: Make sure an indicative error message is returned when tryin
Done
http://gerrit.cloudera.org:8080/#/c/12926/9/tests/webserver/test_web_pages.py@562
PS9, Line 562: # Because the query is unregistered, client should receive
exception with
: # "Query is already unregistered" or "Invalid query
handle" notice.
: # For the query gets "Invalid query handle" is because
query log might be
: # full and then removed the oldest unregistered query
> nit: you can remove this comment after you make the change to run this seri
Done
--
To view, visit http://gerrit.cloudera.org:8080/12926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56983d40e0542bc734ec5a66c339b5131b7b56c8
Gerrit-Change-Number: 12926
Gerrit-PatchSet: 9
Gerrit-Owner: Alice Fan
Gerrit-Reviewer: Alice Fan
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 09 May 2019 03:16:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7031: Add additional info to query canceled from http endpoint
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12926 to look at the new patch set (#10). Change subject: IMPALA-7031: Add additional info to query canceled from http endpoint .. IMPALA-7031: Add additional info to query canceled from http endpoint When a running query is canceled from the http endpoint, the query will be unregistered. This can result in a scenario where a client that has a reference to the query handle is unaware that the query has been unregistered via http endpoint, and attempts to run an operation on that query handle but encounters "Invalid query handle" error. >From the user prospective the error is not indicative, therefore the patch added additional information to the error to notify that the query is already unregistered. Testing: - Added a test to test_web_pages.py - Ran all webserver tests Change-Id: I56983d40e0542bc734ec5a66c339b5131b7b56c8 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M tests/webserver/test_web_pages.py 5 files changed, 55 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/12926/10 -- To view, visit http://gerrit.cloudera.org:8080/12926 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56983d40e0542bc734ec5a66c339b5131b7b56c8 Gerrit-Change-Number: 12926 Gerrit-PatchSet: 10 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7176: bump timeout for test insert mem limit
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/13278 ) Change subject: IMPALA-7176: bump timeout for test_insert_mem_limit .. Abandoned will go with michael's patch -- To view, visit http://gerrit.cloudera.org:8080/13278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I8eab7f2df46b588b3f0f8e948f713db95ebf7348 Gerrit-Change-Number: 13278 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13292 ) Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 03:11:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13292 ) Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. Patch Set 1: Let's just go with this one -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 03:11:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13292 ) Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3143/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 03:03:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13292 ) Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. Patch Set 1: Didn't know you were working on the same thing. Oh well..may be you can update the commit message of your patch to include that there was testing done with timeout of 180s. -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 01:43:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 7: (4 comments) Looks good just a few more nits http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2332 PS6, Line 2332: exprLists.size() == > Thanks for catching this! yes it will be but in that case you will have to change the description of this function to not return anything if exprLists.size() == 1. I think its more consistent if we return something for a non empty exprLists http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2326 PS7, Line 2326: Return a list of the first widest compatible expression for each column in an :* order consistent with the order of target columns. sorry for changing the suggestions for the comments again and again, the terminology is kinda tricky. I think the following should be sufficient and generic enough to make sense in the context of this method's input params: Returns a list of exprs such that for every i-th expr in that list, it is the first widest compatible expression encountered among all i-th exprs in the expr lists. http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2332 PS7, Line 2332: if (exprLists == null || exprLists.size() == 0) return null; I think having an if statement for exprLists.size() == 1 wont hurt. Just return exprLists.get(0) in that case. http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/13050/7/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@161 PS7, Line 161: // this contains a list of the first widest (highest precision) compatible : // expression for each column. Contains a list of exprs such that for every i-th expr in that list, it is the first widest compatible expression encountered among all i-th exprs in every result expr list of the union operands. -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 7 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Thu, 09 May 2019 01:19:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3142/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 09 May 2019 01:15:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13282 ) Change subject: IMPALA-8524: Avoid calling "hive" via command line in EE tests .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4195/ -- To view, visit http://gerrit.cloudera.org:8080/13282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I874ac344ffd176ffd7b8540d57126a7026f4c4f6 Gerrit-Change-Number: 13282 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 09 May 2019 01:06:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13292 ) Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. Patch Set 1: I wrote almost the same patch. It looks like you did more validation so maybe we should go with yours - https://gerrit.cloudera.org/#/c/13278/ -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 01:05:25 + Gerrit-HasComments: No
[Impala-ASF-CR] expr-test: use gtest parameterization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13289 ) Change subject: expr-test: use gtest parameterization .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3140/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:58:50 + Gerrit-HasComments: No
[Impala-ASF-CR] expr-test: use gtest parameterization
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13289 ) Change subject: expr-test: use gtest parameterization .. Patch Set 1: This is pretty mechanical so I'm probably OK to +2 once the TODOs are removed. -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:56:24 + Gerrit-HasComments: No
[Impala-ASF-CR] expr-test: use gtest parameterization
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13289 ) Change subject: expr-test: use gtest parameterization .. Patch Set 1: (1 comment) This seems like an improvement. I think the previous approach was just from lack of knowledge about gtest. I know I did something similar with buffer-pool-test too :) http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc@282 PS1, Line 282: // TODO(todd): why are these settings only for this case? > I think this only needs to be for (!disable_codegen_). These are important Actually this can just be unconditional because it has no effect if codegen is disabled. I looked at the original code and it was an artefact of how that code was structured instead of intentional. -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:55:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] expr-test: use gtest parameterization
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13289 ) Change subject: expr-test: use gtest parameterization .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc@282 PS1, Line 282: // TODO(todd): why are these settings only for this case? I think this only needs to be for (!disable_codegen_). These are important in that case so codegen doesn't get auto-disabled. We could just set them unconditionally though even so that the planner doesn't do anything clever in any cases. http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc@9296 PS1, Line 9296: // TODO(todd) why don't we test the false/true case? IIRC just because it didn't add much coverage and adds a lot to test runtime (running this test with codegen enabled is really slow). It was a bit of a fight to even convince people that we should run it in one configuration with codegen enabled. Mostly we get the best backend codegened/interpreted coverage from running the unmodified (i.e. more complex) expr trees enabled. Running the trees in interpreted mode should be enough to validate correctness of the rewrites. So enabling the remaining combination might provide some additional incidental coverage from codegening some additional expr tree shapes but mostly it isn't that interesting since the majorit y of expressions get folded to a constant anyway. -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:53:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7176: Increase wait time in test insert mem limit
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13292 Change subject: IMPALA-7176: Increase wait time in test_insert_mem_limit .. IMPALA-7176: Increase wait time in test_insert_mem_limit test_insert_mem_limit in test_insert.py will wait for all fragments to exit before proceeding after the test query hits a memory limit. On some slower EC2 instances with Centos6, the default wait time of 60s may occasionally cause the test to fail due to time out waiting for all fragments to exit. This change increases the timeout to 180 seconds. Testing done: - Looped test_insert_mem_limit for 100 times on Centos6; Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 --- M tests/query_test/test_insert.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/13292/1 -- To view, visit http://gerrit.cloudera.org:8080/13292 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2e14bef79c6c6fb0004270319f1c491194260438 Gerrit-Change-Number: 13292 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] WIP: Allow running tests sharded and in parallel
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13290 ) Change subject: WIP: Allow running tests sharded and in parallel .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3141/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d44475e3f5a45c806f00d5003eeadf59683b009 Gerrit-Change-Number: 13290 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:43:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. Patch Set 5: Rebased to fix the Maven hang issue. -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 09 May 2019 00:42:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. IMPALA-8528: Refactor authorization check in AnalysisContext This patch moves the authorization check logic from AnalysisContext into BaseAuthorizationChecker to consolidate the logic into a single place. This patch also converts AuthorizationChecker into an interface and it also introduces preAuthorize() and postAuthorize() methods to add custom logic before and after the authorization check. The existing implementation code of AuthorizationChecker is now moved to BaseAuthorizationChecker. This patch has no functionality change. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationContext.java A fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 8 files changed, 419 insertions(+), 283 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13285/5 -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3139/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 09 May 2019 00:28:28 + Gerrit-HasComments: No
[native-toolchain-CR] Update cmake to 3.14.3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13286 ) Change subject: Update cmake to 3.14.3 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Gerrit-Change-Number: 13286 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 09 May 2019 00:11:26 + Gerrit-HasComments: No
[Impala-ASF-CR] expr-test: use gtest parameterization
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13289 )
Change subject: expr-test: use gtest parameterization
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:
http://gerrit.cloudera.org:8080/#/c/13289/1/be/src/exprs/expr-test.cc@4811
PS1, Line 4811: TEST_P(ExprTest, StringParseUrlFunction) { // TODO: For now,
our parse_url my not behave exactly like Hive
line too long (106 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b
Gerrit-Change-Number: 13289
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 09 May 2019 00:13:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] expr-test: use gtest parameterization
Hello Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13289 to review the following change. Change subject: expr-test: use gtest parameterization .. expr-test: use gtest parameterization Instead of running the tests three times with different flags from main(), this uses gtest's parameterization feature to accomplish the same. The advantage here is that we end up with different test names for each of the runs. Additionally, this moves the setup code into a proper setup method so that executing expr-test --gtest_list_tests doesn't waste time starting a cluster. This is prep work towards adding multi-threaded test execution for long-running tests. expr-test seems to currently be one of the worst offenders. Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b --- M be/src/exprs/expr-test.cc 1 file changed, 126 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/13289/1 -- To view, visit http://gerrit.cloudera.org:8080/13289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idc9fb24ad62b4aa2e120a99d74ae04bb221c034b Gerrit-Change-Number: 13289 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13275 ) Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 09 May 2019 00:12:10 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Allow running tests sharded and in parallel
Hello Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13290 to review the following change. Change subject: WIP: Allow running tests sharded and in parallel .. WIP: Allow running tests sharded and in parallel This improves the CMake functions for adding a test, adding a NUM_SHARDS parameter and management of the RUN_SERIAL propery. For testing purposes, I enabled parallel runs of expr-test and tested that 'ctest -j 8 -R expr-test' runs the shards in parallel. Change-Id: I0d44475e3f5a45c806f00d5003eeadf59683b009 --- M be/CMakeLists.txt M be/src/catalog/CMakeLists.txt M be/src/codegen/CMakeLists.txt M be/src/common/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/parquet/CMakeLists.txt M be/src/experiments/CMakeLists.txt M be/src/exprs/CMakeLists.txt M be/src/rpc/CMakeLists.txt M be/src/runtime/CMakeLists.txt M be/src/runtime/bufferpool/CMakeLists.txt M be/src/runtime/io/CMakeLists.txt M be/src/scheduling/CMakeLists.txt M be/src/service/CMakeLists.txt M be/src/statestore/CMakeLists.txt M be/src/util/CMakeLists.txt 16 files changed, 135 insertions(+), 77 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13290/1 -- To view, visit http://gerrit.cloudera.org:8080/13290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0d44475e3f5a45c806f00d5003eeadf59683b009 Gerrit-Change-Number: 13290 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3136/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 09 May 2019 00:06:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4197/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 23:58:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13288 Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. IMPALA-8072: addendum: don't require fe rebuild for config Previously config changes wouldn't be picked up by containers until maven copied the files from fe/src/test/resources to fe/target/test-classes. This makes it more convenient - after running ./bin/create-test-configuration.sh new configs are picked up by any newly-run containers. Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 --- M bin/start-impala-cluster.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/13288/1 -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13275 ) Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3138/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 23:54:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3135/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 May 2019 23:44:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4192/ -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 23:34:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: bump timeout for test insert mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13278 ) Change subject: IMPALA-7176: bump timeout for test_insert_mem_limit .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4196/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8eab7f2df46b588b3f0f8e948f713db95ebf7348 Gerrit-Change-Number: 13278 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 23:40:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. Patch Set 2: I terminated clang-tidy-ub1604 because it was stuck. You probably want to rebase. -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 23:31:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3134/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 23:31:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Drop statestore update frequency during data loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13260 ) Change subject: Drop statestore update frequency during data loading .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4191/ -- To view, visit http://gerrit.cloudera.org:8080/13260 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8931a88aa04e0b4e8ef26a92bfe50a539a3c2505 Gerrit-Change-Number: 13260 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 23:32:49 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4193/ -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 23:35:08 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 3: I killed the test job because it appeared to be stuck. You probably want to rebase and try again. -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 23:35:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: I killed the test job because it appeared to be stuck. You probably want to rebase and try again. -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 23:34:01 + Gerrit-HasComments: No
[Impala-ASF-CR] Drop statestore update frequency during data loading
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13260 ) Change subject: Drop statestore update frequency during data loading .. Patch Set 3: I killed the test job because it appeared to be stuck. You probably want to rebase and try again. -- To view, visit http://gerrit.cloudera.org:8080/13260 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8931a88aa04e0b4e8ef26a92bfe50a539a3c2505 Gerrit-Change-Number: 13260 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 23:33:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3137/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 7 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Wed, 08 May 2019 23:30:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13275 ) Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13275/1/be/src/util/jni-util.h File be/src/util/jni-util.h: http://gerrit.cloudera.org:8080/#/c/13275/1/be/src/util/jni-util.h@395 PS1, Line 395: tls_env > nit: _ on end to be consistent. Done -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 23:20:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13275 to look at the new patch set (#2). Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. IMPALA-2029. Implement our own getJNIEnv equivalent The libhdfs getJNIEnv function was made non-exported in Hadoop 2. For a while in CDH we were hacking around this with a vendor-specific patch that re-exported it. However, that was always a bit annoying to maintain our own patch each time we rebased to new versions, etc. Earlier attempts to solve this issue turned up strange bugs around coordinating whether we or libhdfs were responsible for attaching and detaching to the JVM/JNI environment. So, this patch takes a new approach: rather than directly creating/attaching to the JVM, we just look for an existing attached environment. If there isn't one, we call some simple libhdfs function which forces it to attach the current thread, and then try again. Performance is maintained (or maybe improved) by adding a thread-local cache of the attached JVM, with an inlined fast-path. I tested this with a CDP build of Hadoop which doesn't have the getJNIEnv workaround. Prior to this fix, I wasn't able to run Java tests against that build because it would fail to link getJNIEnv() at runtime. Now, they pass. Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 --- M be/src/catalog/catalog-util.cc M be/src/catalog/catalog.cc M be/src/exec/external-data-source-executor.cc M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-table-scanner.cc M be/src/exec/hbase-table-writer.cc M be/src/exprs/hive-udf-call.cc M be/src/runtime/hbase-table-factory.cc M be/src/runtime/hbase-table.cc M be/src/scheduling/request-pool-service.cc M be/src/service/fe-support.cc M be/src/service/frontend.cc M be/src/util/jni-util.cc M be/src/util/jni-util.h M be/src/util/logging-support.cc M be/src/util/zip-util.cc 16 files changed, 91 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/13275/2 -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3133/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 May 2019 23:06:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13282 ) Change subject: IMPALA-8524: Avoid calling "hive" via command line in EE tests .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4195/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I874ac344ffd176ffd7b8540d57126a7026f4c4f6 Gerrit-Change-Number: 13282 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 08 May 2019 22:25:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13282 ) Change subject: IMPALA-8524: Avoid calling "hive" via command line in EE tests .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I874ac344ffd176ffd7b8540d57126a7026f4c4f6 Gerrit-Change-Number: 13282 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 08 May 2019 22:25:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Hello Bharath Vissapragada, Paul Rogers, xiaom...@cloudera.com, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13050 to look at the new patch set (#7). Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. IMPALA-966: Attribute type error to the right expression in an insert statement Currently, if an insert statement contains multiple expressions that are incompatible with the column type, the error message returned attributes the error to the wrong expression. This patch makes sure the right expression is blamed. If there are multiple incompatible type values for the target column, then the error is attributed to the first widest (highest precision) incompatible type expression. Testing: - Added tests to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 6 files changed, 82 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13050/7 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 7 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Alice Fan has posted comments on this change. (
http://gerrit.cloudera.org:8080/13050 )
Change subject: IMPALA-966: Attribute type error to the right expression in an
insert statement
..
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2332
PS6, Line 2332: exprLists.size() < 2
> looks like if exprLists.size() =1 then it should return exprLists.get(0)
Thanks for catching this!
Originally, I thought the exprLists.size() == 1 case will be handle by
statementBase.checkTypeCompatibility(). widestExpr == null, basically will be
replaced by srcExpr, which is the exprLists.get(0).
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2348
PS6, Line 2348: if (preType != compatibleType) {
: widestExprs.set(i, exprLists.get(j).get(i));
: }
> nit: might fit in one line
Done
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@699
PS6, Line 699: Expr widestTypeExpr = null;
: if (widestTypeExprList != null) {
: widestTypeExpr = widestTypeExprList.get(i);
: }
> nit: can be one line
Done
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java:
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@55
PS6, Line 55: // Widest (highest precision) expression is a list of the first
widest compatible
: // expression for each column. The list is stored in the order
of target columns.
: protected List widestExprs_ = new ArrayList<>();
> this needs to go after L133 ( in the list of members that need to be reset(
Moved into the list of "members that need to be reset()".
Remove the part for stored in order, because there are other list members
doesn't not explain the order. Also, the order is explained at comment area of
analyzer.castToUnionCompatibleTypes() when the widestExprs is returned.
reset() method has been updated at previous patch. which is at line648. also
update clone at line 199
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@3404
PS6, Line 3404: on
> nit: typo: blame the widest
Updated comments.
http://gerrit.cloudera.org:8080/#/c/13050/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@3407
PS6, Line 3407:
> can you add a one line comment for each test case as to what it is testing.
added short comment for each of test.
--
To view, visit http://gerrit.cloudera.org:8080/13050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39
Gerrit-Change-Number: 13050
Gerrit-PatchSet: 6
Gerrit-Owner: Alice Fan
Gerrit-Reviewer: Alice Fan
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Wed, 08 May 2019 22:18:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4194/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 19 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 22:15:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7176: bump timeout for test insert mem limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13278 ) Change subject: IMPALA-7176: bump timeout for test_insert_mem_limit .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4190/ -- To view, visit http://gerrit.cloudera.org:8080/13278 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8eab7f2df46b588b3f0f8e948f713db95ebf7348 Gerrit-Change-Number: 13278 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 22:01:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13285 ) Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. IMPALA-8528: Refactor authorization check in AnalysisContext This patch moves the authorization check logic from AnalysisContext into BaseAuthorizationChecker to consolidate the logic into a single place. This patch also converts AuthorizationChecker as an interface and introduces BaseAuthorizationChecker that contains authorization helper methods. This patch has no functionality change. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 7 files changed, 372 insertions(+), 283 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13285/2 -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] [IMPALA-8281] Sentry frontend decoupling
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13284 )
Change subject: [IMPALA-8281] Sentry frontend decoupling
..
Patch Set 3:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/13284/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13284/3//COMMIT_MSG@7
PS3, Line 7: [IMPALA-8281]
nit: IMPALA-8281:
http://gerrit.cloudera.org:8080/#/c/13284/3/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/13284/3/common/thrift/Frontend.thrift@267
PS3, Line 267: // 2: required bool is_admin_op
add: REMOVED - prefix
http://gerrit.cloudera.org:8080/#/c/13284/3/common/thrift/Frontend.thrift@295
PS3, Line 295: // 4: required bool is_admin_op
add: REMOVED - prefix
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java
File fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java:
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java@36
PS3, Line 36: loaded
nit: authorization provider?
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java@39
PS3, Line 39: public class GrantRevokePrivStmt extends AuthorizationStmt {
can we refactor this logic here:
https://github.com/apache/impala/blob/4c6ac151ef2c6efac8a8a3d02c342334bf9d688c/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java#L105-L107
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java@44
PS3, Line 44: private final TPrincipalType principalType_;
can we also refactor the GrantRevokeRoleStmt?
https://github.com/apache/impala/blob/4c6ac151ef2c6efac8a8a3d02c342334bf9d688c/fe/src/main/java/org/apache/impala/analysis/GrantRevokeRoleStmt.java#L60-L62
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java@106
PS3, Line 106: SHOW ROLE
typo: SHOW ROLES?
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/13284/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java@118
PS3, Line 118: }
We don't want to reduce the test coverage, if it's not possible to test it in
FE test, can we write an E2E test instead? Make sure we have tests for whatever
code that we refactor.
--
To view, visit http://gerrit.cloudera.org:8080/13284
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539
Gerrit-Change-Number: 13284
Gerrit-PatchSet: 3
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 08 May 2019 21:42:02 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Update cmake to 3.14.3
Hello Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13286 to review the following change. Change subject: Update cmake to 3.14.3 .. Update cmake to 3.14.3 This updates to the latest cmake, which includes a few improvements that may be helpful. In particular, the Ninja generator avoids generating some false dependencies[1] Instead of carrying over the patch to change one of the #defines in the CMake source, this uses a newly-provided variable when building CMake to accomplish the same. I tested building Impala with this version of CMake on el7 and it worked fine. [1] https://gitlab.kitware.com/cmake/cmake/merge_requests/430 Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 --- M init.sh M source/cmake/build.sh 2 files changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/86/13286/1 -- To view, visit http://gerrit.cloudera.org:8080/13286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Gerrit-Change-Number: 13286 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [DOCS] Fixed more typos for SHOW SQL statements
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13273 ) Change subject: [DOCS] Fixed more typos for SHOW SQL statements .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13273/1/docs/topics/impala_show.xml File docs/topics/impala_show.xml: http://gerrit.cloudera.org:8080/#/c/13273/1/docs/topics/impala_show.xml@429 PS1, Line 429: SHOW GRANT ROLE Statement Sorry, I may actually make a mistake here, but this doc is also confusing. It looks like "SHOW GRANT ROLE" has already been documented in L467. I think this statement was meant to be "SHOW ROLE GRANT GROUP". Without "GROUP" I thought it was a typo :) http://gerrit.cloudera.org:8080/#/c/13273/1/docs/topics/impala_show.xml@442 PS1, Line 442: SHOW GRANT ROLE fix this to "SHOW ROLE GRANT GROUP" -- To view, visit http://gerrit.cloudera.org:8080/13273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2a903e229ba1875fdc23feab33ee69e9cd0142d Gerrit-Change-Number: 13273 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 21:35:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8281] Sentry frontend decoupling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13284 ) Change subject: [IMPALA-8281] Sentry frontend decoupling .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3130/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 Gerrit-Change-Number: 13284 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 21:19:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h@125 PS6, Line 125: void SetLocalBeDescFn(BackendDescriptorPtrFn fn); > Unfortunately I don't think these Set*Fn() functions are really thread-safe I ended up with the second approach. Deferring Init() of the ClusterMembershipMgr until the ImpalaServer registers itself with the ExecEnv also would have meant to defer the initialization of the Scheduler and AdmissionController and I found the additional synchronization easier to reason about than changing our initialization order and interlacing it with ImpalaServer::Start(). Let me know if you feel that that's what we should do. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 May 2019 21:28:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Hello Michael Ho, Thomas Marshall, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13207 to look at the new patch set (#8). Change subject: IMPALA-8460: Simplify cluster membership management .. IMPALA-8460: Simplify cluster membership management This change adds a class to track cluster membership called ClusterMembershipMgr. It replaced the logic that was partially duplicated between the ImpalaServer and the Coordinator and makes sure that the local backend descriptor is consistent (IMPALA-8469). The ClusterMembershipMgr maintains a view of the cluster membership and incorporates incoming updates from the statestore. It also registers the local backend with the statestore after startup. Clients can obtain a consistent, immutable snapshot of the current cluster membership from the ClusterMembershipMgr. Additionally, callbacks can be registered to receive notifications of cluster membership changes. The ImpalaServer and Frontend use this mechanism. This change also unifies the naming of executor-related classes, in particular it renames "BackendConfig" to "ExecutorGroup". In anticipation of a subsequent change, it adds maps to store multiple executor groups. Testing: This change does not introduce new functionality and the new class is covered by the existing scheduler unit test and the end to end tests. Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 --- M be/src/benchmarks/scheduler-benchmark.cc M be/src/common/logging.h M be/src/gutil/strings/split.cc M be/src/gutil/strings/split.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/CMakeLists.txt D be/src/scheduling/backend-config-test.cc D be/src/scheduling/backend-config.cc D be/src/scheduling/backend-config.h A be/src/scheduling/cluster-membership-mgr.cc A be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-group-test.cc A be/src/scheduling/executor-group.cc A be/src/scheduling/executor-group.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M common/thrift/StatestoreService.thrift M tests/custom_cluster/test_coordinators.py 26 files changed, 1,177 insertions(+), 923 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13207/8 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 ) Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3132/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e Gerrit-Change-Number: 13178 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 21:20:15 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-8281] Sentry frontend decoupling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13284 ) Change subject: [IMPALA-8281] Sentry frontend decoupling .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3131/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 Gerrit-Change-Number: 13284 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 21:19:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8528: Refactor authorization check in AnalysisContext
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13285 Change subject: IMPALA-8528: Refactor authorization check in AnalysisContext .. IMPALA-8528: Refactor authorization check in AnalysisContext This patch moves the authorization check logic from AnalysisContext into BaseAuthorizationChecker to consolidate the logic into a single place. This patch also converts AuthorizationChecker as an interface and introduces BaseAuthorizationChecker that contains authorization helper methods. This patch has no functionality change. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 7 files changed, 372 insertions(+), 283 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13285/1 -- To view, visit http://gerrit.cloudera.org:8080/13285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3bc3a11220dae0f49ef3e73d9ff27a90e9d4a71c Gerrit-Change-Number: 13285 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 21:21:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13282 ) Change subject: IMPALA-8524: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4187/ -- To view, visit http://gerrit.cloudera.org:8080/13282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I874ac344ffd176ffd7b8540d57126a7026f4c4f6 Gerrit-Change-Number: 13282 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 08 May 2019 21:20:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8516. Update maven for Jenkins builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13268 ) Change subject: IMPALA-8516. Update maven for Jenkins builds .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4189/ -- To view, visit http://gerrit.cloudera.org:8080/13268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I793409eb4e9f4533b75bfe089a497c0ea62ad1ff Gerrit-Change-Number: 13268 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 21:20:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13207 )
Change subject: IMPALA-8460: Simplify cluster membership management
..
Patch Set 7:
(3 comments)
Had a few comments, still need to complete review
http://gerrit.cloudera.org:8080/#/c/13207/5/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/13207/5/be/src/runtime/exec-env.cc@454
PS5, Line 454: impala_server_ = server;
: // Register the ImpalaServer with the cluster membership
manager
: cluster_membership_mgr_->SetLocalBeDescFn([server]() {
: return server->GetLocalBackendDescriptor();
: });
: cluster_membership_mgr_->SetUpdateLocalServerFn(
: [server](const ClusterMembershipMgr::BackendAddressSet&
current_backends) {
:
> Thanks the DCHECK would definitely help. I was also just thinking of a situ
We shouldn't be running ~ExecEnv outside of backend tests AFAIK.
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h
File be/src/scheduling/cluster-membership-mgr.h:
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h@125
PS6, Line 125: /// membership. This callback will only be called when
backends are deleted from the
Unfortunately I don't think these Set*Fn() functions are really thread-safe,
and it looks like Init() registers ClusterMembershipMgr with the statestore
subscriber, so it's possible that a statestore update could race with these
functions getting updated.
It's probably unlikely to lead to a bug so long as these functions are always
set from non-NULL to NULL, but it hard to reason about given that std::function
isn't just a simple pointer, so the the reader could see a "torn" state. I
think it would also trip TSAN, etc.
I think we could solve it two ways:
* Require that all the callbacks are registered before Init() is called.
* Add some synchronisations around the function pointers. I don't think any of
this is on a critical path, so I wouldn't be too concerned about a simple
SpinLock.
I think the first would be cleaner, although I understand there are some
initialisation order issues between ExecEnv and ImpalaServer. Maybe we can
defer initialisation of ClusterMembershipMgr until after SetImpalaServer() is
called or something like that?
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.cc@54
PS6, Line 54: void
ClusterMembershipMgr::SetLocalBeDescFn(BackendDescriptorPtrFn fn) {
Maybe DCHECK that it's non-NULL here and in the other functions below.
--
To view, visit http://gerrit.cloudera.org:8080/13207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3
Gerrit-Change-Number: 13207
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 08 May 2019 20:51:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/13207 )
Change subject: IMPALA-8460: Simplify cluster membership management
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13207/5/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/13207/5/be/src/runtime/exec-env.cc@454
PS5, Line 454: // Register the ImpalaServer with the cluster membership
manager
: cluster_membership_mgr_->SetLocalBeDescFn([server]() {
: return server->GetLocalBackendDescriptor();
: });
: cluster_membership_mgr_->SetUpdateLocalServerFn(
: [server](const ClusterMembershipMgr::BackendAddressSet&
current_backends) {
: server->CancelQueriesOnFailedBackends(current_backends);
: });
> Thanks the DCHECK would definitely help. I was also just thinking of a situ
I think we have this problem in every place where we call
ExecEnv::impala_server(). The server doesn't unregister itself and the ExecEnv
doesn't own it. Furthermore that holds true for all code that depends on
ExecEnv, since it doesn't implement a d'tor itself and we rely on GetInstance()
to return a valid instance.
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h
File be/src/scheduling/cluster-membership-mgr.h:
http://gerrit.cloudera.org:8080/#/c/13207/6/be/src/scheduling/cluster-membership-mgr.h@54
PS6, Line 54: ions of changes to the cluster
: /// membership.
> should this also be after a TODO tag?
I don't feel strongly here. The class already has code in place that supports
this, and consumers should expect it. However, it's currently not used by any
client code. The TODO below in L57 captures the intent to add the missing parts
in IMPALA-8484, so I removed the sentence altogether.
--
To view, visit http://gerrit.cloudera.org:8080/13207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3
Gerrit-Change-Number: 13207
Gerrit-PatchSet: 5
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 08 May 2019 20:51:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Hello Michael Ho, Thomas Marshall, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13207 to look at the new patch set (#7). Change subject: IMPALA-8460: Simplify cluster membership management .. IMPALA-8460: Simplify cluster membership management This change adds a class to track cluster membership called ClusterMembershipMgr. It replaced the logic that was partially duplicated between the ImpalaServer and the Coordinator and makes sure that the local backend descriptor is consistent (IMPALA-8469). The ClusterMembershipMgr maintains a view of the cluster membership and incorporates incoming updates from the statestore. It also registers the local backend with the statestore after startup. Clients can obtain a consistent, immutable snapshot of the current cluster membership from the ClusterMembershipMgr. Additionally, callbacks can be registered to receive notifications of cluster membership changes. The ImpalaServer and Frontend use this mechanism. This change also unifies the naming of executor-related classes, in particular it renames "BackendConfig" to "ExecutorGroup". In anticipation of a subsequent change, it adds maps to store multiple executor groups. Testing: This change does not introduce new functionality and the new class is covered by the existing scheduler unit test and the end to end tests. Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 --- M be/src/benchmarks/scheduler-benchmark.cc M be/src/common/logging.h M be/src/gutil/strings/split.cc M be/src/gutil/strings/split.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/CMakeLists.txt D be/src/scheduling/backend-config-test.cc D be/src/scheduling/backend-config.cc D be/src/scheduling/backend-config.h A be/src/scheduling/cluster-membership-mgr.cc A be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-group-test.cc A be/src/scheduling/executor-group.cc A be/src/scheduling/executor-group.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M common/thrift/StatestoreService.thrift M tests/custom_cluster/test_coordinators.py 26 files changed, 1,159 insertions(+), 923 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13207/7 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 7: Code-Review+1 Carrying Bikram's +1 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 May 2019 20:51:11 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4193/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 20:24:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
Abhishek Rawat has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13178 ) Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage. .. IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage. The ExecNode::num_rows_returned_ member is shared by the scan node (thread) and the related scanner threads. There is a potential race here since the compiler is free to reorder the load/store of num_rows_returned_ member. Also, num_rows_returned_ is not accessed in a thread safe manner. This change introduces regular and thread safe functions for accessing and updating the num_rows_returned_ member, which is now private to the ExecNode class. The getters/modifiers for single threaded code-paths are: rows_returned() ReachedLimit() SetNumRowsReturned() IncrementNumRowsReturned() DecrementNumRowsReturned() CheckLimitAndTruncateRowBatchIfNeeded() Thread safe counterparts are: rows_returned_shared() ReachedLimitShared() IncrementNumRowsReturnedShared() DecrementNumRowsReturnedShared() CheckLimitAndTruncateRowBatchIfNeededShared() Debug checks are added to ensure that regular and thread safe functions are used properly in single-threaded and multi-threaded code paths, respectively. The initial attempt to address this issue by making num_rows_returned_ atomic caused performance regressions in single threaded code paths which update the variable inside a loop, such as ExchangeNode::GetNext(). The regression could be attributed to extra path length introduced by atomic operations such as read modify write. And since the GetNext() function could be called multiple times, with each call updating the atomic multiple times, the regression could be huge. Some serious thoughts were given to other approaches such as updating the atomic num_rows_returned_ variable outside of a loop, but such a change would have been very extensive and error prone. There is nothing stopping someone from not adding atomic updates in a tight loop. After much deliberation it was decided that the single threaded code path should not be penalised and as such it probably makes sense to use separate functions for single threaded and multi-threaded code paths, provided the interfaces are simple enough, easy to maintain and there are checks in place to ensure proper use. The scan nodes with task based multi-threading (MT) also use the new thread-safe functions in the code-paths shared with the non MT scan nodes, even though the variable is not shared amongst various threads. This is because both the task based MT code-path and the regular scan node code path calls some common functions and it was getting very tricky and ugly to differentiate between the two. The overhead of using atomics is minimal for MT code path since the atomic is not accessed for every row. The only exceptions are HBaseScanNode and DataSourceScanNode which uses the regular functions. Testing: - Performance tests were run on tpch and tpcds workload. No real regressions were found, but both baseline and test runs showed high degree of variability for some queries. Multiple runs were done and it was concluded that the variability is not introduced by this patch. Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e --- M be/src/exec/aggregation-node.cc M be/src/exec/analytic-eval-node.cc M be/src/exec/cardinality-check-node.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/exchange-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-rcfile-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scan-node-mt.cc M be/src/exec/kudu-scan-node-mt.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/kudu-scanner.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/select-node-ir.cc M be/src/exec/select-node.cc M be/src/exec/sort-node.cc M be/src/exec/streaming-aggregation-node.cc M be/src/exec/subplan-node.cc M be/src/exec/topn-node.cc M be/src/exec/union-node.cc M be/src/exec/unnest-node.cc 36 files changed, 235 insertions(+), 137 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/7 -- To view, visit http://gerrit.cloudera.org:8080/13178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e Gerrit-Change-Number:
[Impala-ASF-CR] IMPALA-2029. Implement our own getJNIEnv equivalent
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13275 ) Change subject: IMPALA-2029. Implement our own getJNIEnv equivalent .. Patch Set 1: Code-Review+2 (1 comment) This seems to make sense to me. http://gerrit.cloudera.org:8080/#/c/13275/1/be/src/util/jni-util.h File be/src/util/jni-util.h: http://gerrit.cloudera.org:8080/#/c/13275/1/be/src/util/jni-util.h@395 PS1, Line 395: tls_env nit: _ on end to be consistent. -- To view, visit http://gerrit.cloudera.org:8080/13275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I766bcfd70addb00e9fd8a860e89c2a1c5d4c71d5 Gerrit-Change-Number: 13275 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 20:27:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 6: Verified+1 It was still somewhat slow in the download phase, but made it in the end. So I'll merge this and continue to keep an eye on things. -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 20:14:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. IMPALA-8527: prune maven repo dependencies We transitively pull in references to repository.apache.org, which in turn means we'll go looking for most of our dependencies there. Downloading from repository.apache.org occasionally hangs, so there's a high probability of a build getting stuck. I was able to disable repository.apache.org entirely - the same packages are available from other repositories that we don't see the same issues with. Locating snapshot versions is very chatty - we reach out to mvnrepository.com and apache.org repeatedly, but I don't think we actually need to consume any snapshots from them. So I tried to minimise the number of repositories that we'll consume snapshots from - I think we only intend to download snapshots from the CDH repo. Also remove the plugin snapshots repository. We historically needed it because we used a snapshot version of the cup plugin, but that was fixed by IMPALA-3418. Otherwise depending on plugin snapshots seems like a bad idea. Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Reviewed-on: http://gerrit.cloudera.org:8080/13279 Reviewed-by: Fredy Wijaya Reviewed-by: Todd Lipcon Tested-by: Tim Armstrong --- M impala-parent/pom.xml 1 file changed, 38 insertions(+), 9 deletions(-) Approvals: Fredy Wijaya: Looks good to me, but someone else must approve Todd Lipcon: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3126/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 20:04:02 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-8281] Sentry frontend decoupling
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13284 to look at the new patch set (#3). Change subject: [IMPALA-8281] Sentry frontend decoupling .. [IMPALA-8281] Sentry frontend decoupling This patch moves Sentry specific code from the Frontend into the SentryImpaladAuthorizationManager. Testing: - Ran all FE tests - Ran E2E authorization tests Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 --- M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowRolesStmt.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java 7 files changed, 49 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/13284/3 -- To view, visit http://gerrit.cloudera.org:8080/13284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 Gerrit-Change-Number: 13284 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [IMPALA-8281] Sentry frontend decoupling
Austin Nobis has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13284 Change subject: [IMPALA-8281] Sentry frontend decoupling .. [IMPALA-8281] Sentry frontend decoupling This patch moves Sentry specific code from the Frontend into the SentryImpaladAuthorizationManager. Testing: - Ran all FE tests - Ran E2E authorization tests Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 --- M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowRolesStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java 8 files changed, 50 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/13284/2 -- To view, visit http://gerrit.cloudera.org:8080/13284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id24a00dd395e30e4c392f085893e9561da2ee539 Gerrit-Change-Number: 13284 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 19: The job was stuck because of the maven issues - killed to free up some resources. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 19 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 19:55:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8516. Update maven for Jenkins builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13268 ) Change subject: IMPALA-8516. Update maven for Jenkins builds .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3124/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13268 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I793409eb4e9f4533b75bfe089a497c0ea62ad1ff Gerrit-Change-Number: 13268 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 19:46:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 19: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4183/ -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 19 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 19:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3129/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 19:52:49 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 2: No Builds Executed -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 19:52:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3128/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 19:52:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3127/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 19:53:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4186/ -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 19:51:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4185/ -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 19:50:50 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13185 )
Change subject: fe: set classpath using maven dependency resolution
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13185/1/bin/set-classpath.sh
File bin/set-classpath.sh:
http://gerrit.cloudera.org:8080/#/c/13185/1/bin/set-classpath.sh@34
PS1, Line 34: ="$IMPALA_HOME/fe/target/build-classpath.txt"
> maybe add an error message that if this file does not exist, we should buil
Done
http://gerrit.cloudera.org:8080/#/c/13185/1/docker/setup_build_context.py
File docker/setup_build_context.py:
http://gerrit.cloudera.org:8080/#/c/13185/1/docker/setup_build_context.py@66
PS1, Line 66: }".fo
> nit: use {0} instead of +
Done
--
To view, visit http://gerrit.cloudera.org:8080/13185
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94
Gerrit-Change-Number: 13185
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 08 May 2019 19:21:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4192/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 19:18:55 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Hello Vihang Karajgaonkar, Fredy Wijaya, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13185 to look at the new patch set (#2). Change subject: fe: set classpath using maven dependency resolution .. fe: set classpath using maven dependency resolution This changes the FE pom to generate a build classpath file in the target/ directory. Then, bin/set-classpath.sh uses this file to generate the classpath to start the cluster. This replaces the former approach of including all of the jars found in target/dependency/ The advantage of this is that a clean build is no longer required when switching artifact versions. Prior to this patch, if you changed an artifact version and rebuilt, both the old and new artifact would be left in the target/dependency/ directory and pollute the classpath. This doesn't fully remove the target/dependency/ directory, because its existence is likely important for downstream packaging of Impala. We can likely assume that such packaging always does a clean build. This also changes the set-classpath script to no longer load jars from testdata/target/dependency/ since it appears that directory doesn't actually get created during the build. Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 --- M bin/set-classpath.sh M docker/setup_build_context.py M fe/pom.xml 3 files changed, 36 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13185/2 -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13283/3/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13283/3/tests/common/impala_test_suite.py@738 PS3, Line 738: get_partition_names is this a change of behavior in hive client in version 3? If yes, is it breaking backwards compatibility? http://gerrit.cloudera.org:8080/#/c/13283/3/tests/common/skip.py File tests/common/skip.py: http://gerrit.cloudera.org:8080/#/c/13283/3/tests/common/skip.py@216 PS3, Line 216: == should this be >= 3 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 May 2019 19:25:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 6: https://jenkins.impala.io/job/gerrit-code-review-checks/3126/flowGraphTable/ is running it right now, let's see what happens -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 19:25:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Drop statestore update frequency during data loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13260 ) Change subject: Drop statestore update frequency during data loading .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4191/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13260 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8931a88aa04e0b4e8ef26a92bfe50a539a3c2505 Gerrit-Change-Number: 13260 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 May 2019 19:09:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13283/2/tests/common/skip.py File tests/common/skip.py: http://gerrit.cloudera.org:8080/#/c/13283/2/tests/common/skip.py@214 PS2, Line 214: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 19:11:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13283/2/tests/common/skip.py File tests/common/skip.py: http://gerrit.cloudera.org:8080/#/c/13283/2/tests/common/skip.py@214 PS2, Line 214: class SkipIfHive3: flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2019 19:09:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8527: prune maven repo dependencies
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13279 ) Change subject: IMPALA-8527: prune maven repo dependencies .. Patch Set 6: Code-Review+2 Let's get this in as soon as we see the build-only job succeed :) -- To view, visit http://gerrit.cloudera.org:8080/13279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08e1f1b7d7742edd61179ee52b5e268c3b4dc61d Gerrit-Change-Number: 13279 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 08 May 2019 19:17:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Csaba Ringhofer has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. IMPALA-8369: Fixing some core tests in Hive environment Fixes: impala_test_suite.py: DROP PARTITIONS in the SETUP section of test files did not work with Hive 3, because 'max_parts' argument of hive_client.get_partition_names() was 0, while it should be -1 to return all partitions. The issue broke sevaral 'insert' tests. all other changed files: query/test_mt_dop.py:test_parquet_filtering was broken because Hive 3 generates different number of files for tpch_nested_parquet.customer than Hive 2. Tests that use this table were moved to a seperate .test file, which is not run in Hive 3 builds. Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de --- A testdata/workloads/functional-query/queries/QueryTest/parquet-filtering-tpch-nested-customer.test M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test M tests/common/environ.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/query_test/test_mt_dop.py 6 files changed, 151 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13283/2 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13283 to look at the new patch set (#3). Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. IMPALA-8369: Fixing some core tests in Hive environment Fixes: impala_test_suite.py: DROP PARTITIONS in the SETUP section of test files did not work with Hive 3, because 'max_parts' argument of hive_client.get_partition_names() was 0, while it should be -1 to return all partitions. The issue broke sevaral 'insert' tests. all other changed files: query/test_mt_dop.py:test_parquet_filtering was broken because Hive 3 generates different number of files for tpch_nested_parquet.customer than Hive 2. Tests that use this table were moved to a seperate .test file, which is not run in Hive 3 builds. Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de --- A testdata/workloads/functional-query/queries/QueryTest/parquet-filtering-tpch-nested-customer.test M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test M tests/common/environ.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/query_test/test_mt_dop.py 6 files changed, 152 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13283/3 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
