[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8890 Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test 9 files changed, 2,171 insertions(+), 1,064 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/1 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
[Impala-ASF-CR] KUDU-2198. Allow disregarding system-wide auth-to-local mapping
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/8875 ) Change subject: KUDU-2198. Allow disregarding system-wide auth-to-local mapping .. Patch Set 1: Code-Review+2 This is a straight-forward cherry pick. -- To view, visit http://gerrit.cloudera.org:8080/8875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e893493f52965ea54d2ceaac83d375285b49486 Gerrit-Change-Number: 8875 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 20 Dec 2017 02:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5054: [SECURITY] Enable KRPC w/ TLS in Impala
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8439 ) Change subject: IMPALA-5054: [SECURITY] Enable KRPC w/ TLS in Impala .. Patch Set 1: The backport for the fix of KUDU-2228 is being reviewed here: https://gerrit.cloudera.org/#/c/8878/ -- To view, visit http://gerrit.cloudera.org:8080/8439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9a14a44fdea9ab668f3714eb69fdb188bce38f5a Gerrit-Change-Number: 8439 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Wed, 20 Dec 2017 01:08:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8853 ) Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zach Amsden Gerrit-Comment-Date: Tue, 19 Dec 2017 23:28:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8880 ) Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Revert "IMPALA-3887: Wait for HDFS replication in data loading" Using fsck breaks non-HDFS builds: local, S3, and Isilon. This reverts commit 5a7c10ec3d500c1bb582e985cb299e7dbcfe5520. Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Reviewed-on: http://gerrit.cloudera.org:8080/8880 Reviewed-by: Michael Brown Tested-by: Impala Public Jenkins --- M testdata/bin/create-load-data.sh 1 file changed, 0 insertions(+), 19 deletions(-) Approvals: Michael Brown: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Gerrit-Change-Number: 8880 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8880 ) Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Gerrit-Change-Number: 8880 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 19 Dec 2017 23:26:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters.
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/8349 )
Change subject: IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set
of characters.
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/8349/4/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/8349/4/be/src/exprs/string-functions-ir.cc@475
PS4, Line 475: tionContext* context,
> If we're going to skip on is_null, then I think we have to stick with passi
Sure, we can keep passing StringVal(" ") for the no-arg case. It's not used
anyway.
That said, we still need to skip if chars_to_trim.is_null == true because this
essentially means chars_to_trim is undefined. In fact, we should have a test
case for it.
http://gerrit.cloudera.org:8080/#/c/8349/6/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/8349/6/be/src/exprs/string-functions-ir.cc@451
PS6, Line 451: direction == LEADING || direction == BOTH
To avoid a logic or, we can consider doing:
if ((direction & LEADING) != 0) {
}
if ((direction & TRAILING) != 0) {
}
For the case of BOTH, you can pass direction as (LEADING | TRAILING). Of
course, that requires redefining LEADING and TRAILING to bitmaps.
http://gerrit.cloudera.org:8080/#/c/8349/6/be/src/exprs/string-functions-ir.cc@464
PS6, Line 464: //return DoTrimStringImpl(str, unique_chars, direction);
delete
http://gerrit.cloudera.org:8080/#/c/8349/6/be/src/exprs/string-functions-ir.cc@466
PS6, Line 466:
: StringVal StringFunctions::Trim(FunctionContext* context, const
StringVal& str) {
: return DoTrimString(context, str, StringVal(" "), BOTH);
: }
:
: StringVal StringFunctions::Ltrim(FunctionContext* context, const
StringVal& str) {
: return DoTrimString(context, str, StringVal(" "), LEADING);
: }
:
: StringVal StringFunctions::Rtrim(FunctionContext* context, const
StringVal& str) {
: return DoTrimString(context, str, StringVal(" "), TRAILING);
: }
Can you please do a quick benchmark to make sure we didn't regress the perf of
these expressions with this change ?
It seems that we almost should templatize StringFunctions::DoTrimString().
--
To view, visit http://gerrit.cloudera.org:8080/8349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79
Gerrit-Change-Number: 8349
Gerrit-PatchSet: 6
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Tue, 19 Dec 2017 22:59:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/8758 )
Change subject: IMPALA-6190/6246: Add instances tab and event sequence
..
Patch Set 8:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/coordinator-backend-state.cc@626
PS8, Line 626: DCHECK_EQ(instance_stats.Size(), fragments_.size());
: value->AddMember("instance_stats", instance_stats,
document->GetAllocator());
:
: Value val(TNetworkAddressToString(impalad_address()).c_str(),
document->GetAllocator());
: value->AddMember("host", val, document->GetAllocator()
Same question. Can these lines be done outside of the lock ?
http://gerrit.cloudera.org:8080/#/c/8758/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/8758/4/be/src/runtime/coordinator.cc@1279
PS4, Line 1279: doc->AddMember("backend_instances", states,
doc->GetAllocator());
Can this be done without holding the lock ?
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/fragment-instance-state.h
File be/src/runtime/fragment-instance-state.h:
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/fragment-instance-state.h@172
PS8, Line 172: Emitted at the start of Prepare() after the
: /// instance's event sequence has been set up.
nit: just wondering if we can skip the "Emitted at ..." given it's quite
obvious from the code.
http://gerrit.cloudera.org:8080/#/c/8758/4/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/8758/4/be/src/runtime/fragment-instance-state.cc@487
PS4, Line 487:
> There are no updating threads, but there's still a race between updating it
Can you please add a small comment to make the intention of AtomicEnum clear so
readers will know that this function is not meant to be thread safe.
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/runtime/fragment-instance-state.cc@248
PS8, Line 248: if (runtime_state_->ShouldCodegen()) {
Should we start counting CODEGEN_START at this point instead after Prepare()
has been called on the exec tree and the sink has been created ? Conceptually,
it is a new phrase.
In fact, I wonder if we should actually consider merging this with the similar
if-stmt in Open().
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:
http://gerrit.cloudera.org:8080/#/c/8758/8/be/src/util/runtime-profile-counters.h@338
PS8, Line 338: while (i < timestamps.size() && timestamps[i] <=
last_timestamp) ++i;
: for (; i < timestamps.size(); ++i) {
Is this in the perf critical path ? Can we merge these two loops ? Seems easier
to follow that way.
--
To view, visit http://gerrit.cloudera.org:8080/8758
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9
Gerrit-Change-Number: 8758
Gerrit-PatchSet: 8
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Tue, 19 Dec 2017 22:42:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8541 ) Change subject: IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module .. Patch Set 12: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/8541/12/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/8541/12/be/src/codegen/llvm-codegen.cc@859 PS12, Line 859: NULL, NULL nullptr, nullptr http://gerrit.cloudera.org:8080/#/c/8541/12/be/src/codegen/llvm-codegen.cc@1082 PS12, Line 1082: fn->print(stream, nullptr, false, true); Feel free to ignore but do you think it will be sufficient to just print the function name ? I am a bit worried about potential log spew if we print the function body when we run queries with types (e.g. char) which we will always not codegen for. -- To view, visit http://gerrit.cloudera.org:8080/8541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If975cfb3906482b36dd6ede32ca81de6fcee1d7f Gerrit-Change-Number: 8541 Gerrit-PatchSet: 12 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Dec 2017 21:58:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module
Hello Michael Ho, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8541 to look at the new patch set (#12). Change subject: IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module .. IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module Currently, if an error is encountered during the creation of a handcrafted codegen method, then the resulting IR is left in an incomplete state. This patch ensures that all such IRs are cleaned up (method is deleted from the module) before the llvm module is finalized. Testing: - added a backend test to exercise the added code path. - tested manually by executing the following query: select * from charTable A, charTable B where A.charColumn = B.charColumn and A.charColumn = 'foo'; and looking at the logs to verify that 'InsertRuntimeFilters' and 'FilterContextInsert' methods have been removed. Change-Id: If975cfb3906482b36dd6ede32ca81de6fcee1d7f --- M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/hdfs-text-scanner.cc 4 files changed, 99 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/8541/12 -- To view, visit http://gerrit.cloudera.org:8080/8541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If975cfb3906482b36dd6ede32ca81de6fcee1d7f Gerrit-Change-Number: 8541 Gerrit-PatchSet: 12 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/8541 )
Change subject: IMPALA-6177: Cleanup incomplete handcrafted IRs before
finalizing module
..
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc
File be/src/codegen/llvm-codegen-test.cc:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@111
PS11, Line 111: for(auto fn : codegen->handcrafted_functions_){
: if(fn == function) return true;
: }
: return false;
> const auto& hf = codegen->handcrafted_functions_;
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@498
PS11, Line 498: NULL
> nullptr
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@506
PS11, Line 506: NULL
> nullptr
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@507
PS11, Line 507: NULL
> nullptr
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h
File be/src/codegen/llvm-codegen.h:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h@228
PS11, Line 228: NULL
> nullptr
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h@228
PS11, Line 228: NULL
> nullptr
Done
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.cc
File be/src/codegen/llvm-codegen.cc:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.cc@182
PS11, Line 182: NULL
> nullptr
Done
--
To view, visit http://gerrit.cloudera.org:8080/8541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If975cfb3906482b36dd6ede32ca81de6fcee1d7f
Gerrit-Change-Number: 8541
Gerrit-PatchSet: 11
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 19 Dec 2017 21:44:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6177: Cleanup incomplete handcrafted IRs before finalizing module
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/8541 )
Change subject: IMPALA-6177: Cleanup incomplete handcrafted IRs before
finalizing module
..
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc
File be/src/codegen/llvm-codegen-test.cc:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@111
PS11, Line 111: for(auto fn : codegen->handcrafted_functions_){
: if(fn == function) return true;
: }
: return false;
const auto& hf = codegen->handcrafted_functions_;
return find(hf.begin(), hf.end(), function) != hf.end();
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@498
PS11, Line 498: NULL
nullptr
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@506
PS11, Line 506: NULL
nullptr
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen-test.cc@507
PS11, Line 507: NULL
nullptr
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h
File be/src/codegen/llvm-codegen.h:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h@228
PS11, Line 228: NULL
nullptr
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.h@228
PS11, Line 228: NULL
nullptr
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.cc
File be/src/codegen/llvm-codegen.cc:
http://gerrit.cloudera.org:8080/#/c/8541/11/be/src/codegen/llvm-codegen.cc@182
PS11, Line 182: NULL
nullptr
--
To view, visit http://gerrit.cloudera.org:8080/8541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If975cfb3906482b36dd6ede32ca81de6fcee1d7f
Gerrit-Change-Number: 8541
Gerrit-PatchSet: 11
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 19 Dec 2017 20:42:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8880 ) Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1638/ -- To view, visit http://gerrit.cloudera.org:8080/8880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Gerrit-Change-Number: 8880 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 19 Dec 2017 19:51:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/8880 ) Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Gerrit-Change-Number: 8880 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 19 Dec 2017 19:22:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8880 Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Revert "IMPALA-3887: Wait for HDFS replication in data loading" Using fsck breaks non-HDFS builds: local, S3, and Isilon. This reverts commit 5a7c10ec3d500c1bb582e985cb299e7dbcfe5520. Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc --- M testdata/bin/create-load-data.sh 1 file changed, 0 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/8880/2 -- To view, visit http://gerrit.cloudera.org:8080/8880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc Gerrit-Change-Number: 8880 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8853 ) Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 19 Dec 2017 19:08:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8853 ) Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Patch Set 2: (3 comments) Thanks for the review, Thomas. Good catch on IMPALA_JAVA_THRIFT_VERSION. I updated the change, and added the commons.io change which was hanging out uncommitted. https://jenkins.impala.io/job/gerrit-verify-dryrun-external/40/ is running with this new patch http://gerrit.cloudera.org:8080/#/c/8853/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/8853/2/bin/impala-config.sh@142 PS2, Line 142: IMPALA_THRIFT_JAVA_VERSION > Is this still being used anywhere now? Should have been :) http://gerrit.cloudera.org:8080/#/c/8853/2/impala-parent/pom.xml File impala-parent/pom.xml: http://gerrit.cloudera.org:8080/#/c/8853/2/impala-parent/pom.xml@48 PS2, Line 48: 2.4 I changed this too; missed a diff in my first post. http://gerrit.cloudera.org:8080/#/c/8853/2/impala-parent/pom.xml@57 PS2, Line 57: 0.9.0< > Should this be env.IMPALA_THRIFT_VERSION? Good eye; this should be IMPALA_JAVA_THRIFT_VERSION. -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 19 Dec 2017 19:01:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Vuk Ercegovac has abandoned this change. ( http://gerrit.cloudera.org:8080/8879 ) Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Abandoned going w/ dave's patch -- To view, visit http://gerrit.cloudera.org:8080/8879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I4318b4fcb94aa67ac55692ae5b961915e038 Gerrit-Change-Number: 8879 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5929: Remove redundant explicit casts to string
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8660 ) Change subject: IMPALA-5929: Remove redundant explicit casts to string .. Patch Set 4: (10 comments) http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java File fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java: http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@31 PS4, Line 31: * "cast( to ) " > string literal (not string constant) Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@35 PS4, Line 35: * if the following is true: cast(cast( as typeOf( string literal (not string constant) Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@36 PS4, Line 36: * expr>)) as typeOf()) == > string literal (not string constant) Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@39 PS4, Line 39: * is always on the right hand side and all constant Exprs have been > (not ) Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@48 PS4, Line 48: * Few cases that are are not rewritten as the redundancy test fails: > double "are" Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@67 PS4, Line 67: !expr.getChild(0).isLiteral() && > not needed, we already require that child(0) is a CastExpr Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@69 PS4, Line 69: (op == BinaryPredicate.Operator.EQ || op == BinaryPredicate.Operator.NE); > this is the cheapest check, might want to to the front of the conjunction o Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java@75 PS4, Line 75: LiteralExpr constantExpr = (LiteralExpr) expr.getChild(1); > literalExpr Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java File fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java: http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java@619 PS4, Line 619: // Works for other string types > // Works for VARCHAR/CHAR Done http://gerrit.cloudera.org:8080/#/c/8660/4/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java@641 PS4, Line 641: > remove extra blank line Done -- To view, visit http://gerrit.cloudera.org:8080/8660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b7c6452d0693115f9b9ed9ba09f3ffe0f36b2b Gerrit-Change-Number: 8660 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Dec 2017 19:01:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5929: Remove redundant explicit casts to string
Hello Thomas Tauber-Marshall, Tim Armstrong, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8660 to look at the new patch set (#5). Change subject: IMPALA-5929: Remove redundant explicit casts to string .. IMPALA-5929: Remove redundant explicit casts to string This patch adds a query rewriter to remove redundant explicit casts to a string type (string, char, varchar) from binary predicates of the form "cast( to ) ". The cast is redundant if the predicate evaluation is the same even if the cast is removed and the constant is converted to the original type of the expression. For example: cast(int_col as string) = '123456' -> int_col = 123456 Performance: For the following query on a table having 6001215 records - select * from tpch.lineitem where cast(l_linenumber as string) = '0' +-+---++ | | Scan Time | +-+---++ | | Avg | St dev | | Without rewrite | 1s406ms | 44ms | | With rewrite| 1s099ms | 28ms | +-+---++ Testing: - Added unit tests to ExprRewriteRulesTest - Added functional test to expr.test - Current FE planner tests and BE expr-test run successfully with this change. Change-Id: I91b7c6452d0693115f9b9ed9ba09f3ffe0f36b2b --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java A fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M testdata/workloads/functional-query/queries/QueryTest/exprs.test 4 files changed, 196 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/8660/5 -- To view, visit http://gerrit.cloudera.org:8080/8660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91b7c6452d0693115f9b9ed9ba09f3ffe0f36b2b Gerrit-Change-Number: 8660 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8853 to look at the new patch set (#3). Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. As a follow-on to centralizing into one parent pom, we can now manage thirdparty dependency versions in Java a little bit more clearly. Upgrades SLF4J, commons.io: slf4j: 1.7.5 -> 1.7.25 commons.io: 2.4 -> 2.6 The SLF4J upgrade is nice to be able to run under Java9. The release notes at https://www.slf4j.org/news.html are uneventful. Commons IO 2.6 supports Java 9 and is source and binary compatible, per https://commons.apache.org/proper/commons-io/upgradeto2_6.html and https://commons.apache.org/proper/commons-io/upgradeto2_5.html. Removes the following dependencies: htrace-core hadoop-mapreduce-client-core hive-shims com.stumbleupon:async commons-dbcp jdo-api I ran "mvn dependency:analyze" and these were some (but not all) of the "Unused declared dependencies found." Spelunking in git logs, these dependencies are from 2013 and possibly from an effort to run with dependencies from the filesystem. They don't seem to be required anymore. Stops pulling in an old version of hadoop-client and kite-data-core in testdata/TableFlattener by using the same versions as the Hadoop we use. Doing so was unnecessarily causing us to download extra, old Hadoop jars, and the new Hadoop jars seem to work just as well. This is the kind of divergence that centralizing the versions into variables will help with. Creates variables for: junit.version slf4j.version hadoop.version commons-io.version httpcomponents.core.version thrift.version kite.version (controlled via $IMPALA_KITE_VERSION in impala-config.sh) Cleans up unused IMPALA_PARQUET_URL variables in impala-config.sh. We only download Parquet via Maven, rather than downloading it in the toolchain, so this variable wasn't doing anything. I ran the core tests with this change. Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 --- M bin/impala-config.sh M fe/pom.xml M impala-parent/pom.xml M testdata/TableFlattener/pom.xml M testdata/pom.xml 5 files changed, 23 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/8853/3 -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8879 Change subject: Revert "IMPALA-3887: Wait for HDFS replication in data loading" .. Revert "IMPALA-3887: Wait for HDFS replication in data loading" Using fsck on non-hdfs file systems causes errors. This reverts commit 5a7c10ec3d500c1bb582e985cb299e7dbcfe5520. Change-Id: I4318b4fcb94aa67ac55692ae5b961915e038 --- M testdata/bin/create-load-data.sh 1 file changed, 0 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/8879/1 -- To view, visit http://gerrit.cloudera.org:8080/8879 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4318b4fcb94aa67ac55692ae5b961915e038 Gerrit-Change-Number: 8879 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8853 ) Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/8853/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/8853/2/bin/impala-config.sh@142 PS2, Line 142: IMPALA_THRIFT_JAVA_VERSION Is this still being used anywhere now? http://gerrit.cloudera.org:8080/#/c/8853/2/impala-parent/pom.xml File impala-parent/pom.xml: http://gerrit.cloudera.org:8080/#/c/8853/2/impala-parent/pom.xml@57 PS2, Line 57: 0.9.0< Should this be env.IMPALA_THRIFT_VERSION? -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 19 Dec 2017 18:47:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6318: Adjustment for hanging query cancellation test
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8852 ) Change subject: IMPALA-6318: Adjustment for hanging query cancellation test .. Patch Set 5: Code-Review+1 (1 comment) I think this is good to go. http://gerrit.cloudera.org:8080/#/c/8852/5/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/8852/5/tests/shell/util.py@162 PS5, Line 162: def _start_new_shell_process(self, args=None, env=None, omit_stdout=False): It's weird that this both takes self and arguments. I would have expected __init__ to set self.args, self.env, and self.omit_stdout, and that to be used here. No need to block on this, just wanted to point it out. -- To view, visit http://gerrit.cloudera.org:8080/8852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I082c83b91b6d0c527de92c7992f0dc9d1b290433 Gerrit-Change-Number: 8852 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2017 18:30:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8853 ) Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Patch Set 2: https://jenkins.impala.io/job/gerrit-verify-dryrun-external/39/ ran successfully with this change. The difference between the first patch set (which it ran on) and the current one is an update to the commit message only. -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 19 Dec 2017 18:11:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io.
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8853 Change subject: Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. .. Remove unused deps, centralize some pom versions, upgrade SLF4J and commons-io. As a follow-on to centralizing into one parent pom, we can now manage thirdparty dependency versions in Java a little bit more clearly. Upgrades SLF4J, commons.io: slf4j: 1.7.5 -> 1.7.25 commons.io: 2.4 -> 2.6 The SLF4J upgrade is nice to be able to run under Java9. The release notes at https://www.slf4j.org/news.html are uneventful. Commons IO 2.6 supports Java 9 and is source and binary compatible, per https://commons.apache.org/proper/commons-io/upgradeto2_6.html and https://commons.apache.org/proper/commons-io/upgradeto2_5.html. Removes the following dependencies: htrace-core hadoop-mapreduce-client-core hive-shims com.stumbleupon:async commons-dbcp jdo-api I ran "mvn dependency:analyze" and these were some (but not all) of the "Unused declared dependencies found." Spelunking in git logs, these dependencies are from 2013 and possibly from an effort to run with dependencies from the filesystem. They don't seem to be required anymore. Stops pulling in an old version of hadoop-client and kite-data-core in testdata/TableFlattener by using the same versions as the Hadoop we use. Doing so was unnecessarily causing us to download extra, old Hadoop jars, and the new Hadoop jars seem to work just as well. This is the kind of divergence that centralizing the versions into variables will help with. Creates variables for: junit.version slf4j.version hadoop.version commons-io.version httpcomponents.core.version thrift.version kite.version (controlled via $IMPALA_KITE_VERSION in impala-config.sh) Cleans up unused IMPALA_PARQUET_URL variables in impala-config.sh. We only download Parquet via Maven, rather than downloading it in the toolchain, so this variable wasn't doing anything. I ran the core tests with this change. Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 --- M bin/impala-config.sh M fe/pom.xml M impala-parent/pom.xml M testdata/TableFlattener/pom.xml M testdata/pom.xml 5 files changed, 22 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/8853/2 -- To view, visit http://gerrit.cloudera.org:8080/8853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I717e0625dfe0fdbf7e9161312e9e80f405a359c5 Gerrit-Change-Number: 8853 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-2248: Make idle session timeout a query option
Hello Michael Ho, Thomas Tauber-Marshall, Laszlo Gaal, Gabor Kaszab, Attila Jeges, Tim Armstrong, Csaba Ringhofer, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8490 to look at the new patch set (#17). Change subject: IMPALA-2248: Make idle_session_timeout a query option .. IMPALA-2248: Make idle_session_timeout a query option This commit makes idle_session_timeout a query option. idle_session_timeout currently can be set as a command line option, which will be the default timeout for sessions. HS2 sessions can override it with a smaller value by setting it in the configuration overlay of HS2 OpenSession(). However, we can't override idle_session_timeout for JDBC/ODBC connections, because we cannot put this in the connection string. This commit is a workaround for this problem, it allows JDBC/ODBC connections to set the session timeout as a query option with the SET statement. After this commit, the session timeout can be overridden to any value, i.e. the command line flag idle_session_timeout doesn't limit this option anymore. I created an automated test case in JdbcTest.java based on test_hs2.py::test_concurrent_session_mixed_idle_timeout. I also extended the test_session_expiration and test_set_and_unset test suites. Change-Id: I32e2775f80da387b0df4195fe2c5435b3f8e585e --- M be/src/service/client-request-state.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/test/java/org/apache/impala/service/JdbcTest.java A fe/src/test/java/org/apache/impala/util/Metrics.java M tests/custom_cluster/test_session_expiration.py M tests/custom_cluster/test_set_and_unset.py M tests/hs2/hs2_test_suite.py 13 files changed, 380 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8490/17 -- To view, visit http://gerrit.cloudera.org:8080/8490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32e2775f80da387b0df4195fe2c5435b3f8e585e Gerrit-Change-Number: 8490 Gerrit-PatchSet: 17 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
