[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2541/ -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 07:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Trimming build-all-flag-combinations and adding minicluster profile.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10497 ) Change subject: Trimming build-all-flag-combinations and adding minicluster profile. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I258732a3d963958f76d99f9d7b51450ed67bec21 Gerrit-Change-Number: 10497 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 07:40:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 08:25:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4025: Part 1: Generalize and cleanup StmtRewriter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10495 ) Change subject: IMPALA-4025: Part 1: Generalize and cleanup StmtRewriter .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10495 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e7a6108d3d49be12ae032fdb54b5c3c23152a47 Gerrit-Change-Number: 10495 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 09:49:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4025: Part 1: Generalize and cleanup StmtRewriter
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10495 ) Change subject: IMPALA-4025: Part 1: Generalize and cleanup StmtRewriter .. IMPALA-4025: Part 1: Generalize and cleanup StmtRewriter This patch generalizes StmtRewriter, allowing it to be subclassed. The base class would traverse the stmt tree while the subclasses can install hooks to execute specific rewrite rules at certain places. Existing rewriting rules are moved into SubqueryRewriter. Change-Id: I9e7a6108d3d49be12ae032fdb54b5c3c23152a47 Reviewed-on: http://gerrit.cloudera.org:8080/10495 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java 2 files changed, 995 insertions(+), 984 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10495 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9e7a6108d3d49be12ae032fdb54b5c3c23152a47 Gerrit-Change-Number: 10495 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2546/ -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 17 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 10:00:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Hello Jim Apple, Attila Jeges, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10486 to look at the new patch set (#2). Change subject: IMPALA-7060: Restrict Impala to only support timezones that work in Hive .. IMPALA-7060: Restrict Impala to only support timezones that work in Hive IANA timezone integration (IMPALA-3307) will drop the support for many non-standard timezone aliases. As IMPALA-3307 is a very large change, its release may be delayed - meanwhile, it would be better to discourage Impala 3.x users from using timezone names that will not be supported in the future. For this reason, this change drops the support of non-standard aliases from the current boost based implementation. This is supposed to be a temporary solution till IMPALA-3307, so I have put minimal effort into it. Testing: Timestamp aliases were largely untested, so only a few changes were needed - these were copied from the review for IMPALA-3307, gerrit.cloudera.org/#/c/9986/ It may be useful to add more tests, e.g. to check that using dropped aliases return NULL and lead to warning, but I think that these kind of changes should be done first in the review of IMPALA-3307, and synced here afterwards. Change-Id: I90859398081bae4976af31b09b3121c198b6adac --- M be/src/exprs/timestamp-functions.cc M be/src/exprs/timezone_db.cc M testdata/data/timezoneverification.csv 3 files changed, 2 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/10486/2 -- To view, visit http://gerrit.cloudera.org:8080/10486 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac Gerrit-Change-Number: 10486 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Hello Jim Apple, Attila Jeges, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10486 to look at the new patch set (#3). Change subject: IMPALA-7060: Restrict Impala to only support timezones that work in Hive .. IMPALA-7060: Restrict Impala to only support timezones that work in Hive IANA timezone integration (IMPALA-3307) will drop the support for many non-standard timezone aliases. As IMPALA-3307 is a very large change, its release may be delayed - meanwhile, it would be better to discourage Impala 3.x users from using timezone names that will not be supported in the future. For this reason, this change drops the support of non-standard aliases from the current boost based implementation. This is supposed to be a temporary solution till IMPALA-3307, so I have put minimal effort into it. Testing: Timestamp aliases were largely untested, so only a few changes were needed - these were copied from the review for IMPALA-3307, gerrit.cloudera.org/#/c/9986/ It may be useful to add more tests, e.g. to check that using dropped aliases return NULL and lead to warning, but I think that these kind of changes should be done first in the review of IMPALA-3307, and synced here afterwards. Cherry-picks: not for 2.x. Change-Id: I90859398081bae4976af31b09b3121c198b6adac --- M be/src/exprs/timestamp-functions.cc M be/src/exprs/timezone_db.cc M testdata/data/timezoneverification.csv 3 files changed, 2 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/10486/3 -- To view, visit http://gerrit.cloudera.org:8080/10486 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac Gerrit-Change-Number: 10486 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/10486 )
Change subject: IMPALA-7060: Restrict Impala to only support timezones that
work in Hive
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10486/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10486/1//COMMIT_MSG@29
PS1, Line 29:
> I think you should add "Cherry-picks: not for 2.x." line.
Done
http://gerrit.cloudera.org:8080/#/c/10486/1/be/src/exprs/timezone_db.h
File be/src/exprs/timezone_db.h:
http://gerrit.cloudera.org:8080/#/c/10486/1/be/src/exprs/timezone_db.h@55
PS1, Line 55: map
> const?
This map turned out to be unnecessary.
http://gerrit.cloudera.org:8080/#/c/10486/1/be/src/exprs/timezone_db.cc
File be/src/exprs/timezone_db.cc:
http://gerrit.cloudera.org:8080/#/c/10486/1/be/src/exprs/timezone_db.cc@68
PS1, Line 68: {"AEST", "Australia/Sydney"},
: {"CDT", "America/Chicago"},
: {"CEST", "CET"},
: {"EDT", "EST5EDT"},
: {"ICT", "Asia/Ho_Chi_Minh"},
: {"KST", "Asia/Seoul"},
: {"MDT", "MST7MDT"},
: {"PHT", "Asia/Manila"},
: {"PDT", "America/Los_Angeles"}
> Are these time-zone abbreviations accepted by Hive? If not, I think we shou
Done - I have checked TimezoneDatabase::TIMEZONE_DATABASE_STR, and as all the
Java abbreviations are already there, I have removed this map.
http://gerrit.cloudera.org:8080/#/c/10486/1/testdata/data/timezoneverification.csv
File testdata/data/timezoneverification.csv:
http://gerrit.cloudera.org:8080/#/c/10486/1/testdata/data/timezoneverification.csv@716
PS1, Line 716: JST
> Please make sure hat all the Java-supported time-zone abbreviations are tes
I have checked, all of them are included here.
--
To view, visit http://gerrit.cloudera.org:8080/10486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac
Gerrit-Change-Number: 10486
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Jim Apple
Gerrit-Comment-Date: Thu, 24 May 2018 14:27:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10486 ) Change subject: IMPALA-7060: Restrict Impala to only support timezones that work in Hive .. Patch Set 1: > Is this a breaking change? We normally hold on braking changes > until we bump the major version. > > Is IMPALA-3307 going to go in on a minor release? Is it a breaking > change? Yes, this and IMPALA-3307 are breaking changes. This change should remove all timezones that won't be supported after IMPALA-3307. IMPALA-3307 will introduce some changes related to timezones that were changed during history, but I consider those changes to be fixes rather than breaking changes. -- To view, visit http://gerrit.cloudera.org:8080/10486 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac Gerrit-Change-Number: 10486 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 24 May 2018 14:31:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 3: Hit IMPALA-7067 -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 15:42:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7055: fix race with DML errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10491 ) Change subject: IMPALA-7055: fix race with DML errors .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2547/ -- To view, visit http://gerrit.cloudera.org:8080/10491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idafd0b342e77a065be7cc28fa8c8a9df445622c2 Gerrit-Change-Number: 10491 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 15:43:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7055: fix race with DML errors
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10491 ) Change subject: IMPALA-7055: fix race with DML errors .. Patch Set 3: It looks like that flake occurred on a different branch too. I'll try to merge this again since it fixes a test failure. -- To view, visit http://gerrit.cloudera.org:8080/10491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idafd0b342e77a065be7cc28fa8c8a9df445622c2 Gerrit-Change-Number: 10491 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 15:43:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 17: hmm. looks like some dependency order for generating flat buffers. -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 17 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 16:02:38 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Trimming build-all-flag-combinations and adding minicluster profile.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10497 ) Change subject: Trimming build-all-flag-combinations and adding minicluster profile. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I258732a3d963958f76d99f9d7b51450ed67bec21 Gerrit-Change-Number: 10497 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:14:11 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Trimming build-all-flag-combinations and adding minicluster profile.
Philip Zeyliger has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10497 ) Change subject: Trimming build-all-flag-combinations and adding minicluster profile. .. Trimming build-all-flag-combinations and adding minicluster profile. [For the 2.x branch cherrypick, this change removes the "profile" stuff.] build-all-flag-combinations.sh is the long pole in pre-commit testing. It takes about 2.5 hours and runs about 40 builds per the following. $ curl --silent https://jenkins.impala.io/job/all-build-options-ub1604/1707/consoleText | grep 'Building with OPTIONS' | wc -l 40 This commit changes this to run only 7 builds, exercising every path, but not every combination of paths. The paths exercised are: Options -skiptests -noclean and profile 3 Options -skiptests -noclean -so and profile 2 Options -skiptests -noclean -release and profile 3 Options -skiptests -noclean -release -so -ninja and profile 3 Options -skiptests -noclean -asan and profile 3 Options -skiptests -noclean -ubsan -so -ninja and profile 2 Options -skiptests -noclean -tsan and profile 3 I've also added explicitly building with both minicluster profiles and a trivial dryrun option to print the above. The options are simply hard-coded rather than being clever and generating them. We decided this was easier to deal with. Change-Id: I258732a3d963958f76d99f9d7b51450ed67bec21 Reviewed-on: http://gerrit.cloudera.org:8080/10489 Reviewed-by: Joe McDonnell Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10497 --- M bin/jenkins/build-all-flag-combinations.sh 1 file changed, 41 insertions(+), 30 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I258732a3d963958f76d99f9d7b51450ed67bec21 Gerrit-Change-Number: 10497 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. Patch Set 1: Code-Review+2 Seems strictly better. It's a parallel test, so I guess we can't wait for the query to finish before exiting the test. -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:36:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. Patch Set 11: Code-Review+2 carry -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:36:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10486 ) Change subject: IMPALA-7060: Restrict Impala to only support timezones that work in Hive .. Patch Set 3: > > Is this a breaking change? We normally hold on braking changes > > until we bump the major version. > > > > Is IMPALA-3307 going to go in on a minor release? Is it a > breaking > > change? > > Yes, this and IMPALA-3307 are breaking changes. This change should > remove all timezones that won't be supported after IMPALA-3307. > IMPALA-3307 will introduce some changes related to timezones that > were changed during history, but I consider those changes to be > fixes rather than breaking changes. Thanks for the clarification. Let's talk about this on dev@, which has more visibility, since intentionally introducing a breaking change between major versinos is usually something Impala has avoided. -- To view, visit http://gerrit.cloudera.org:8080/10486 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac Gerrit-Change-Number: 10486 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 24 May 2018 16:39:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. Patch Set 11: The crash this hit is IMPALA-7069. I can reproduce it without this change, so I'm going to go ahead with merging. -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:42:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:42:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2548/ -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:42:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5392: Added all stack frames to ThreadInfo summary.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10145 ) Change subject: IMPALA-5392: Added all stack frames to ThreadInfo summary. .. Patch Set 10: > Hi Jim, > > Please ignore the last comment. I understand your concern. I > checked the current implementation, the same issue holds there as > well. HTML DOES NOT respect "newlines" and "tabs". > > Another point, as per my understanding, this API MUST NOT worry > about how it is rendered in HTML. The responsibility of rendering > it properly must lie with a frontend component. That is, we SHOULD > NOT add HTML line breaks in this API's return string. > > Please do let me know how should I address your concern. So it seems like we are embedding this text in HTML, but it's already PREformatted. What are your thoughts on how to handle that PREformatting? -- To view, visit http://gerrit.cloudera.org:8080/10145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 Gerrit-Change-Number: 10145 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Sharma Gerrit-Reviewer: Abhishek Sharma Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Charles Agnello Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 16:47:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query.
Rahul Shivu Mahadev has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10490 Change subject: IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query. .. IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query. Added code to track the slowest RPC to BackendState and identifying that node as the slowest node. Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc --- M be/src/runtime/coordinator.cc 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/10490/1 -- To view, visit http://gerrit.cloudera.org:8080/10490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc Gerrit-Change-Number: 10490 Gerrit-PatchSet: 1 Gerrit-Owner: Rahul Shivu Mahadev
[Impala-ASF-CR] IMPALA-6813: Hedged reads metrics broken when scanning non-HDFS based table
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9966 ) Change subject: IMPALA-6813: Hedged reads metrics broken when scanning non-HDFS based table .. Patch Set 4: > > preads add some overhead to HDFS, since it sends out redundant > read > > requests to replicas. Having it on by default might cause some > > confusion while running multiple heavy concurrent workloads. > > You're talking about hedged reads, not preads. We can use pread > with hedge reads on or off. So that can't be the blocker. You're right, however the knob to turn on hedged reads is a cluster wide setting and not an Impala only setting. So if any other component in the ecosystem (Hive, etc.) requires hedged reads turned on, and we (Impala) have preads() enabled by default, we'll end up doing hedged reads every time too. IIRC, that was the initial reasoning behind controlling preads()with a flag. -- To view, visit http://gerrit.cloudera.org:8080/9966 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48fe80dfd9a1ed68a8f2b7038e5f42b5a3df3baa Gerrit-Change-Number: 9966 Gerrit-PatchSet: 4 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 17:07:56 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Sentry is required for Impala to enable delegation
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10451 ) Change subject: [DOCS] Sentry is required for Impala to enable delegation .. Patch Set 2: Code-Review+2 Thanks for sharing the info. -- To view, visit http://gerrit.cloudera.org:8080/10451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I002d3d33eee6a9b9336f21c81a4de75ed3bd5efb Gerrit-Change-Number: 10451 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 17:09:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. Patch Set 1: Well, we could make the test wait until we know the sleep has finished, but I wanted to avoid a 60s+ test. -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 17:21:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. Patch Set 2: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 17:21:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7060: Restrict Impala to only support timezones that work in Hive
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/10486 ) Change subject: IMPALA-7060: Restrict Impala to only support timezones that work in Hive .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10486 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90859398081bae4976af31b09b3121c198b6adac Gerrit-Change-Number: 10486 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 24 May 2018 17:33:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 17: Odd that this happens with gvo, but not for internal builds. -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 17 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 17:40:13 + Gerrit-HasComments: No
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh@97 PS1, Line 97: if (( BUILD_HISTORICAL )) ; then > I think it might not be necessary to build old versions since we've switche Just following the pattern of BUILD_HISTORICAL for other components here. Should we remove all of them in one shot ? -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 17:50:31 + Gerrit-HasComments: Yes
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 1: Should we run benchmarks to see if upgrading this causes any regressions? -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 17:52:08 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Sentry is required for Impala to enable delegation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10451 ) Change subject: [DOCS] Sentry is required for Impala to enable delegation .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/295/ -- To view, visit http://gerrit.cloudera.org:8080/10451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I002d3d33eee6a9b9336f21c81a4de75ed3bd5efb Gerrit-Change-Number: 10451 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 18:02:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7061: Rework HBase splitting and assignment
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10447 ) Change subject: IMPALA-7061: Rework HBase splitting and assignment .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d639128a856254a6ccb93d6750f531974b5f897 Gerrit-Change-Number: 10447 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 24 May 2018 18:12:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not requirerd in comments
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/10474 ) Change subject: IMPALA-2751: Matching quotes are not requirerd in comments .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic899fdddc182947f73101ddbc2e3c8caf97d9085 Gerrit-Change-Number: 10474 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 18:11:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10503 Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. IMPALA-6812: Fix flaky Kudu scan tests Many of our Kudu related tests have been flaky with the symptom that scans appear to not return rows that were just inserted. This occurs because our default Kudu scan level of READ_LATEST doesn't make any consistency guarantees. This patch bumps the default scan level for running tests to READ_AT_SNAPSHOT, which should give us more consistent test results. Testing: - Passed a full exhaustive run. Does not appear to increase time to run by any significant amount. Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c --- M bin/run-all-tests.sh 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/10503/1 -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-7061: Rework HBase splitting and assignment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10447 ) Change subject: IMPALA-7061: Rework HBase splitting and assignment .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2550/ -- To view, visit http://gerrit.cloudera.org:8080/10447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d639128a856254a6ccb93d6750f531974b5f897 Gerrit-Change-Number: 10447 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 24 May 2018 18:15:08 + Gerrit-HasComments: No
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh@97 PS1, Line 97: if (( BUILD_HISTORICAL )) ; then > Just following the pattern of BUILD_HISTORICAL for other components here. S Let's do the cleanup in a separate change, and maybe once we retire the 2.x branch. Until then we might need cherry picks in the toolchain repo and those will see more conflicts if we remove all the old versions. We don't build with BUILD_HISTORICAL when publishing bits, either, so there's no overhead in keeping them. -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 18:22:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Sentry is required for Impala to enable delegation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10451 ) Change subject: [DOCS] Sentry is required for Impala to enable delegation .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I002d3d33eee6a9b9336f21c81a4de75ed3bd5efb Gerrit-Change-Number: 10451 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 18:26:26 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Sentry is required for Impala to enable delegation
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10451 ) Change subject: [DOCS] Sentry is required for Impala to enable delegation .. [DOCS] Sentry is required for Impala to enable delegation Change-Id: I002d3d33eee6a9b9336f21c81a4de75ed3bd5efb Reviewed-on: http://gerrit.cloudera.org:8080/10451 Reviewed-by: Sailesh Mukil Tested-by: Impala Public Jenkins --- M docs/topics/impala_delegation.xml 1 file changed, 42 insertions(+), 51 deletions(-) Approvals: Sailesh Mukil: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I002d3d33eee6a9b9336f21c81a4de75ed3bd5efb Gerrit-Change-Number: 10451 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-4025: Part 2: Add percentile disc aggregation function
Tianyi Wang has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/9777 ) Change subject: IMPALA-4025: Part 2: Add percentile_disc aggregation function .. IMPALA-4025: Part 2: Add percentile_disc aggregation function This patch adds aggregation function percentile_disc. The implementation is rewriting it into an inline view. The inline view computes the row number on the ordering expr using analytic functions. The parent query then picks the desired row using aggregation. An Example of such rewrite is in StmtRewriter.java. The behavior of this function is mostly the same as in PostgreSQL. The handling of percentile expr not in [0, 1] is different: PostgreSQL throws an error and impala returns NULL. Some FE and EE tests are added. EE tests not related to the above difference are verified against PostgreSQL. Change-Id: Iacef7b3fcd74c4c73d88400ce27307c3baa0121e --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java A fe/src/main/java/org/apache/impala/analysis/PercentileAggExpr.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/catalog/AggregateFunction.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-planner/queries/PlannerTest/aggregation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/aggregation.test 20 files changed, 1,219 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/9777/10 -- To view, visit http://gerrit.cloudera.org:8080/9777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacef7b3fcd74c4c73d88400ce27307c3baa0121e Gerrit-Change-Number: 9777 Gerrit-PatchSet: 10 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-4025: Part 3: Add percentile cont & median aggregation functions
Tianyi Wang has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/9778 ) Change subject: IMPALA-4025: Part 3: Add percentile_cont & median aggregation functions .. IMPALA-4025: Part 3: Add percentile_cont & median aggregation functions percentile_cont is implemented in the similar way as percentile_disc, except for using a BE custom aggregation function for interpolating the final result. median is rewritten into percentile_cont(0.5). Some EE tests are added. Tests not related to error handling are verified against PostgreSQL. Change-Id: I2cc184682bb1bf4a5011b69a89e9ae253f3fd88d --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/PercentileAggExpr.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-query/queries/QueryTest/aggregation.test 11 files changed, 242 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/9778/8 -- To view, visit http://gerrit.cloudera.org:8080/9778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2cc184682bb1bf4a5011b69a89e9ae253f3fd88d Gerrit-Change-Number: 9778 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 7: ping -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 24 May 2018 18:29:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not requirerd in comments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10474 ) Change subject: IMPALA-2751: Matching quotes are not requirerd in comments .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2551/ -- To view, visit http://gerrit.cloudera.org:8080/10474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic899fdddc182947f73101ddbc2e3c8caf97d9085 Gerrit-Change-Number: 10474 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 18:32:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 24 May 2018 18:35:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 1: It looks like there's some history here, but can we make READ_AT_SNAPSHOT the default? It would be best to test the default mode as much as possible. -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 18:40:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7055: fix race with DML errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10491 ) Change subject: IMPALA-7055: fix race with DML errors .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idafd0b342e77a065be7cc28fa8c8a9df445622c2 Gerrit-Change-Number: 10491 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 19:09:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7055: fix race with DML errors
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10491 ) Change subject: IMPALA-7055: fix race with DML errors .. IMPALA-7055: fix race with DML errors Error statuses could be lost because backend_exec_complete_barrier_ went to 0 before the query was transitioned to an error state. Reordering the UpdateExecState() and backend_exec_complete_barrier_ calls prevents this race. Change-Id: Idafd0b342e77a065be7cc28fa8c8a9df445622c2 Reviewed-on: http://gerrit.cloudera.org:8080/10491 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h 2 files changed, 19 insertions(+), 8 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idafd0b342e77a065be7cc28fa8c8a9df445622c2 Gerrit-Change-Number: 10491 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. IMPALA-7067: deflake test_cancellation Tweak the query so that it still runs for a long time but can cancel the fragment quicker instead of being stuck in a long sleep() call. Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Reviewed-on: http://gerrit.cloudera.org:8080/10499 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M tests/shell/test_shell_commandline.py 1 file changed, 6 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7067: deflake test cancellation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10499 ) Change subject: IMPALA-7067: deflake test_cancellation .. Patch Set 3: Verified+1 Code-Review+2 Rebase and carry verification/+2 -- To view, visit http://gerrit.cloudera.org:8080/10499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90d4f5c277f7b0d5561637944b454f7a44c76e Gerrit-Change-Number: 10499 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 19:27:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query.
Hello Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10490 to look at the new patch set (#2). Change subject: IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query. .. IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query. Added code to track the slowest RPC to BackendState and identifying that node as the slowest node. Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc --- M be/src/runtime/coordinator.cc 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/10490/2 -- To view, visit http://gerrit.cloudera.org:8080/10490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc Gerrit-Change-Number: 10490 Gerrit-PatchSet: 2 Gerrit-Owner: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 20:10:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 1: clean up DiskIoMgr
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10245 ) Change subject: IMPALA-6953: part 1: clean up DiskIoMgr .. IMPALA-6953: part 1: clean up DiskIoMgr There should be no behavioural changes as a result of this refactoring. Make DiskQueue an encapsulated class. Remove friend classes where possible, either by using public methods or moving code between classes. Move method into protected in some cases. Split GetNextRequestRange() into two methods that operate on DiskQueue and RequestContext state. The methods belong to the respective classes. Reduce transitive #include dependencies to hopefully help with build time. Testing: Ran core tests. Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Reviewed-on: http://gerrit.cloudera.org:8080/10245 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/exec/blocking-join-node.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-scanner.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-table-sink.cc M be/src/exec/parquet-column-readers.cc M be/src/runtime/io/disk-io-mgr-internal.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/error-converter.cc M be/src/runtime/io/local-file-system.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/row-batch.h 21 files changed, 590 insertions(+), 534 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10245 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1 Gerrit-Change-Number: 10245 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Dan Hecht has uploaded this change for review. (
http://gerrit.cloudera.org:8080/10504
Change subject: IMPALA-7068: fix test_unsupported_text_compression table
location
..
IMPALA-7068: fix test_unsupported_text_compression table location
WAREHOUSE is defined as get_fs_path('/test-warehouse') so we
shouldn't run that through get_fs_path again. Instead, I copied
the TBL_LOCATION format from line 59.
Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960
---
M tests/metadata/test_partition_metadata.py
1 file changed, 1 insertion(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/10504/1
--
To view, visit http://gerrit.cloudera.org:8080/10504
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960
Gerrit-Change-Number: 10504
Gerrit-PatchSet: 1
Gerrit-Owner: Dan Hecht
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10504 ) Change subject: IMPALA-7068: fix test_unsupported_text_compression table location .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960 Gerrit-Change-Number: 10504 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 20:30:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10504 ) Change subject: IMPALA-7068: fix test_unsupported_text_compression table location .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10504/1/tests/metadata/test_partition_metadata.py File tests/metadata/test_partition_metadata.py: http://gerrit.cloudera.org:8080/#/c/10504/1/tests/metadata/test_partition_metadata.py@168 PS1, Line 168: TBL_LOCATION = '%s/%s.db/%s' % (WAREHOUSE, unique_database, TBL_NAME) Is there a way we could detect this mistake or avoid it? E.g. make get_fs_path() idempotent? Seems like we'll keep hitting errors of this nature until we can detect it in precommit. -- To view, visit http://gerrit.cloudera.org:8080/10504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960 Gerrit-Change-Number: 10504 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 20:30:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10504 ) Change subject: IMPALA-7068: fix test_unsupported_text_compression table location .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2553/ -- To view, visit http://gerrit.cloudera.org:8080/10504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960 Gerrit-Change-Number: 10504 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 20:37:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10504 ) Change subject: IMPALA-7068: fix test_unsupported_text_compression table location .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10504/1/tests/metadata/test_partition_metadata.py File tests/metadata/test_partition_metadata.py: http://gerrit.cloudera.org:8080/#/c/10504/1/tests/metadata/test_partition_metadata.py@168 PS1, Line 168: TBL_LOCATION = '%s/%s.db/%s' % (WAREHOUSE, unique_database, TBL_NAME) > Is there a way we could detect this mistake or avoid it? E.g. make get_fs_p I was going to add an assert, but making it idempotent is even better. In any case, will do in a separate commit after builds are green again. -- To view, visit http://gerrit.cloudera.org:8080/10504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960 Gerrit-Change-Number: 10504 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 20:38:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10490 ) Change subject: IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per query. .. Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-4970: Record identity of largest latency ExecQueryFInstances() RPC per : query. nit: Commit message titles have to be in one line, so if it exceeds the char limit, it's okay, but only the titles. http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@10 PS2, Line 10: Added code to track the slowest RPC to BackendState and identifying that node : as the slowest node. nit: Tracked the slowest ExecQueryFInstances() RPC to executors and added the identity of that executor to the RuntimeProfile. http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@12 PS2, Line 12: Can you add a note saying how you manually tested it? Add "Testing: ..." http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@357 PS2, Line 357: INT_MIN This can be 0, since we never expect a negative latency. http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@357 PS2, Line 357: int max_latency = INT_MIN; Move this into the for loop just above L365 to limit the scope of its lifetime. http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@367 PS2, Line 367: max_latency_host = backend_state->impalad_address().hostname + ":" + std::to_string(backend_state->impalad_address().port); nit: long line. Keep lines to 90 chars, and make it a multi-line statement if it doesn't fit. http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@367 PS2, Line 367: backend_state->impalad_address().hostname + ":" + std::to_string(backend_state->impalad_address().port); use the TNetworkAddressToString() utility here to print the impalad_address() Eg: Look at L686 http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@373 PS2, Line 373: query_profile_->AddInfoString( : "Slowest Host", max_latency_host); This can fit on one line. http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@374 PS2, Line 374: Slowest Host The previous line says "Backend startup latencies". So lets keep the terms consistent. Maybe change this to "Slowest backend to startup" or something similar. -- To view, visit http://gerrit.cloudera.org:8080/10490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc Gerrit-Change-Number: 10490 Gerrit-PatchSet: 2 Gerrit-Owner: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 21:03:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Hello Lars Volker, Tianyi Wang, Dimitris Tsirogiannis, Alex Behm, Mostafa Mokhtar, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8523 to look at the new patch set (#18). Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. IMPALA-5931: Generates scan ranges in planner for s3/adls Currently, for filesystems that do not include physical block information (e.g., block replica locations, caching), synthetic blocks are generated and stored in the catalog when metadata is loaded. Example file systems for which this is done includes S3, ADLS, and local fs. This change avoids generating these blocks when metadata is loaded. Instead, scan ranges are directly generated from such files by the backend coordinator. Previously, all scan ranges were produced by the planner in HDFSScanNode in the frontend. Now, those files without block information are sent to the coordinator represented by a split specification that determines how the coordinator will create scan ranges to send to executors. This change reduces the space needed in the catalog and reduces the scan range data structures that are passed from the frontend to the backend when planning and coordinating a query. In addition a bug is avoided where non-splittable files were being split anyways to support the query parameter that places a limit on scan ranges. Testing: - added backend scheduler tests - mixed-filesystems test covers tables/queries with multiple fs's. - local fs tests cover the code paths in this change - all core tests pass when configured with s3 - manually tried larger local filesystem tables (tpch) with multiple partitions and observed the same scan ranges. - TODO: adls testing Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 --- M CMakeLists.txt M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/util/CMakeLists.txt A be/src/util/flat_buffer.cc A be/src/util/flat_buffer.h M common/thrift/Frontend.thrift M common/thrift/PlanNodes.thrift M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 22 files changed, 677 insertions(+), 247 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/8523/18 -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 18 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 18: latest change explicitly adds a dep for flat buffers for the backend (in the same place that the be depends on thrift and protos). afaict, this should have been in there for this change-- not sure why some of the builds didn't need it. -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 18 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 21:13:36 + Gerrit-HasComments: No
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch File source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch: http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch@37 PS1, Line 37: ( nit: space. What's the complaint from ASAN ? -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 24 May 2018 21:25:12 + Gerrit-HasComments: Yes
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10496/1/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/10496/1/buildall.sh@a272 PS1, Line 272: : : Do you want to keep this for now in this patch to be consistent ? -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 24 May 2018 21:26:13 + Gerrit-HasComments: Yes
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/10496/1/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/10496/1/buildall.sh@a272 PS1, Line 272: : : > Do you want to keep this for now in this patch to be consistent ? I considered it, but we don't even use it in Impala's build itself, so even if we build it, there's no way to consume it. As far as I can tell, the dependencies (here: glog) will still only build against the latest version. http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch File source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch: http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch@37 PS1, Line 37: ( > nit: space. I imported this patch from Kudu. The commit message (above) has the error and how the workaround was borrowed from the Abseil lib. I'd prefer to keep the patch unchanged, including the whitespace mistakes. Do you feel strongly about fixing it? -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 24 May 2018 21:32:23 + Gerrit-HasComments: Yes
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch File source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch: http://gerrit.cloudera.org:8080/#/c/10496/1/source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch@37 PS1, Line 37: ( > I imported this patch from Kudu. The commit message (above) has the error a Missed the message above. Make sense. -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 24 May 2018 21:39:57 + Gerrit-HasComments: Yes
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Lars Volker has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Bump libunwind version to 1.3-rc1 This changes bumps libunwind to version 1.3-rc1, the same version as in Kudu. It also adds two patches to improve its interaction with asan and tsan. The patch we had for 1.1 is not needed anymore. I tested this on my local dev environment with a prototype that rebases our KRPC dependencies. Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Reviewed-on: http://gerrit.cloudera.org:8080/10496 Reviewed-by: Michael Ho Tested-by: Lars Volker --- M buildall.sh M source/libunwind/build.sh D source/libunwind/libunwind-1.1-patches/0001-Build-libunwind-1.1-on-ppc64le.patch A source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch A source/libunwind/libunwind-1.3-rc1-patches/0002-libunwind-trace-cache-destructor.patch 5 files changed, 78 insertions(+), 11 deletions(-) Approvals: Michael Ho: Looks good to me, approved Lars Volker: Verified -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[native-toolchain-CR] Bump libunwind version to 1.3-rc1
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10496 ) Change subject: Bump libunwind version to 1.3-rc1 .. Patch Set 1: Verified+1 Built the toolchain locally -- To view, visit http://gerrit.cloudera.org:8080/10496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34b78fd3272c12eb529ff06c7f9f1569d5def98 Gerrit-Change-Number: 10496 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 24 May 2018 21:43:39 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Complex types in DDL not supported for text format files
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10508 Change subject: [DOCS] Complex types in DDL not supported for text format files .. [DOCS] Complex types in DDL not supported for text format files Change-Id: Icc67c9d74de7e952d13b7ecc511ad263b3915272 --- M docs/topics/impala_complex_types.xml 1 file changed, 33 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/10508/1 -- To view, visit http://gerrit.cloudera.org:8080/10508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icc67c9d74de7e952d13b7ecc511ad263b3915272 Gerrit-Change-Number: 10508 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-4970: Record identity of largest latency ExecQueryFInstances()
Hello Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10490 to look at the new patch set (#3). Change subject: IMPALA-4970: Record identity of largest latency ExecQueryFInstances() .. IMPALA-4970: Record identity of largest latency ExecQueryFInstances() Tracked the slowest ExecQueryFInstances() RPC to executors and added the identity of that executor to the RuntimeProfile. Testing: Manually tested to verify working. Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc --- M be/src/runtime/coordinator.cc 1 file changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/10490/3 -- To view, visit http://gerrit.cloudera.org:8080/10490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc Gerrit-Change-Number: 10490 Gerrit-PatchSet: 3 Gerrit-Owner: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not requirerd in comments
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10474 ) Change subject: IMPALA-2751: Matching quotes are not requirerd in comments .. IMPALA-2751: Matching quotes are not requirerd in comments This patch fixes the issue where non-matching quotes inside comments will cause the shell to not terminate. The fix is to strip any SQL comments before sending to shlex since shlex does not understand SQL comments and will raise an exception when it sees unmatched quotes regardless whether the quotes are in the comments or not. Testing: - Added new shell tests - Ran all end-to-end shell tests Change-Id: Ic899fdddc182947f73101ddbc2e3c8caf97d9085 Reviewed-on: http://gerrit.cloudera.org:8080/10474 Reviewed-by: Michael Brown Tested-by: Impala Public Jenkins --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 25 insertions(+), 5 deletions(-) Approvals: Michael Brown: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic899fdddc182947f73101ddbc2e3c8caf97d9085 Gerrit-Change-Number: 10474 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not requirerd in comments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10474 ) Change subject: IMPALA-2751: Matching quotes are not requirerd in comments .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic899fdddc182947f73101ddbc2e3c8caf97d9085 Gerrit-Change-Number: 10474 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 21:59:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4970: Record identity of largest latency ExecQueryFInstances()
Rahul Shivu Mahadev has posted comments on this change. (
http://gerrit.cloudera.org:8080/10490 )
Change subject: IMPALA-4970: Record identity of largest latency
ExecQueryFInstances()
..
Patch Set 3:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-4970: Record identity of largest latency
ExecQueryFInstances()
:
> nit: Commit message titles have to be in one line, so if it exceeds the cha
Done
http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@10
PS2, Line 10: added the identity of that executor to the RuntimeProfile.
:
> nit: Tracked the slowest ExecQueryFInstances() RPC to executors and added t
Done
http://gerrit.cloudera.org:8080/#/c/10490/2//COMMIT_MSG@12
PS2, Line 12: Testing: Manually tested to verify working.
> Can you add a note saying how you manually tested it?
Done
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@357
PS2, Line 357: int max_latency = 0;
> Move this into the for loop just above L365 to limit the scope of its lifet
I won't be able to do that as we need to iterate and find the max latency
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@357
PS2, Line 357: 0;
> This can be 0, since we never expect a negative latency.
Done
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@367
PS2, Line 367: TNetworkAddressToString(backend_state->impalad_address());
> use the TNetworkAddressToString() utility here to print the impalad_address
Done
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@367
PS2, Line 367: max_latency_host =
TNetworkAddressToString(backend_state->impalad_address());
> nit: long line. Keep lines to 90 chars, and make it a multi-line statement
Done
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@374
PS2, Line 374: n UpdateExec
> The previous line says "Backend startup latencies". So lets keep the terms
Done
http://gerrit.cloudera.org:8080/#/c/10490/2/be/src/runtime/coordinator.cc@373
PS2, Line 373: query_profile_->AddInfoString("Slowest Backend to startup",
max_latency_host);
: return UpdateExecState(status, nullptr
> This can fit on one line.
Done
--
To view, visit http://gerrit.cloudera.org:8080/10490
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf9ce0d087c91c2e92ba5be08c96cc5364ae38dc
Gerrit-Change-Number: 10490
Gerrit-PatchSet: 3
Gerrit-Owner: Rahul Shivu Mahadev
Gerrit-Reviewer: Rahul Shivu Mahadev
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Thu, 24 May 2018 21:59:28 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/10480/1/buildall.sh@97 PS1, Line 97: if (( BUILD_HISTORICAL )) ; then > I think it might not be necessary to build old versions since we've switche Done -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 22:36:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 14: (2 comments) > Do we have sufficient test coverage for the various webserver/RPC > handlers that can happen concurrently with admission control > (previously, less cases were possible since the query handle wasn't > returned until after admission control)? AFAIK we currently dont have any test coverage for webserver calls apart from those in test_redaction. We should probably add that coverage in a separate patch. However, I have added tests for the getState and getExecSummary client RPCs. Also added a session expiry test to check if queries in queue get cancelled. http://gerrit.cloudera.org:8080/#/c/10060/12/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/10060/12/be/src/service/client-request-state.cc@975 PS12, Line 975: } > Maybe we could add something short explaining this condition. E.g. "Ensure Done http://gerrit.cloudera.org:8080/#/c/10060/12/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/10060/12/be/src/service/impala-http-handler.cc@776 PS12, Line 776: request_state->operation_state() != TOperationState::ERROR_STATE > @Tim You are right, anything not used/set is just returned as empty which i Done -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 14 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 22:37:30 + Gerrit-HasComments: Yes
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Hello Lars Volker, Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10480 to look at the new patch set (#2). Change subject: Upgrade Protobuf to 3.5.1 .. Upgrade Protobuf to 3.5.1 Protobuf 3.0.0+ has support for both v2 and v3 of protobuf syntax. Apparently, the current version of protobuf in toolchain (2.6.1) doesn't seem to recognize the map type documented in protobuf v2 syntax. Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c --- M buildall.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/80/10480/2 -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Hello Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10060 to look at the new patch set (#15). Change subject: IMPALA-5216: Make admission control queuing async .. IMPALA-5216: Make admission control queuing async Implement asynchronous admission control queuing. This is achieved by running the admission control code-path in a separate thread. Major changes include: propagating cancellation to the admission control thread and dequeuing thread, and adding a new Query Operation State called "PENDING" that represents the state between completion of planning and starting of query execution. Testing: - Added a deterministic end to end test and a session expiry test. - Ran multiple stress tests successfully with a cancellation probability of 60% and with different values for the following parameters: max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a valid state afterwards (no orphan fragments or wrong metrics). - Ran all exhaustive tests and ASAN core tests successfully. - Ran data load successfully. Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf --- M be/src/common/atomic.h M be/src/common/logging.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/promise-test.cc M be/src/util/promise.h M common/thrift/ImpalaService.thrift M tests/authorization/test_authorization.py M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_krpc_mem_usage.py M tests/custom_cluster/test_session_expiration.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M www/query_backends.tmpl 28 files changed, 665 insertions(+), 220 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/10060/15 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 22:41:38 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update composite or nested types in UDF
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10509 Change subject: [DOCS] Update composite or nested types in UDF .. [DOCS] Update composite or nested types in UDF Change-Id: Ieda055f22e3d1e3507aacc8ba1e86e56d6f2e142 --- M docs/topics/impala_udf.xml 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10509/1 -- To view, visit http://gerrit.cloudera.org:8080/10509 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieda055f22e3d1e3507aacc8ba1e86e56d6f2e142 Gerrit-Change-Number: 10509 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 18 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 24 May 2018 23:09:02 + Gerrit-HasComments: No
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 23:26:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5522: Add support for authorized proxy groups
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10510 Change subject: IMPALA-5522: Add support for authorized proxy groups .. IMPALA-5522: Add support for authorized proxy groups The patch adds support for mapping of users to a list of proxy groups. The following flags are added in impalad: - authorized_proxy_group_config - authorized_proxy_group_config_delimiter Example: --authorized_proxy_group_config=hue=group1,group2;user1=* Testing: - Added a new test in test_authorization.py - Ran all end-to-end test_authorization.py Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/service/JniFrontend.java M tests/authorization/test_authorization.py 8 files changed, 166 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/10510/2 -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 23:31:30 + Gerrit-HasComments: No
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Patch Set 2: Verified the change by building on various platforms and tried the new bits on Ubuntu on 14.04. Will do more testing before upgrading Impala to the new version. -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 24 May 2018 23:32:28 + Gerrit-HasComments: No
[native-toolchain-CR] Upgrade Protobuf to 3.5.1
Michael Ho has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10480 ) Change subject: Upgrade Protobuf to 3.5.1 .. Upgrade Protobuf to 3.5.1 Protobuf 3.0.0+ has support for both v2 and v3 of protobuf syntax. Apparently, the current version of protobuf in toolchain (2.6.1) doesn't seem to recognize the map type documented in protobuf v2 syntax. Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Reviewed-on: http://gerrit.cloudera.org:8080/10480 Reviewed-by: Lars Volker Reviewed-by: Sailesh Mukil Tested-by: Michael Ho --- M buildall.sh 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Lars Volker: Looks good to me, but someone else must approve Sailesh Mukil: Looks good to me, approved Michael Ho: Verified -- To view, visit http://gerrit.cloudera.org:8080/10480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4e4e9dc301ff16f9aecc32d30c22b523dda3033c Gerrit-Change-Number: 10480 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Hello Thomas Marshall, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10483 to look at the new patch set (#3). Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. IMPALA-7044: Prevent overflow when computing Parquet block size When writing Parquet files we compute a minimum block size based on the number of columns in the target table: 3 * page_size * num_cols For tables with a large number of columns (> ~10k), this value will get larger than 2GB. When we pass it to hdfsOpenFile() in HdfsTableSink::CreateNewTmpFile() it gets cast to a signed int32 and can overflow. To fix this we return an error if we detect that the minimum block size exceed 2GB. This change adds a test using CTAS into a table with 12k columns, making sure that Impala returns the correct error. Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M tests/query_test/test_insert_parquet.py 4 files changed, 36 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/10483/3 -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10511 Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. IMPALA-5740: [DOCS] Correct the max length of STRING Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 --- M docs/topics/impala_string.xml 1 file changed, 100 insertions(+), 118 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/10511/1 -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Hello Thomas Marshall, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10483 to look at the new patch set (#4). Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. IMPALA-7044: Prevent overflow when computing Parquet block size When writing Parquet files we compute a minimum block size based on the number of columns in the target table: 3 * page_size * num_cols For tables with a large number of columns (> ~10k), this value will get larger than 2GB. When we pass it to hdfsOpenFile() in HdfsTableSink::CreateNewTmpFile() it gets cast to a signed int32 and can overflow. To fix this we return an error if we detect that the minimum block size exceed 2GB. This change adds a test using CTAS into a table with 12k columns, making sure that Impala returns the correct error. Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M tests/query_test/test_insert_parquet.py 4 files changed, 38 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/10483/4 -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. Patch Set 2: (1 comment) Thanks for the review. Please see PS4. http://gerrit.cloudera.org:8080/#/c/10483/2/be/src/exec/hdfs-parquet-table-writer.cc File be/src/exec/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/10483/2/be/src/exec/hdfs-parquet-table-writer.cc@789 PS2, Line 789: stringstream ss; : ss << "Minimum required block size is too large, try reducing the number of columns " : "in the target table."; > why use stringstream if you're not going to construct a string? Done -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 24 May 2018 23:44:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10511 to look at the new patch set (#2). Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. IMPALA-5740: [DOCS] Correct the max length of STRING Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 --- M docs/topics/impala_string.xml 1 file changed, 142 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/10511/2 -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10511 ) Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. Patch Set 2: Only Line #63 - #133 were actually changed and need to be reviewed. -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 May 2018 23:46:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/10504 )
Change subject: IMPALA-7068: fix test_unsupported_text_compression table
location
..
IMPALA-7068: fix test_unsupported_text_compression table location
WAREHOUSE is defined as get_fs_path('/test-warehouse') so we
shouldn't run that through get_fs_path again. Instead, I copied
the TBL_LOCATION format from line 59.
Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960
Reviewed-on: http://gerrit.cloudera.org:8080/10504
Reviewed-by: Tim Armstrong
Tested-by: Impala Public Jenkins
---
M tests/metadata/test_partition_metadata.py
1 file changed, 1 insertion(+), 2 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/10504
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960
Gerrit-Change-Number: 10504
Gerrit-PatchSet: 2
Gerrit-Owner: Dan Hecht
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7068: fix test unsupported text compression table location
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10504 ) Change subject: IMPALA-7068: fix test_unsupported_text_compression table location .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73f21866d5c2bd7d3f4a5110d4d088a718b56960 Gerrit-Change-Number: 10504 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 25 May 2018 00:03:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor tje INVALIDATE METADATA and REFRESH docs
Hello Balazs Jeszenszky, Alex Behm, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10339 to look at the new patch set (#3). Change subject: IMPALA-6987: [DOCS] Refactor tje INVALIDATE METADATA and REFRESH docs .. IMPALA-6987: [DOCS] Refactor tje INVALIDATE METADATA and REFRESH docs Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 --- M docs/shared/impala_common.xml M docs/topics/impala_invalidate_metadata.xml M docs/topics/impala_refresh.xml 3 files changed, 269 insertions(+), 415 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10339/3 -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7061: Rework HBase splitting and assignment
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10447 ) Change subject: IMPALA-7061: Rework HBase splitting and assignment .. IMPALA-7061: Rework HBase splitting and assignment Some frontend PlannerTests rely on HBase tables being arranged in a deterministic way. Specifically, the HBase tables need to be split with specific region boundaries and those regions need to be assigned to specific HBase region servers. Currently, the tables are created without splits and testdata/bin/split-hbase.sh runs Java code in HBaseTestDataRegionAssignment to split and assign the tables. This runs during dataload via testdata/bin/create-load-data.sh and during tests with bin/run-all-tests.sh. There are problems with both parts of this process. The table splitting is flaky. Since significant time can pass between the assignments and the tests, rebalancing means the assignments are not always stable. This changes the process so that the HBase tables are created with the splits already specified via the HBase shell. The splits remain stable over time. PlannerTestBase runs the assignment code in HBaseTestDataRegionAssignment at the start of the PlannerTests. This makes the assignments deterministic. No other tests depends on the exact assignments, so this does not regress anything. Testing: - Local testing - Ran gerrit-verify-dryrun-external - Verified minicluster profile 2 compiles Change-Id: I3d639128a856254a6ccb93d6750f531974b5f897 Reviewed-on: http://gerrit.cloudera.org:8080/10447 Reviewed-by: Philip Zeyliger Tested-by: Impala Public Jenkins --- M bin/run-all-tests.sh A fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java A fe/src/compat-minicluster-profile-3/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/bin/create-load-data.sh M testdata/bin/generate-schema-statements.py D testdata/bin/split-hbase.sh M testdata/datasets/functional/functional_schema_template.sql D testdata/src/compat-minicluster-profile-2/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssigment.java D testdata/src/compat-minicluster-profile-3/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssigment.java 10 files changed, 329 insertions(+), 728 deletions(-) Approvals: Philip Zeyliger: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3d639128a856254a6ccb93d6750f531974b5f897 Gerrit-Change-Number: 10447 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7061: Rework HBase splitting and assignment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10447 ) Change subject: IMPALA-7061: Rework HBase splitting and assignment .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d639128a856254a6ccb93d6750f531974b5f897 Gerrit-Change-Number: 10447 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 25 May 2018 00:28:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor tje INVALIDATE METADATA and REFRESH docs
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10339 ) Change subject: IMPALA-6987: [DOCS] Refactor tje INVALIDATE METADATA and REFRESH docs .. Patch Set 3: The latest HTML docs were shared with Balazs and Dimitris. -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 25 May 2018 00:28:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs
Hello Balazs Jeszenszky, Alex Behm, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10339 to look at the new patch set (#4). Change subject: IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs .. IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 --- M docs/shared/impala_common.xml M docs/topics/impala_invalidate_metadata.xml M docs/topics/impala_refresh.xml 3 files changed, 269 insertions(+), 415 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10339/4 -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7012: Fix NPE when parsing unexpected tokens
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10512 Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. IMPALA-7012: Fix NPE when parsing unexpected tokens Currently some token are not added to tokenIdMap in the sql scanner and an NPE will be thrown if such are parsed as unexpected tokens. This patch fixes this bug and adds a case to ParserTest. Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 --- M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 3 files changed, 28 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/10512/1 -- To view, visit http://gerrit.cloudera.org:8080/10512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10512 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-5392: Added all stack frames to ThreadInfo summary.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10145 ) Change subject: IMPALA-5392: Added all stack frames to ThreadInfo summary. .. Patch Set 10: > > (1 comment) > > Hi Jim, in the original implementation > (http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/management/ThreadInfo.java#631), > if number of stack frames are more than the max_limit, an ellipses > is added followed by a newline. > > In my implementation, I removed this piece of code @ line 631 > above. Do you mean that newline is missing? I do not think we need > to add ellipses now. > > Let me know if this is what you are suggesting. I also just realized that this is GPL and so cannot be in our repository: https://www.apache.org/legal/resolved.html#category-x Let me do some research to see how to deal with this. -- To view, visit http://gerrit.cloudera.org:8080/10145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 Gerrit-Change-Number: 10145 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Sharma Gerrit-Reviewer: Abhishek Sharma Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Charles Agnello Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 25 May 2018 02:47:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5392: Added all stack frames to ThreadInfo summary.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10145 ) Change subject: IMPALA-5392: Added all stack frames to ThreadInfo summary. .. Patch Set 10: Code-Review-2 Unfortunately, we cannot merge this, and we should be extraordinarily careful about merging another patch authored by you or I or anyone who looked at the OpenJDK code. https://issues.apache.org/jira/browse/LEGAL-388 -- To view, visit http://gerrit.cloudera.org:8080/10145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 Gerrit-Change-Number: 10145 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Sharma Gerrit-Reviewer: Abhishek Sharma Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Charles Agnello Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 25 May 2018 06:32:30 + Gerrit-HasComments: No
