[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 5: Code-Review-2 Marking this -2 while I still evaluate whether it serves our purpose when deploying Impala in containers. -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 05:35:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8434: retain tables and functions in altering database
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13904 )
Change subject: IMPALA-8434: retain tables and functions in altering database
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13904/2/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
File fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13904/2/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java@394
PS2, Line 394: CatalogObjectVersionSet.INSTANCE.updateVersions(
: existingDb.getCatalogVersion(), catalogVersion);
:
CatalogObjectVersionSet.INSTANCE.removeAll(existingDb.getTables());
: CatalogObjectVersionSet.INSTANCE.removeAll(
: existingDb.getFunctions(null, new PatternMatcher()));
: // IMPALA-8434: add back the existing tables/functions.
Note that their version
: // counters in CatalogObjectVersionSet have been
decreased by the above removeAll
: // statements, meaning their references from the old db
are deleted since the old
: // db object has been replaced by newDb.
: for (Table tbl: existingDb.getTables()) {
: newDb.addTable(tbl);
: }
This looks buggy to me. What does it mean to have tables in the db, without
having their version number maintained in the CatalogObjectVersionSet ? Aren't
we breaking the invariant that CatalogObjectVersionSet can get the correct min
version across all the objects (invalidate metadata relies on it)? or am I
misunderstanding something?
--
To view, visit http://gerrit.cloudera.org:8080/13904
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3dc9857fd2733e20cf10fbe17bb1a4670d7d015
Gerrit-Change-Number: 13904
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 25 Jul 2019 05:09:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4648/ -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 05:03:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4655/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 04:53:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4654/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 5 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 04:51:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 04:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 5 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 04:51:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 04:49:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4653/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 04:49:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Jul 2019 03:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3868 PS2, Line 3868: tEvents(Table table, > I think it is more readable if you rename this to partitionsBeforeInsert or I would keep this name as this name would highlight that the insert events are created only for affected existing partitions. http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3936 PS2, Line 3936:*/ > static Done http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3939 PS2, Line 3939: return partitions.stream().map(p -> (HdfsPartition) p).collect( > you could simply this (and below) with stream() and collect() Done http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3948 PS2, Line 3948:* - The table is missing (not yet loaded). > static? done -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Jul 2019 03:16:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. createInsertEvents() uses partition ids to keep track of the files added to the partitions by the insert by finding the delta of files in a partition before and after load() call. However, if partitions are marked dirty (for eg.: partitions created by RECOVER PARTITIONS), load() will drop and re-create them which will change the partition ids. createInsertEvents() then cannot find these parittions and fails with exception. In this patch, partitions are tracked by partition names instead of partition ids so drop + reload will not affect the logic. Testing: 1. Ran TestRecoverPartitions.test_post_invalidate() which was failing. 2. Ran MetastoreEventProcessorTest FE tests. Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 2 files changed, 35 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/4 -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 9 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 25 Jul 2019 03:01:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3990/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 02:57:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3989/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 02:50:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Jiawei Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/13801 )
Change subject: IMPALA-5149: Provide query profile in JSON format
..
Patch Set 9:
(4 comments)
Done, thanks for the feedback. I think we need more discussion on the version
control thing as Andrew and Bharath mentioned.
http://gerrit.cloudera.org:8080/#/c/13801/8/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:
http://gerrit.cloudera.org:8080/#/c/13801/8/be/src/util/runtime-profile-counters.h@292
PS8, Line 292: val->RemoveMember("kind");
> Should we not update the kind ? Why remove it?
SummaryStatsCounter and TimeSeriesCounter are treated separately with normal
counters in our JSON output. So a field like "kind" would be redundant.
“counters”: [
{
“counter_name”: xxx,
“value”: xxx,
“kind”: xxx,
“unit”: xxx,
“child_counters”: [{...}]// same structure as counter
… //type special values
},{...}
],
“summary_stats_counters” : [
{
“counter_name”: xxx,
“value”: xxx,
“unit”: xxx,
“min”: xxx,
“max”: xxx
“avg”: xxx
“num_of_samples”: xxx
},{...}
]
The difference as you can see from the schema, it's that we already know all
the counters from summary_stats_counters belongs to SummaryStatsCounter. So it
would not be necessary to keep the "kind" field again.
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py
File tests/hs2/test_hs2.py:
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py@668
PS8, Line 668:
> This looks flaky. Instead assert that they exist before you can loop throug
Done
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py@670
PS8, Line 670: if ("child_profiles" not in json_res["contents"]) or \
> dump the json_res if the assert fails, helps with debugging test failures.
Done
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/webserver/test_web_pages.py@594
PS8, Line 594: assert False, "Download JSON format query profile
cannot be parsed. " \
> dump json
Done
--
To view, visit http://gerrit.cloudera.org:8080/13801
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1
Gerrit-Change-Number: 13801
Gerrit-PatchSet: 9
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Thu, 25 Jul 2019 02:20:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Jiawei Wang has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. IMPALA-5149: Provide query profile in JSON format Description: Today there is a text and Thrift version of the query profile, but it would be useful to have a JSON version for portability and machine consumption. The ultimate goal is to have a Download JSON format profile link along with the other two formats. Modification: 1.Add Json format download option in impalad profile page 2.Add ToJson() function for RuntimeProfile, Counters, EventSequence 3.Add JSON format into QueryStateRecord 4.Add tests for E2E test, unit test, hs2 test 5.Modify query profile page to different download option 6.Modify HS2 server to support get JSON format profile Tests: E2E tests: tests/webserver/test_web_pages.py - test_download_profile * merge text and json format download together HS2 tests: tests/hs2/test_hs2.py - test_get_profile * add json format test BE Unit tests: be/src/util/runtime-profile-test.cc * ToJson.RuntimeProfileToJsonTest * ToJson.EmptyTest * ToJson.EventSequenceToJsonTest * ToJson.TimeSeriesCounterToJsonTest Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/CMakeLists.txt M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/ImpalaService.thrift M common/thrift/RuntimeProfile.thrift M tests/hs2/test_hs2.py M tests/webserver/test_web_pages.py M www/query_profile.tmpl 16 files changed, 620 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13801/9 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 9 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13893 to look at the new patch set (#4). Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. IMPALA-8772: Import Testcase failed for SQL without table refs Description: Query like this: select 5 * 4; can generate valid testcase, but cannot be loaded because it does not involve any table and view references. Fix: Add null check for CatalogOpExecutor when doing testcase data load Tests: tests/metadata/test_testcase_builder Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e TODO: NEED TO DELETE GENERATED TESTCASE AFTER THIS IS DONE --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/metadata/test_testcase_builder.py 2 files changed, 92 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13893/4 -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Jiawei Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 3: (1 comment) Thanks for the feedback. http://gerrit.cloudera.org:8080/#/c/13893/2/tests/metadata/test_testcase_builder.py File tests/metadata/test_testcase_builder.py: http://gerrit.cloudera.org:8080/#/c/13893/2/tests/metadata/test_testcase_builder.py@51 PS2, Line 51: try: : # Test load testcase works > you can use delete_file_dir from hdfs_util.py . Also please put it in a fin Done -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 01:55:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13893/3/tests/metadata/test_testcase_builder.py File tests/metadata/test_testcase_builder.py: http://gerrit.cloudera.org:8080/#/c/13893/3/tests/metadata/test_testcase_builder.py@22 PS3, Line 22: from subprocess import call flake8: F401 'subprocess.call' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 25 Jul 2019 01:55:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13893 to look at the new patch set (#3). Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. IMPALA-8772: Import Testcase failed for SQL without table refs Description: Query like this: select 5 * 4; can generate valid testcase, but cannot be loaded because it does not involve any table and view references. Fix: Add null check for CatalogOpExecutor when doing testcase data load Tests: tests/metadata/test_testcase_builder Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e TODO: NEED TO DELETE GENERATED TESTCASE AFTER THIS IS DONE --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/metadata/test_testcase_builder.py 2 files changed, 93 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13893/3 -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 01:23:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 9: (7 comments) http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc@121 PS9, Line 121: *eos = true; Are you missing a consumer_eos_.NotifyAll() here ? http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/plan-root-sink.h File be/src/exec/plan-root-sink.h: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/plan-root-sink.h@84 PS9, Line 84: validation rules a May be helpful to spell out the details here. This function seems to be a bit of a misnomer as it does a couple of things: - validate the collection slots - update the number of row batches sent to this sink - check if the number of rows sent to this sink exceeds limit and if so, return error status Not sure what a good name is. ValidateBatchAndCheckLimit() ?? http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/plan-root-sink.h@84 PS9, Line 84: approriate typo http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/plan-root-sink.h@107 PS9, Line 107: Send() stale http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/blocking-row-batch-queue.cc File be/src/runtime/blocking-row-batch-queue.cc: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/blocking-row-batch-queue.cc@66 PS9, Line 66: row_batches_put_timer_->Set(batch_queue_->total_put_wait_time()); : row_batches_get_timer_->Set(batch_queue_->total_get_wait_time()); I believe the TODO has more to do with not using this pattern of setting the timer at the end. Instead of waiting till the end to set the timer, it may be better for the batch_queue_ to directly use the two timers so we don't have to wait until the scan node is closed before we can tell how much time is spent in the queue. http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/deque-row-batch-queue.h File be/src/runtime/deque-row-batch-queue.h: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/deque-row-batch-queue.h@57 PS9, Line 57: remanining typo. Also this line doesn't quite parse correctly. http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/deque-row-batch-queue.h@68 PS9, Line 68: int max_batches_; const -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 01:15:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13893 )
Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13893/2/tests/metadata/test_testcase_builder.py
File tests/metadata/test_testcase_builder.py:
http://gerrit.cloudera.org:8080/#/c/13893/2/tests/metadata/test_testcase_builder.py@51
PS2, Line 51: status = call(["hdfs", "dfs", "-rm", "-skipTrash",
testcase_path[1: -1]])
: assert status == 0, "Delete generated testcase file failed
with {0}".format(status)
you can use delete_file_dir from hdfs_util.py . Also please put it in a finally
block. (to make sure the deletion happens regardless of test case export
failures).
--
To view, visit http://gerrit.cloudera.org:8080/13893
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e
Gerrit-Change-Number: 13893
Gerrit-PatchSet: 2
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Comment-Date: Thu, 25 Jul 2019 01:02:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 9: (4 comments) I played around with this a little bit and was definitely able to see the spooling behaviour by suspending impala-shell. I ran into a couple of crashes though - I think the problems might be around null row batches being dequeued from the queue http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc@65 PS9, Line 65: rows_available_.NotifyAll(); It's better to drop the lock before calling notify, so that the woken thread can always grab the lock immediately. See https://issues.apache.org/jira/browse/IMPALA-6135 http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc@74 PS9, Line 74: way wake http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/exec/buffered-plan-root-sink.cc@104 PS9, Line 104: Status BufferedPlanRootSink::GetNext( I saw a null pointer dereference causing a crash in the function. Emailed Sahil the logs. http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/deque-row-batch-queue.cc File be/src/runtime/deque-row-batch-queue.cc: http://gerrit.cloudera.org:8080/#/c/13883/9/be/src/runtime/deque-row-batch-queue.cc@61 PS9, Line 61: result->Reset(); I was able to trigger a segfault from a null pointer here by running "select * from tpch.lineitem" from impala-shell and then cancelling the query. C [impalad+0xbd0462] impala::RowBatch::Reset()+0x12 C [impalad+0x116fd74] impala::DequeRowBatchQueue::Close()+0x44 C [impalad+0xf701f5] impala::BufferedPlanRootSink::Close(impala::RuntimeState*)+0x1b5 C [impalad+0xc0d04c] impala::FragmentInstanceState::Close()+0x2c C [impalad+0xc0fde2] impala::FragmentInstanceState::Exec()+0xc2 C [impalad+0xbfb4b7] impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)+0x287 C [impalad+0xe50220] impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::ThreadDebugInfo const*, impala::Promise*)+0x310 C [impalad+0xe50dca] boost::detail::thread_data, impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list5, boost::_bi::value, boost::_bi::value >, boost::_bi::value, boost::_bi::value*> > > >::run()+0x7a -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:53:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4647/ -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Jul 2019 00:48:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13889 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c7db75b564045106edf3d46e2c4a62be77359f Gerrit-Change-Number: 13889 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:46:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13889 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4651/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c7db75b564045106edf3d46e2c4a62be77359f Gerrit-Change-Number: 13889 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:46:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Specified the client flags should be on coordinator
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13914 ) Change subject: [DOCS] Specified the client flags should be on coordinator .. [DOCS] Specified the client flags should be on coordinator Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Reviewed-on: http://gerrit.cloudera.org:8080/13914 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_client.xml 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Gerrit-Change-Number: 13914 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Specified the client flags should be on coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13914 ) Change subject: [DOCS] Specified the client flags should be on coordinator .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/407/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Gerrit-Change-Number: 13914 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 00:32:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3988/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:30:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h File be/src/util/bit-packing.inline.h: http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h@84 PS4, Line 84: if (LIKELY((std::is_same::value > I modified ParquetBoolDecoder and some tests to only allow unsigned integer Thank you for doing this, nice to have fewer cases to handle in this code. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:17:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc@38 PS7, Line 38: output_batch->AcquireState(batch); > Thanks for the info Tim. Replaced this DeepCopyTo. I think BufferedTupleStr Yeah BTS always does a full copy of the RowBatch into its own buffers. -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Jul 2019 00:14:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Specified the client flags should be on coordinator
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13914 Change subject: [DOCS] Specified the client flags should be on coordinator .. [DOCS] Specified the client flags should be on coordinator Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 --- M docs/topics/impala_client.xml 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/13914/1 -- To view, visit http://gerrit.cloudera.org:8080/13914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Gerrit-Change-Number: 13914 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Specified the client flags should be on coordinator
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13914 ) Change subject: [DOCS] Specified the client flags should be on coordinator .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Gerrit-Change-Number: 13914 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 00:09:47 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Specified the client flags should be on coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13914 ) Change subject: [DOCS] Specified the client flags should be on coordinator .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/407/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778 Gerrit-Change-Number: 13914 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 00:09:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:50:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4650/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:48:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Hello Anurag Mantripragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13912 to look at the new patch set (#4). Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. IMPALA-8789: Add helper to initiate graceful shutdown This change adds a helper script to initiate graceful daemon shutdown via the signaling mechanism. It also includes that helper script in the docker containers. Testing: This change adds a test to verify that the script works as expected. In addition, I manually verified that the script gets added to the containers and that calling it inside the container will cause a shutdown as expected. Change-Id: I877483a385cd0747f69b82a6488de203a4029599 --- A bin/graceful_shutdown_backends.sh M docker/CMakeLists.txt M docker/setup_build_context.py M tests/custom_cluster/test_restart_services.py 4 files changed, 79 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/13912/4 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Add support to tag docker images when pushing them
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13913 ) Change subject: Add support to tag docker images when pushing them .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I370542127f190cc3e0be3facb3a0e691f101ef70 Gerrit-Change-Number: 13913 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:41:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/13868 )
Change subject: IMPALA-8339: Add local executor blacklist to coordinators
..
Patch Set 3:
(19 comments)
Looking good.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/admission-controller.cc@737
PS3, Line 737: std::s
nit: std:: not needed
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/cluster-membership-mgr.cc@272
PS3, Line 272: already be on the blacklist or probation.
This may be a case worth double checking. If a node is restarted and
re-register with the Statestore, is it guaranteed that the "delete" update will
happen before the "create" update ? Can updates arrive out of order ?
If it's somehow possible that the node being added is still on blacklist. It
may be safer to call FindAndRemove() unconditionally and log it if we happen to
find it on the blacklist in the "create" path.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/cluster-membership-mgr.cc@273
PS3, Line 273: DCHECK
DCHECK_EQ
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/cluster-membership-mgr.cc@350
PS3, Line 350: return;
Given the for-loop above, can an executor belong to more than one executor
groups ? If so, why is it okay to return early if it's not found in only one
of the groups ?
Also, may help to comment on the intention for doing this check. I suspect it's
to skip the rather heavy weight copying of the membership state below.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/cluster-membership-mgr.cc@355
PS3, Line 355: recovering_membership_.get() != nullptr
This is repeated at multiple places in this function. May make sense to factor
it out to a local variable for clarity.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h
File be/src/scheduling/executor-blacklist.h:
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h@32
PS3, Line 32: Remove()
FindAndRemove()
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h@38
PS3, Line 38: When a blacklisted executor has passed this timeout
: /// and Maintenance() is called, the executor is put on
'probation'
May help to also document what if an executor on blacklist is removed during
cluster membership updates.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h@42
PS3, Line 42: THere
nit: typo
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h@79
PS3, Line 79: unblacklisted
May be clearer to call it "probation_list" or something so it can relates to
the state in which the backends on this list are in.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.h@124
PS3, Line 124: pzrobation
typo
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.cc
File be/src/scheduling/executor-blacklist.cc:
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.cc@29
PS3, Line 29: 0 disables blacklisting
This seems a bit weird. Setting it to 0 means the blacklist timeout ==
statestore_max_missed_heartbeats * statestore_heartbeat_frequency_ms, right ?
Would it make more sense to set this to 120 or something by default so it means
(statestore_max_missed_heartbeats * statestore_heartbeat_frequency_ms) * (this
flag / 100.0) ? So setting this flag to 0 means the timeout is 0 which makes
sense to mean blacklisting is disabled.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.cc@39
PS3, Line 39: ExecutorBlacklist::Blacklist(const TBackendDescriptor& be_desc) {
Should this check if blacklisting is disabled at the beginning ?
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.cc@54
PS3, Line 54: UnixMillis()
Why not MonotonicMillis() ? This field shouldn't need to be shared across
multiple hosts, right ? MonotonicMillis() makes sure the clock will not go
backward in case the TSC on different cores are slightly out of sync.
http://gerrit.cloudera.org:8080/#/c/13868/3/be/src/scheduling/executor-blacklist.cc@83
PS3, Line 83: auto eq = [_desc](const Entry& existing) {
: // The IP addresses must already match, so it is sufficient
to check the port.
: DCHECK_EQ(existing.be_desc.ip_address, be_desc.ip_address);
: return existing.be_desc.address.port == be_desc.address.port;
: };
This seems to be used at more than one functions. May be worth factoring it out
as a separate function.
[Impala-ASF-CR] IMPALA-8641: Document compression codec zstd in Parquet
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13910 ) Change subject: IMPALA-8641: Document compression codec zstd in Parquet .. Patch Set 1: Hi Abhishek, I only gave +1 in case you want a review by Tim. If you are confident with the content, I can give +2. -- To view, visit http://gerrit.cloudera.org:8080/13910 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55dbc3297ec1560b04e4da2c93cc1aeb1e6fb2f Gerrit-Change-Number: 13910 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:29:30 + Gerrit-HasComments: No
[Impala-ASF-CR] Add support to tag docker images when pushing them
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13913 ) Change subject: Add support to tag docker images when pushing them .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I370542127f190cc3e0be3facb3a0e691f101ef70 Gerrit-Change-Number: 13913 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:22:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 4 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 24 Jul 2019 23:22:45 + Gerrit-HasComments: No
[Impala-ASF-CR] Add support to tag docker images when pushing them
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13913 Change subject: Add support to tag docker images when pushing them .. Add support to tag docker images when pushing them This change adds an optional flag -t to docker/push-images.sh which allows to specify a tag. Leaving it empty will omit adding a specific tag and docker will fall back to "latest". Testing: I tested this manually and confirmed that the flag works as expected. Change-Id: I370542127f190cc3e0be3facb3a0e691f101ef70 --- M docker/push-images.sh 1 file changed, 11 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13913/1 -- To view, visit http://gerrit.cloudera.org:8080/13913 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I370542127f190cc3e0be3facb3a0e691f101ef70 Gerrit-Change-Number: 13913 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3985/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:15:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 12 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jul 2019 23:06:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4649/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:02:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 23:02:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13889 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c7db75b564045106edf3d46e2c4a62be77359f Gerrit-Change-Number: 13889 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:45:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Hello Abhishek Rawat, Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13857 to look at the new patch set (#4). Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. IMPALA-8549: Add support for scanning DEFLATE text files Hadoop tools such as Hive and MapReduce support reading and writing text files compressed using the deflate algorithm. In Hadoop, the zlib library (an implementation of the DEFLATE algorithm) is used to compress text files into .DEFLATE files, which are not in the raw deflate format but rather the zlib format (the zlib library supports three flavors of deflate, and Hadoop is using the flavor that compresses data into deflate with zlib wrappings rather than the raw deflate format) This patch adds support to Impala for scanning .DEFLATE files of tables stored as text. To avoid confusion, it should be noted that although these files have a compression type of DEFLATE in Impala, they should be treated as if their compression type is ZLIB. Testing: There is a pre-existing unit test that validates compressing/decompressing data with compression type DEFLATE. Also, modified existing end-to-end testing that simulates querying files of various formats and compression types. All core tests pass. Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a --- M be/src/exec/hdfs-text-scanner.cc M be/src/exec/hdfs-text-scanner.h M be/src/util/codec.cc M testdata/workloads/functional-query/functional-query_exhaustive.csv M tests/query_test/test_compressed_formats.py 5 files changed, 18 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/13857/4 -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 4 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/13905/6/.gitignore File .gitignore: PS6: Need to undo these changes -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:36:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 6: I was able to make it work! -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:35:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4648/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:36:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Hello Anurag Mantripragada, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13905 to look at the new patch set (#6). Change subject: IMPALA-8785: give debug docker images a different name .. IMPALA-8785: give debug docker images a different name * Build scripts are generalised to have different targets for release and debug images. * Added new targets for the debug images: docker_debug_images, statestored_debug images. The release images still have the same names. * Separate build contexts are set up for the different base images. * The debug or release base image can be specified as the FROM for the daemon images. * start-impala-cluster.py picks the correct images for the build type Future work: We would like to generalise this to allow building from non-ubuntu-16.04 base images. This probably requires another layer of dockerfiles to specify a base image for impala_base with the required packages installed. Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 --- M .gitignore M bin/jenkins/dockerized-impala-run-tests.sh M bin/start-impala-cluster.py M docker/CMakeLists.txt M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M docker/impalad_executor/Dockerfile M docker/setup_build_context.py M docker/statestored/Dockerfile 10 files changed, 100 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13905/6 -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:10:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3984/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 22:10:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3983/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 21:59:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4646/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 21:48:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13912/1/docker/setup_build_context.py File docker/setup_build_context.py: http://gerrit.cloudera.org:8080/#/c/13912/1/docker/setup_build_context.py@86 PS1, Line 86: D > flake8: E501 line too long (94 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 21:28:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Hello Anurag Mantripragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13912 to look at the new patch set (#2). Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. IMPALA-8789: Add helper to initiate graceful shutdown This change adds a helper script to initiate graceful daemon shutdown via the signaling mechanism. It also includes that helper script in the docker containers. Testing: This change adds a test to verify that the script works as expected. In addition, I manually verified that the script gets added to the containers and that calling it inside the container will cause a shutdown as expected. Change-Id: I877483a385cd0747f69b82a6488de203a4029599 --- A bin/graceful_shutdown_backends.sh M docker/CMakeLists.txt M docker/setup_build_context.py M tests/custom_cluster/test_restart_services.py 4 files changed, 89 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/13912/2 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13912 ) Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13912/1/docker/setup_build_context.py File docker/setup_build_context.py: http://gerrit.cloudera.org:8080/#/c/13912/1/docker/setup_build_context.py@86 PS1, Line 86: D flake8: E501 line too long (94 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 21:19:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8789: Add helper to initiate graceful shutdown
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13912 Change subject: IMPALA-8789: Add helper to initiate graceful shutdown .. IMPALA-8789: Add helper to initiate graceful shutdown This change adds a helper script to initiate graceful daemon shutdown via the signaling mechanism. It also includes that helper script in the docker containers. Testing: This change adds a test to verify that the script works as expected. In addition, I manually verified that the script gets added to the containers and that calling it inside the container will cause a shutdown as expected. Change-Id: I877483a385cd0747f69b82a6488de203a4029599 --- A bin/graceful_shutdown_backends.sh M docker/CMakeLists.txt M docker/setup_build_context.py M tests/custom_cluster/test_restart_services.py 4 files changed, 88 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/13912/1 -- To view, visit http://gerrit.cloudera.org:8080/13912 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I877483a385cd0747f69b82a6488de203a4029599 Gerrit-Change-Number: 13912 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13906 ) Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. IMPALA-8615: [DOCS] Document the scalable admission control parameters - Max Memory Multiple - Max Running Queries Multiple - Max Queued Queries Multiple Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Reviewed-on: http://gerrit.cloudera.org:8080/13906 Tested-by: Impala Public Jenkins Reviewed-by: Andrew Sherman --- M docs/topics/impala_admission.xml 1 file changed, 36 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Andrew Sherman: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13906 ) Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 20:45:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3982/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 20:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 20:32:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13883 to look at the new patch set (#9). Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl Improves the encapsulation of RowBatchQueue by the doing the following re-factoring: * Renames RowBatchQueue to BlockingRowBatchQueue which is more indicitive of what the queue does * Re-factors the timers managed by the scan-node into the BlockingRowBatchQueue implementation * Favors composition over inheritance by re-factoring BlockingRowBatchQueue to own a BlockingQueue rather than extending one The re-factoring lays the groundwork for introducing a generic RowBatchQueue that all RowBatch queues inherit from. Adds a new DequeRowBatchQueue which is a simple wrapper around a std::deque that (1) stores unique_ptr to queued RowBatch-es and (2) has a maximum capacity. Implements BufferedPlanRootSink using the new DequeRowBatchQueue. DequeRowBatchQueue is generic enough that replacing it with a SpillableQueue (queue backed by a BufferedTupleStream) should be straightforward. BufferedPlanRootSink is synchronized to protect access to DequeRowBatchQueue since the queue is not thread safe. BufferedPlanRootSink FlushFinal blocks until the consumer thread has processed all RowBatches. This ensures that the coordinator fragment stays alive until all results are fetched, but allows all other fragments to be shutdown immediately. Testing: * Running core tests * Updated tests/query_test/test_result_spooling.py Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/scanner-context.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/blocking-row-batch-queue.cc A be/src/runtime/blocking-row-batch-queue.h A be/src/runtime/deque-row-batch-queue.cc A be/src/runtime/deque-row-batch-queue.h D be/src/runtime/row-batch-queue.cc D be/src/runtime/row-batch-queue.h M tests/query_test/test_result_spooling.py 19 files changed, 517 insertions(+), 194 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13883/9 -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/8/be/src/runtime/blocking-row-batch-queue.cc
File be/src/runtime/blocking-row-batch-queue.cc:
http://gerrit.cloudera.org:8080/#/c/13883/8/be/src/runtime/blocking-row-batch-queue.cc@46
PS8, Line 46: bool
BlockingRowBatchQueue::AddBatchWithTimeout(unique_ptr batch, int64_t
timeout_micros) {
line too long (101 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 19:52:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue interface and BufferedPRS imp
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue interface and
BufferedPRS imp
..
Patch Set 3:
(27 comments)
http://gerrit.cloudera.org:8080/#/c/13883/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13883/3//COMMIT_MSG@9
PS3, Line 9: a blocking and
: non-blocking implementation
> It may help to spell out the exact difference in the two categories.
Updated the commit message now that the changes to add the generic
RowBatchQueue aren't going in this patch.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.h
File be/src/exec/buffered-plan-root-sink.h:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.h@29
PS3, Line 29: The blocking behavior follows
: /// the same semantics as BlockingPlanRootSink.
> Yes, I guess at some point soon, we may want to consolidate on the implemen
Done
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.h@42
PS3, Line 42: RowBatchQueue* batch_queue)
> Thanks for the link. I can see the advantage being that different users of
Agreed. Changed so that BufferedPlanRootSink owns the given batch_queue.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.h@65
PS3, Line 65: boost::mutex
> std::mutex. IIUC, the general direction is to move away from boost library.
It seems that our ConditionVariables (util/condition-variable.h) are tied to
boost::mutex-es. So I can't seem to use std::mutex here with some more
re-factoring.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/buffered-plan-root-sink.cc@38
PS3, Line 38: output_batch->AcquireState(batch);
> This is not making a copy. This is actually transferring the ownership of t
Replaced this with DeepCopyTo per Tim's suggestion. Note this will probably
change when we move to the BufferedTupleStream implementation as a BTS does its
own copy of the given batch.
The call to DeepCopyTo is necessary when using a std::deque because the
FragmentInstanceState owns the given RowBatch.
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc@38
PS7, Line 38: output_batch->AcquireState(batch);
> AcquireState isn't safe to use here - it doesn't copy into fresh memory, ju
Thanks for the info Tim. Replaced this DeepCopyTo. I think BufferedTupleStream
is already doing something like DeepCopyTo, correct?
Currently, the copy is necessary because FragmentInstanceState owns the given
RowBatch. However, once we migrate to the BTS queue, the call to DeepCopyTo
shouldn't be necessary, because BTS already does a copy, right?
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/data-sink.cc
File be/src/exec/data-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/data-sink.cc@114
PS3, Line 114: new NonBlockingRowBatchQueue(10))
> Who owns this object ? It doesn't look like it's being added to pool. We us
Fixed so that BufferedPlanRootSink owns and manages the lifecycle of the queue;
the queue is managed be a unique_ptr.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/plan-root-sink.h
File be/src/exec/plan-root-sink.h:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/plan-root-sink.h@61
PS3, Line 61: = 0;
> Is it still valid in this patch ? Given PlanRootSink::Send() is not pure vi
Per Tim's suggestion. I moved the PlanRootSink::Send implementation to a new
helper method call ValidateRowBatch.
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/plan-root-sink.cc
File be/src/exec/plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/plan-root-sink.cc@64
PS7, Line 64: Status PlanRootSink::Send(RuntimeState* state, RowBatch* batch) {
> I'd find it clearer if this was a helper function instead of the default im
Makes sense, moved this to a new method called ValidateRowBatch.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/scan-node.cc
File be/src/exec/scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/exec/scan-node.cc@315
PS3, Line 315: Shutdown();
> Close();
Now that I removed the generic RowBatchQueue interface from this patch,
BlockingRowBatchQueue has a Shutdown() method.
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/runtime/blocking-row-batch-queue.h
File be/src/runtime/blocking-row-batch-queue.h:
http://gerrit.cloudera.org:8080/#/c/13883/3/be/src/runtime/blocking-row-batch-queue.h@54
PS3, Line 54: public BlockingQueue, RowBatchBytesFn>,
: public RowBatchQueue {
> Not a big fan of multi-inheritance.
Now that I
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13883 to look at the new patch set (#8). Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl Improves the encapsulation of RowBatchQueue by the doing the following re-factoring: * Renames RowBatchQueue to BlockingRowBatchQueue which is more indicitive of what the queue does * Re-factors the timers managed by the scan-node into the BlockingRowBatchQueue implementation * Favors composition over inheritance by re-factoring BlockingRowBatchQueue to own a BlockingQueue rather than extending one The re-factoring lays the groundwork for introducing a generic RowBatchQueue that all RowBatch queues inherit from. Adds a new DequeRowBatchQueue which is a simple wrapper around a std::deque that (1) stores unique_ptr to queued RowBatch-es and (2) has a maximum capacity. Implements BufferedPlanRootSink using the new DequeRowBatchQueue. DequeRowBatchQueue is generic enough that replacing it with a SpillableQueue (queue backed by a BufferedTupleStream) should be straightforward. BufferedPlanRootSink is synchronized to protect access to DequeRowBatchQueue since the queue is not thread safe. BufferedPlanRootSink FlushFinal blocks until the consumer thread has processed all RowBatches. This ensures that the coordinator fragment stays alive until all results are fetched, but allows all other fragments to be shutdown immediately. Testing: * Running core tests * Updated tests/query_test/test_result_spooling.py Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/scanner-context.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/blocking-row-batch-queue.cc A be/src/runtime/blocking-row-batch-queue.h A be/src/runtime/deque-row-batch-queue.cc A be/src/runtime/deque-row-batch-queue.h D be/src/runtime/row-batch-queue.cc D be/src/runtime/row-batch-queue.h M tests/query_test/test_result_spooling.py 19 files changed, 516 insertions(+), 194 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13883/8 -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 5: For IMPALA-8770 I'm leaning towards just using different base dockerfiles for different distros. Using template substitution to install arbitrary packages or do whatever other customisation seems like the wrong mechanism. So I'll go with --build-arg -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 18:52:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Jul 2019 18:14:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4647/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Jul 2019 18:14:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Jul 2019 18:13:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 2: > (1 comment) > > Latest patch looks good to me. I think since this bug can be > reproduced reliably, it would be great we should add a test to > existing python test for inserts. Discussed this with Anurag. It seems like there is already a test which fails (TestRecoverPartitions.test_post_invalidate()) due to this bug. Even if we add a new test it will be doing exactly the same as the failing test. Since we have coverage already, we can skip adding a new test. -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Jul 2019 18:13:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt File docker/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt@62 PS5, Line 62: Docker does not support : # variables in the FROM clause > It does work for me now, wonder what I did wrong first time. Before I go ba The build args would handle the build_type stuff in this change. I think it could potentially handle IMPALA-8770. I would maybe lean towards using the build args for now. Even so, I don't think I have strong preferences one way or the other. The templating isn't that bad in terms of complexity. -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 18:07:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3980/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:56:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt File docker/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt@62 PS5, Line 62: Docker does not support : # variables in the FROM clause > I tried to do this but didn't get it to work. Maybe I just made a mistake a It does work for me now, wonder what I did wrong first time. Before I go back and undo the templating, you think that the --build-arg mechanism is the right way to go? I think I agree... * Pros for templating: it's more extensible in future, matches standard practices in the docker community. * Cons for templating: it adds a little more complexity that we don't need right now -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:54:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 9: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3979/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:49:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt File docker/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt@62 PS5, Line 62: Docker does not support : # variables in the FROM clause > When I was looking into this, it seemed like ARG might work: I tried to do this but didn't get it to work. Maybe I just made a mistake and gave up too soon. I did see some docs that suggested it only supported substituting the version field, but it's entirely possible I just misunderstood. I'll have another go. -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:47:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3978/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:44:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3977/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:38:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/16/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/16/be/src/util/vectorised_bit_unpacking_generator.py@192 PS16, Line 192: = flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:36:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h File be/src/util/bit-packing.inline.h: http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h@84 PS4, Line 84: std::pair BitPacking::UnpackValues( > Probably ParquetBoolDecoder::DecodeValue(). Might be worth seeing if you ca I modified ParquetBoolDecoder and some tests to only allow unsigned integer types, it seems to work. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:36:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/exec/parquet/parquet-bool-decoder.h M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 12 files changed, 6,023 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/16 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/13905 )
Change subject: IMPALA-8785: give debug docker images a different name
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt
File docker/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/13905/5/docker/CMakeLists.txt@62
PS5, Line 62: Docker does not support
: # variables in the FROM clause
When I was looking into this, it seemed like ARG might work:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
I did a quick test where I changed impalad_coordinator so that it looks like
this:
ARG BASE_IMAGE=impala_base
FROM ${BASE_IMAGE}
and it seems to work. Then you should be able to pass in an argument to docker
build with --build-arg =
--
To view, visit http://gerrit.cloudera.org:8080/13905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244
Gerrit-Change-Number: 13905
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:35:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 15: (2 comments) http://gerrit.cloudera.org:8080/#/c/13807/15/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/15/be/src/util/vectorised_bit_unpacking_generator.py@5 PS15, Line 5: """ Can you extend this comment? The following info could be useful: - this script doesn't need to be run during impala builds/test, so it can be python 3 - the general signature of the generated functions (maybe even a whole function could be added as an example, and an assert could be added to main as a smoke test to check whether it is generated for the given parameters) - the main types of the different implementations (bit scatter, AVX2 4 values at a time, AVX2 8 values at a time) could be mentioned with some rough explanation, possibly links to the most important instructions. The reasons why so many different algorithms were needed could be also mentioned, e.g. that vector operations are both constrained during read (how many values fit to a 32/64 bit broadcast read) and unpacking (there is only 32/64 bit variable length shifting, so there is no way to handle more than 8 values at a time) http://gerrit.cloudera.org:8080/#/c/13807/15/be/src/util/vectorised_bit_unpacking_generator.py@39 PS15, Line 39: style: we generally use indentation of 2 for block in Impala, even in py code -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 17:32:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue interface and BufferedPRS imp
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue interface and
BufferedPRS imp
..
Patch Set 7:
(2 comments)
Didn't do a full review, but wanted to point out a memory-safety issue because
of the arcane memory transfer rules around RowBatches.
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/buffered-plan-root-sink.cc@38
PS7, Line 38: output_batch->AcquireState(batch);
AcquireState isn't safe to use here - it doesn't copy into fresh memory, just
does a shallow move of the batches state. You don't know what the batch might
have pointers to. There are two potential problems
* If the FLUSH_RESOURCES flag is set, it's necessary for correctness to copy
this batch and all preceding batches, since the memory they reference may be
going away
* The batches may be keeping alive a lot more memory than they're actually
references, e.g. if a lot of rows got filtered out by predicates
NljBuilder correctly handled both of these problems, but I don't think it's
worth the complexity of trying to do that (you'd, at minimum, need to iterate
through the queue and deep copy all the batches).
This is all a consequence of Impala very aggressively trying to avoid copying
memory.
I'd recommend using DeepCopyTo() like here:
https://github.com/apache/impala/blob/master/be/src/exec/nested-loop-join-builder.cc#L97
There are probably relatively few common query patterns that would exercise
this since most common plans have an exchange feeding into PlanRootSink. A
grouping aggregation feeding directly into PlanRootSink might be able to
reproduce it (you'd have to set num_nodes=1 to get a non-distributed plan).
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/plan-root-sink.cc
File be/src/exec/plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/7/be/src/exec/plan-root-sink.cc@64
PS7, Line 64: Status PlanRootSink::Send(RuntimeState* state, RowBatch* batch) {
I'd find it clearer if this was a helper function instead of the default
implementation of Send() - that would make it clear that it wasn't a (possibly)
complete implementation of Send(), which is what I'd generally expect for a
default implementation.
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:30:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13906 ) Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/406/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 17:11:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
File fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java:
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java@123
PS8, Line 123: String[] commandWithoutAuth = {"impala-shell.sh", "",
String.format("--query=%s", query)};
> line too long (94 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 8
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:07:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#9). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 16 files changed, 288 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/9 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13906 ) Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/406/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 17:06:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13906 ) Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13906/2/docs/topics/impala_admission.xml File docs/topics/impala_admission.xml: http://gerrit.cloudera.org:8080/#/c/13906/2/docs/topics/impala_admission.xml@114 PS2, Line 114: This setting : scales with the number of hosts in the resource pool. > Instead of 'hosts' I think you should say 'executors', to be consistent. Done -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 17:06:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8615: [DOCS] Document the scalable admission control parameters
Hello Andrew Sherman, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13906 to look at the new patch set (#3). Change subject: IMPALA-8615: [DOCS] Document the scalable admission control parameters .. IMPALA-8615: [DOCS] Document the scalable admission control parameters - Max Memory Multiple - Max Running Queries Multiple - Max Queued Queries Multiple Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 --- M docs/topics/impala_admission.xml 1 file changed, 36 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13906/3 -- To view, visit http://gerrit.cloudera.org:8080/13906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibca9cf9586359ee0f1ce0dd8744b4709752a26f1 Gerrit-Change-Number: 13906 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7991 IMPALA-8575: [DOCS] Document the query options for Parquet page indexes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13900 ) Change subject: IMPALA-7991 IMPALA-8575: [DOCS] Document the query options for Parquet page indexes .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/405/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c1941269feff18306863f784aa36f5037da1db Gerrit-Change-Number: 13900 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jul 2019 17:04:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py@804
PS6, Line 804: print_to_stderr("protocol: " + options.protocol.lower())
: assert options.protocol.lower() == 'beeswax'
> Maybe I'm wrong, but it doesn't look like this value is validated anywhere
Done
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:02:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
File fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java:
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java@123
PS8, Line 123: String[] commandWithoutAuth = {"impala-shell.sh", "",
String.format("--query=%s", query)};
line too long (94 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 8
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:02:28 +
Gerrit-HasComments: Yes
