[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3955/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 23 Jul 2019 05:56:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/13868 )
Change subject: IMPALA-8339: Add local executor blacklist to coordinators
..
Patch Set 2:
(31 comments)
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/runtime/coordinator-backend-state.h@284
PS1, Line 284: ExecQu
> ExecQueryFInstance()
Done
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/runtime/coordinator.cc@394
PS1, Line 394: // The Exec() rpc failed, so blacklist the executor.
: LOG(INFO) << "Blacklisting "
: <<
TNetworkAddressToString(backend_state->impalad_address()
> One interesting case is what if the coordinator cannot RPC to itself. I sup
Good point. Probably makes sense to have a special case for not blacklisting
the local backend.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h
File be/src/scheduling/cluster-membership-mgr.h:
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h@96
PS1, Line 96: Only
> typo
Done
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h@168
PS1, Line 168: void UpdateMembership(const
StatestoreSubscriber::TopicDeltaMap& incoming_topic_deltas,
> nit: Adds, Updates.
Done
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h@170
PS1, Line 170:
> Should this method accept an IP address instead?
Discussed on another comment.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h@206
PS1, Line 206: cutorGroups& executor_gr
> Is there any chance this mutex may be held at the same time as various othe
Done
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.h@206
PS1, Line 206: const ExecutorGroups& executor_groups, const
ExecutorBlacklist& executor_blacklist);
> Does this also need the mutable modifier like the other two locks ?
No, 'mutable' is used to allow a field to be modified in a function that's
marked 'const'. update_membership_lock_ isn't taken in any const functions.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc@119
PS1, Line 119:
: // Check if the local backend is up and needs updating.
> I suppose this relies on StateStore being alive and keeps sending periodic
That's a good point. This approach (maintaining the blacklist on the statestore
update thread) is what we had put in the design doc, but since I've gone with a
design that requires the lock 'update_membership_lock_' anyways, it might be
nice to just put this in its own thread.
That would also make it easier to change the frequency that we check the
blacklist for updates - we receive statestore updates every 100ms by default
(most of which of course are no-ops) which might be more frequent than we
really need to do this check (by default, the shortest time a node is
blacklisted for is 12s anyways, probably not a big deal if we only do this
update once per second or so)
It could remove having to reason about how to handle nodes that were
deleted/updated/added in the same update as we decide to unblacklist them,
which you had questions about below.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc@168
PS1, Line 168: if (recovering_
> How would this work if the deleted item is in the list unblacklisted ?
I think this and your other similar concerns below are addressed by taking
Lars's suggestion of only constructing 'unblacklisted' after the statestore
update has been fully processed, since we call 'Remove' on the blacklist here.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc@175
PS1, Line 175: rship_);
> Otherwise, won't it eventually get into the
> probation state and get added back with the stale BE descriptor ?
NodeBlacklist::Remove() completely removes executors from the blacklist, it
doesn't put them on probation.
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc@186
PS1, Line 186: for (const TTopicItem& item : update.topic_entries) {
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13868/1/be/src/scheduling/cluster-membership-mgr.cc@265
PS1, Line 265: new_backend_map->insert(make_pair(item.key, be_desc))
> How would this work if the cluster membership in the past indicates that th
Done
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13868 to look at the new patch set (#3). Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added an e2e test case that kills an impalad. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-blacklist.cc A be/src/scheduling/executor-blacklist.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_blacklist.py 15 files changed, 656 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13868/3 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13868 )
Change subject: IMPALA-8339: Add local executor blacklist to coordinators
..
Patch Set 2:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/13868/2/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/13868/2/be/src/scheduling/cluster-membership-mgr.cc@468
PS2, Line 468: const ExecutorGroups& executor_groups, const
ExecutorBlacklist& executor_blacklist) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py
File tests/custom_cluster/test_blacklist.py:
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@25
PS2, Line 25: class TestBlacklist(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@36
PS2, Line 36:
flake8: E203 whitespace before ','
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@36
PS2, Line 36: =
flake8: E711 comparison to None should be 'if cond is None:'
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@52
PS2, Line 52:
flake8: E203 whitespace before ','
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@61
PS2, Line 61:
flake8: E203 whitespace before ','
http://gerrit.cloudera.org:8080/#/c/13868/2/tests/custom_cluster/test_blacklist.py@61
PS2, Line 61: =
flake8: E711 comparison to None should be 'if cond is None:'
--
To view, visit http://gerrit.cloudera.org:8080/13868
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74
Gerrit-Change-Number: 13868
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Tue, 23 Jul 2019 05:16:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13868 to look at the new patch set (#2). Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added an e2e test case that kills an impalad. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-blacklist.cc A be/src/scheduling/executor-blacklist.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_blacklist.py 15 files changed, 661 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13868/2 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/13740 )
Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13740/11/www/admission_controller.tmpl
File www/admission_controller.tmpl:
http://gerrit.cloudera.org:8080/#/c/13740/11/www/admission_controller.tmpl@275
PS11, Line 275: Memory limit for the coordinator
: Memory admitted on the coordinator
: Num of backends it will run on
: Details
:
: {{#queued_queries}}
:
: {{query_id}}
: Memory limit for the executors
: Memory admitted on the executors
: {{coord_backend_mem_limit}}
: {{coord_backend_mem_to_admit}}
This looks wrong (noticed when trying the change on a real cluster)
--
To view, visit http://gerrit.cloudera.org:8080/13740
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Gerrit-Change-Number: 13740
Gerrit-PatchSet: 11
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 23 Jul 2019 05:11:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4636/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 9: Code-Review+2 Thanks for fixing this. -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8746: [DOCS] Document the DEFAULT HINTS INSERT STATEMENT query option
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13885 ) Change subject: IMPALA-8746: [DOCS] Document the DEFAULT_HINTS_INSERT_STATEMENT query option .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia376721f46eb507901f9f64b5c3341dc0f36475b Gerrit-Change-Number: 13885 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 02:50:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 1: Code-Review+2 (4 comments) Thanks for fixing this. http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py File tests/metadata/test_testcase_builder.py: http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@41 PS1, Line 41: execute_query execute_query_expect_success http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@43 PS1, Line 43: # Test load testcase works assert len(result.data) == 1 ? http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@48 PS1, Line 48: # TODO: Delete testcase file from tmp Implement the TODO? http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@49 PS1, Line 49: > flake8: W391 blank line at end of file Yep, remove these blank lines? -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Jul 2019 02:46:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3954/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 01:07:11 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13740/11/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/13740/11/be/src/scheduling/admission-controller.h@809 PS11, Line 809: /// TODO: Update comment Need to fix -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 00:36:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13896 ) Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/396/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 23 Jul 2019 00:31:46 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4634/ -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 00:29:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4635/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 00:29:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has uploaded a new patch set (#11) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. [WIP] IMPALA-7486: Prototype for admitting less mem on coord This patch computes two memory estimates in the frontend: an estimate for any host that is an executor (including a combined coordinator and executor) and an estimate for a dedicated coordinator. This is computed regardless of whether it is a dedicated coordinator or not. Admission control then, in the case when the coordinator is dedicated, uses the coordinator memory estimate for the coordinator node and the executor memory estimate for all other nodes. Complete: - Calculate the mem needed for the coord fragment + accounting for the runtime filter agg mem required + a safety buffer of 100MB - changes to get the mem_to_admit/limit for coord, accesible through the QuerySchedule - add/remove/modify checks in both reject and admit methods to seperate the coord and executor checks - Update memory accounting in admission controller - use coord's mem limit for the coord queryState - added BE unit tests for most cases - Updated the Admission debug page to show the coord mem values TODO: - add e2e tests: - Simple case (done) - query with num_node = 1 - Any additional test cases not covered by BE unit tests or need an equivalent e2e test - e2e tests are not written with dedicated coord, so all should work fine - add e2e tests (identified by tarmstrong) - Interaction between max_mem_estimate_for_admission and dedicated coordinator estimates Testing: * Added unit test for admission/rejection in dedicated coordinator clusters. Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 --- M be/src/runtime/coordinator.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_admission_controller.py M www/admission_controller.tmpl 28 files changed, 2,188 insertions(+), 1,323 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/11 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 ) Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3953/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50 Gerrit-Change-Number: 13872 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 23 Jul 2019 00:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3952/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 00:15:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13896 ) Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/396/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 23 Jul 2019 00:09:28 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Add debugging tools to our docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13895 ) Change subject: WIP: Add debugging tools to our docker images .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3951/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47c7aa7076cebfa3bfad2029fb1da9e64364f0e6 Gerrit-Change-Number: 13895 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Jul 2019 00:09:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13896 to look at the new patch set (#2). Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 --- M docs/topics/impala_client.xml 1 file changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/13896/2 -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4633/ -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 23:46:52 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4634/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 23:46:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13896 ) Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13896/1/docs/topics/impala_client.xml File docs/topics/impala_client.xml: http://gerrit.cloudera.org:8080/#/c/13896/1/docs/topics/impala_client.xml@144 PS1, Line 144: If a client session is idle for the duration of : --idle_client_poll_time_s seconds, the network connection of the : idle session is closed. Not sure if that's entirely accurate. This is how frequently poll will happen to check if a connection is idle and close it if it's idle. A connection is idle if all sessions associated with it are idle. http://gerrit.cloudera.org:8080/#/c/13896/1/docs/topics/impala_client.xml@155 PS1, Line 155: The session will only be closed The connection will only be closed if all sessions associated with it are all idle or closed. Sessions cannot be idle unless the flag --idle_session_timeout is greater than 0. -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 22 Jul 2019 23:40:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13896 ) Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/395/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 22 Jul 2019 23:38:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
Hello Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13872 to look at the new patch set (#4). Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage .. IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage Currently, fe/pom.xml includes direct dependencies on hadoop-aws, hadoop-azure, and hadoop-azure-datalake. Hadoop has a meta-package hadoop-cloud-storage that provides these dependencies with appropriate exclusions. This would not change Impala's build significantly, but it allows different versions or distributions of Hadoop to add necessary runtime jars to the classpath. For example, distributions of Hadoop that use Apache Knox for identity management can add Knox jars to this meta-package. This switches Impala to use the hadoop-cloud-storage package. This does not change the default configuration, but the USE_CDP_HIVE=true configuration gains Apache Knox jars. Due to the structure of the CDP maven repository, it has dependencies on versions of jars that are not in the CDP maven repository. So, USE_CDP_HIVE=true requires the Hortonworks public maven repository. This is added, but it is only enabled with USE_CDP_HIVE=true and only for releases (no SNAPSHOTs). Testing: - Ran core tests - Manually verified classpath changes Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50 --- M fe/pom.xml M impala-parent/pom.xml 2 files changed, 29 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13872/4 -- To view, visit http://gerrit.cloudera.org:8080/13872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50 Gerrit-Change-Number: 13872 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has uploaded a new patch set (#10) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. [WIP] IMPALA-7486: Prototype for admitting less mem on coord This patch computes two memory estimates in the frontend: an estimate for any host that is an executor (including a combined coordinator and executor) and an estimate for a dedicated coordinator. This is computed regardless of whether it is a dedicated coordinator or not. Admission control then, in the case when the coordinator is dedicated, uses the coordinator memory estimate for the coordinator node and the executor memory estimate for all other nodes. Complete: - Calculate the mem needed for the coord fragment + accounting for the runtime filter agg mem required + a safety buffer of 100MB - changes to get the mem_to_admit/limit for coord, accesible through the QuerySchedule - add/remove/modify checks in both reject and admit methods to seperate the coord and executor checks - Update memory accounting in admission controller - use coord's mem limit for the coord queryState - added BE unit tests for most cases - Updated the Admission debug page to show the coord mem values TODO: - add e2e tests: - Simple case (done) - query with num_node = 1 - Any additional test cases not covered by BE unit tests or need an equivalent e2e test - e2e tests are not written with dedicated coord, so all should work fine - add e2e tests (identified by tarmstrong) - Interaction between max_mem_estimate_for_admission and dedicated coordinator estimates - E2e dedicated coordinator estimate is used when dedicated coordinator is enabled. Testing: * Added unit test for admission/rejection in dedicated coordinator clusters. Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 --- M be/src/runtime/coordinator.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_admission_controller.py M www/admission_controller.tmpl 28 files changed, 2,186 insertions(+), 1,323 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/10 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 23:32:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13896 ) Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/395/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 23:29:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8701: [DOCS] Document --idle client poll time s flag
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13896 Change subject: IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag .. IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 --- M docs/topics/impala_client.xml 1 file changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/13896/1 -- To view, visit http://gerrit.cloudera.org:8080/13896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6 Gerrit-Change-Number: 13896 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] WIP: Add debugging tools to our docker images
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13895 Change subject: WIP: Add debugging tools to our docker images .. WIP: Add debugging tools to our docker images I often find it tricky to debug network and Impala issues when using our Docker images. This change adds a handful of tools that I frequently miss having. It adds about 6.5% to the image size, they grow from 984MB to 953MB. If people feel that that is too much, I'm happy to cur back on the tools we install. Change-Id: I47c7aa7076cebfa3bfad2029fb1da9e64364f0e6 --- M docker/impala_base/Dockerfile 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13895/1 -- To view, visit http://gerrit.cloudera.org:8080/13895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I47c7aa7076cebfa3bfad2029fb1da9e64364f0e6 Gerrit-Change-Number: 13895 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 22:59:22 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/13740/9/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13740/9/be/src/scheduling/admission-controller.cc@717 PS9, Line 717: << " and min_admit_mem_limit.second = " << min_executor_admit_mem_limit.second; line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 22:52:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4633/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 22:52:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py File tests/metadata/test_testcase_builder.py: http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@49 PS1, Line 49: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 22:20:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Jiawei Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13893 Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. IMPALA-8772: Import Testcase failed for SQL without table refs Description: Query like this: select 5 * 4; can generate valid testcase, but cannot be loaded because it does not involve any table and view references. Fix: Add null check for CatalogOpExecutor when doing testcase data load Tests: tests/metadata/test_testcase_builder Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e TODO: NEED TO DELETE GENERATED TESTCASE AFTER THIS IS DONE --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/metadata/test_testcase_builder.py 2 files changed, 84 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13893/1 -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang
[Impala-ASF-CR] IMPALA-8765: [DOCS] Document the GET JSON OBJECT function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13886 ) Change subject: IMPALA-8765: [DOCS] Document the GET_JSON_OBJECT function .. Patch Set 1: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/13886/1/docs/topics/impala_misc_functions.xml File docs/topics/impala_misc_functions.xml: http://gerrit.cloudera.org:8080/#/c/13886/1/docs/topics/impala_misc_functions.xml@137 PS1, Line 137: json_str is invalid. > Will also return NULL if nothing is selected, for example: Done http://gerrit.cloudera.org:8080/#/c/13886/1/docs/topics/impala_misc_functions.xml@143 PS1, Line 143:* : Denotes the wildcard for [] > "*" can also be wildcard for ".", for example: Done http://gerrit.cloudera.org:8080/#/c/13886/1/docs/topics/impala_misc_functions.xml@288 PS1, Line 288: Each call to UUID() produces a new arbitrary value. > Is this relative to the get_json_object function? No, I just piggybacked an editorial fix. -- To view, visit http://gerrit.cloudera.org:8080/13886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7135528e84f685bfe1c32d81f4cedb6afc133e04 Gerrit-Change-Number: 13886 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 22:11:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3948/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 21:26:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 6:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java
File fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@133
PS6, Line 133: } catch (AssertionError e) {
: LOG.error("Failed to run cmd: {} stdout: {}", cmd, stdout);
Instead of catching the assertion failure to log and rethrow it, I think you
can just pass the error message in as the first parameter to assertEquals()
http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@140
PS6, Line 140: BASIC
nit: I probably got this wrong in some places in my patches around this, but
this shouldn't all be capitalized
http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@143
PS6, Line 143: public void testShellBasicAuth() throws Exception {
We don't really want to inherit all of the JDBC related setup stuff here, eg.
currently its getting run twice cause JdbcTestBase is parameterized for binary
vs. http, so probably better to put this in its own class.
http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@149
PS6, Line 149: show tables
Could you make the 'select logged_in_user()' and actually verify that its
correct?
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py@804
PS6, Line 804: print_to_stderr("protocol: " + options.protocol.lower())
: assert options.protocol.lower() == 'beeswax'
Probably only want to log if its actually wrong, and we should probably return
a FatalShellException() in that case instead of assert-ing.
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 22 Jul 2019 21:18:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Format changes
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13892 ) Change subject: [DOCS] Format changes .. [DOCS] Format changes Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Reviewed-on: http://gerrit.cloudera.org:8080/13892 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/shared/impala_common.xml M docs/topics/impala_kudu.xml 2 files changed, 2 insertions(+), 3 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Gerrit-Change-Number: 13892 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Format changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13892 ) Change subject: [DOCS] Format changes .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/394/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Gerrit-Change-Number: 13892 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 20:50:47 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 8: PS8 rebases the change to the latest commit on master and executor-group-test, scheduler-test, admission-controller-test, and cluster-membership-mgr-test pass. The e2e tests don't work. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 20:47:31 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/13740/8/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13740/8/be/src/scheduling/admission-controller.cc@717 PS8, Line 717: << " and min_admit_mem_limit.second = " << min_executor_admit_mem_limit.second; line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 20:47:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Lars Volker has uploaded a new patch set (#8) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. [WIP] IMPALA-7486: Prototype for admitting less mem on coord This patch computes two memory estimates in the frontend: an estimate for any host that is an executor (including a combined coordinator and executor) and an estimate for a dedicated coordinator. This is computed regardless of whether it is a dedicated coordinator or not. Admission control then, in the case when the coordinator is dedicated, uses the coordinator memory estimate for the coordinator node and the executor memory estimate for all other nodes. Complete: - Calculate the mem needed for the coord fragment + accounting for the runtime filter agg mem required + a safety buffer of 100MB - changes to get the mem_to_admit/limit for coord, accesible through the QuerySchedule - add/remove/modify checks in both reject and admit methods to seperate the coord and executor checks - Update memory accounting in admission controller - use coord's mem limit for the coord queryState - added BE unit tests for most cases - Updated the Admission debug page to show the coord mem values TODO: - add e2e tests: - Simple case (done) - query with num_node = 1 - Any additional test cases not covered by BE unit tests or need an equivalent e2e test - e2e tests are not written with dedicated coord, so all should work fine Testing: * Added unit test for admission/rejection in dedicated coordinator clusters. Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 --- M be/src/runtime/coordinator.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_admission_controller.py M www/admission_controller.tmpl 27 files changed, 2,178 insertions(+), 1,319 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/8 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8765: [DOCS] Document the GET JSON OBJECT function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13886 ) Change subject: IMPALA-8765: [DOCS] Document the GET_JSON_OBJECT function .. Patch Set 1: Could you review this doc patch? -- To view, visit http://gerrit.cloudera.org:8080/13886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7135528e84f685bfe1c32d81f4cedb6afc133e04 Gerrit-Change-Number: 13886 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 20:32:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8746: [DOCS] Document the DEFAULT HINTS INSERT STATEMENT query option
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13885 ) Change subject: IMPALA-8746: [DOCS] Document the DEFAULT_HINTS_INSERT_STATEMENT query option .. Patch Set 1: Could you review this doc patch? Thanks. -- To view, visit http://gerrit.cloudera.org:8080/13885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia376721f46eb507901f9f64b5c3341dc0f36475b Gerrit-Change-Number: 13885 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 20:31:49 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Format changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13892 ) Change subject: [DOCS] Format changes .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/394/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Gerrit-Change-Number: 13892 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 20:29:12 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Format changes
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13892 Change subject: [DOCS] Format changes .. [DOCS] Format changes Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 --- M docs/shared/impala_common.xml M docs/topics/impala_kudu.xml 2 files changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/13892/1 -- To view, visit http://gerrit.cloudera.org:8080/13892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Gerrit-Change-Number: 13892 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Format changes
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13892 ) Change subject: [DOCS] Format changes .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib74a93bd01940621cbd5d0b5b993a1bddd76ce43 Gerrit-Change-Number: 13892 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Jul 2019 20:29:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13872 ) Change subject: IMPALA-8766: Change cloud dependencies to use hadoop-cloud-storage .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13872/3/impala-parent/pom.xml File impala-parent/pom.xml: http://gerrit.cloudera.org:8080/#/c/13872/3/impala-parent/pom.xml@186 PS3, Line 186: repository I think it would great, if we could add a comment here describing why we need this repository and reference to the JIRA number of this patch. -- To view, visit http://gerrit.cloudera.org:8080/13872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08f1c36ecf54ac277d99e2d2843163eada732e50 Gerrit-Change-Number: 13872 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 22 Jul 2019 19:16:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8727: [DOCS] Impala-side changes for Kudu HMS integration
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13776 ) Change subject: IMPALA-8727: [DOCS] Impala-side changes for Kudu HMS integration .. IMPALA-8727: [DOCS] Impala-side changes for Kudu HMS integration Change-Id: Ieec79ac0bbb860c6394a3bf0617b285a7d23ca9e Reviewed-on: http://gerrit.cloudera.org:8080/13776 Reviewed-by: Alexey Serbin Tested-by: Impala Public Jenkins Reviewed-by: Andrew Wong Reviewed-by: Thomas Tauber-Marshall --- M docs/shared/impala_common.xml M docs/topics/impala_kudu.xml M docs/topics/impala_tables.xml 3 files changed, 85 insertions(+), 64 deletions(-) Approvals: Alexey Serbin: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Andrew Wong: Looks good to me, but someone else must approve Thomas Tauber-Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieec79ac0bbb860c6394a3bf0617b285a7d23ca9e Gerrit-Change-Number: 13776 Gerrit-PatchSet: 11 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 8: Should the json include the explicit version number of the format? "version: "1.0" to help people writing parsers? Like Bharath (and DavidR in the design doc) I would like to see more about future compatibility guarantees -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 8 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 22 Jul 2019 17:50:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 8: There're some bugs in the current master branch of HUE that not showing anything in the table browser tab. I managed to test with HUE in version 4.4.0-release. Everything is fine :) -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Jul 2019 17:28:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13874 to look at the new patch set (#8). Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode In LocalCatalog implementation, LocalDb#getTable will always return a completely loaded table containing all the meta of columns, partitions, files, etc. It's time consuming if we implement the GET_TABLES HiveServer2 operation based on this interface, since GET_TABLES only requires table names, table types and table comments, while this interface will trigger catalogd to fully load the table meta. It becomes worse when we do this for all the tables. This patch introduces a new interface, getTableIfCached, to return a LocalIncompleteTable object if the corresponding table is unloaded, which requires no round trips to the catalogd. It's used to boost the GET_TABLES performance in LocalCatalog mode. Tests - Testing in a HMS with 100 dbs and 3000 tables, without this patch it takes ~2mins in GET_TABLES for all tables on a cold started cluster. With this patch, the time reduces to ~1s. - Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the same as using legacy catalog implementation. Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M tests/hs2/test_hs2.py 11 files changed, 153 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/8 -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8727: [DOCS] Impala-side changes for Kudu HMS integration
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13776 ) Change subject: IMPALA-8727: [DOCS] Impala-side changes for Kudu HMS integration .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieec79ac0bbb860c6394a3bf0617b285a7d23ca9e Gerrit-Change-Number: 13776 Gerrit-PatchSet: 10 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 22 Jul 2019 16:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4631/ -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 13:58:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13880 ) Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3947/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 3 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 12:59:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3946/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 7 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 12:23:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Norbert Luksa has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13880 Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. IMPALA-7770: SPLIT_PART to support negative indexes Third parameter of SPLIT_PART (nth field) accepts now negative values, and searches the string backwards. Testing: * Added unint tests to expr-test.cc Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc 2 files changed, 37 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/13880/3 -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 3 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.h File be/src/util/bit-packing.h: http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.h@67 PS4, Line 67: template > I made measurements and a graph and added it to the Jira issue: Csaba had an idea that we could get rid of the VECTORIZE parameter completely and test the non-vectorised code path by disabling the CPU flags in CpuInfo in the tests. http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h File be/src/util/bit-packing.inline.h: http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing.inline.h@84 PS4, Line 84: if (LIKELY((std::is_same::value > Does it even make sense to unpack values into a different type outside of t We also use bool somewhere, at least bit-packing.cc instantiates the methods with bool. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 11:47:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13870/1/be/src/exprs/string-functions-ir.cc File be/src/exprs/string-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/13870/1/be/src/exprs/string-functions-ir.cc@1190 PS1, Line 1190: if (s1len == s2len && memcmp(s1.ptr, s2.ptr, s1len) == 0) return DoubleVal(1.0); : if (s1len == 0) return DoubleVal(0.0); > I think my previous comment got forgotten. Done -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 7 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 11:41:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Norbert Luksa has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function The added functions return the Jaro/Jaro-Winkler similarity/distance of two strings. The algorithm calcuates the Jaro-Similarity of the strings, then adds more weight to the result if there are common prefixes. (Jaro-Winkler) For more detail, see: https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance Extended the algorithm with another optional parameter: boost threshold The prefix weight will only be applied if the Jaro-similarity exceeds the given threshold. By default, its value is 0.7. The new built-in functions are: * jaro_distance, jaro_dst * jaro_similarity, jaro_sim * jaro_winkler_distance, jw_dst * jaro_winkler_similarity, jw_sim Testing: * Added unit tests to expr-test.cc * Manual testing over 1400 word pairs from http://marvin.cs.uidaho.edu/misspell.html Results match Apache commons Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py 4 files changed, 323 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/13870/7 -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 7 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 12: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3945/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 11:36:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. Patch Set 6: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13870/1/be/src/exprs/string-functions-ir.cc File be/src/exprs/string-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/13870/1/be/src/exprs/string-functions-ir.cc@1190 PS1, Line 1190: if (s1len == s2len && memcmp(s1.ptr, s2.ptr, s1len) == 0) return DoubleVal(1.0); : if (s1len == 0) return DoubleVal(0.0); > Can we also add test cases when both parameters are empty strings? I think my previous comment got forgotten. -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 11:21:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 11:20:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/12/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/12/be/src/util/vectorised_bit_unpacking_generator.py@184 PS12, Line 184: = flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 10:54:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 12: Added consts to avoid clang tidy warnings. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 10:53:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 9 files changed, 5,962 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/12 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function
Norbert Luksa has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13870 ) Change subject: IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function .. IMPALA-8752: Added Jaro-Winkler edit distance and similarity built-in function The added functions return the Jaro/Jaro-Winkler similarity/distance of two strings. The algorithm calcuates the Jaro-Similarity of the strings, then adds more weight to the result if there are common prefixes. (Jaro-Winkler) For more detail, see: https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance Extended the algorithm with another optional parameter: boost threshold The prefix weight will only be applied if the Jaro-similarity exceeds the given threshold. By default, its value is 0.7. The new built-in functions are: * jaro_distance, jaro_dst * jaro_similarity, jaro_sim * jaro_winkler_distance, jw_dst * jaro_winkler_similarity, jw_sim Testing: * Added unit tests to expr-test.cc * Manual testing over 1400 word pairs from http://marvin.cs.uidaho.edu/misspell.html Results match Apache commons Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py 4 files changed, 319 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/13870/6 -- To view, visit http://gerrit.cloudera.org:8080/13870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I64d7f461516c5e66cc27d62612bc8cc0e8f0178c Gerrit-Change-Number: 13870 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8710: Increase allowed bit width to 64 for bit packing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13809 ) Change subject: IMPALA-8710: Increase allowed bit width to 64 for bit packing .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9276ca291e3d36db16f63f8caf3e9248a18d85b Gerrit-Change-Number: 13809 Gerrit-PatchSet: 9 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Jul 2019 10:20:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3943/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 10:07:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3941/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 10 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 09:49:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/11/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/11/be/src/util/vectorised_bit_unpacking_generator.py@184 PS11, Line 184: = flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 09:48:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 9 files changed, 5,962 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/11 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8710: Increase allowed bit width to 64 for bit packing
Daniel Becker has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13809 ) Change subject: IMPALA-8710: Increase allowed bit width to 64 for bit packing .. IMPALA-8710: Increase allowed bit width to 64 for bit packing Fixed the buffer overflow that the previous attempt (commit b1cbf9e6b786132e86699cbb1e472ec98499bb11, https://gerrit.cloudera.org/#/c/13737) introduced. Compared to that change, only bit-packing.inline.h is different. The tests went into the buffer overflow path but it only produced an error in the ASAN builds. Increasing the allowed bit width for bit packing and bit unpacking to 64 bits. This will be needed to support the Parquet delta encoding. Added new methods to BitWriter and BatchedBitReader handling Uleb and ZigZag integers for 64 bits, also needed by delta encoding. The performance of bit unpacking is either the same or better with the new implementation, depending on bit width. Updated the results in bit-packing-benchmark.cc. Testing: - Modified bit packing and unpacking tests to test bit widths up to 64 bits. - Tests covering the additions in BitWriter and BatchedBitReader. Change-Id: Ib9276ca291e3d36db16f63f8caf3e9248a18d85b --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/util/CMakeLists.txt M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/bit-stream-utils.inline.h M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 10 files changed, 430 insertions(+), 209 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/13809/9 -- To view, visit http://gerrit.cloudera.org:8080/13809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib9276ca291e3d36db16f63f8caf3e9248a18d85b Gerrit-Change-Number: 13809 Gerrit-PatchSet: 9 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/13807/10/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/10/be/src/util/vectorised_bit_unpacking_generator.py@184 PS10, Line 184: = flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 10 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 09:08:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 10: (4 comments) http://gerrit.cloudera.org:8080/#/c/13807/4/be/src/util/bit-packing-vectorized.h File be/src/util/bit-packing-vectorized.h: PS4: > Ahh I see. I don't think it should be too bad, we have similar CMake rules, I'll discuss it with other team members also. The generated file now includes a comment with the exact command line that was used to generate it. http://gerrit.cloudera.org:8080/#/c/13807/8/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/8/be/src/util/vectorised_bit_unpacking_generator.py@1540 PS8, Line 1540: > I'm included to remove this option and pick one alternative as the canonica I've removed the option and we always use clang-format. Using clang-format, the generated file conforms to the coding standard automatically and we don't have to take it into account in the code generating script. http://gerrit.cloudera.org:8080/#/c/13807/8/be/src/util/vectorised_bit_unpacking_generator.py@1544 PS8, Line 1544: stdout=subprocess.PIPE) > If we're going with clang-format, I think we should write the non-formatted I used pipes to avoid overwriting the file and using temporary files. http://gerrit.cloudera.org:8080/#/c/13807/8/be/src/util/vectorised_bit_unpacking_generator.py@1547 PS8, Line 1547: output_file.write(clang_format_process_result.stdout) > I think the script should just fail if clang_format was enabled and couldn' The script fails if it cannot find clang-format. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 10 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 09:07:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 9 files changed, 5,962 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/10 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 10 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3940/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 08:16:02 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13740/7//COMMIT_MSG Commit Message: PS7: I had a couple of higher-level thoughts: * It looks like the estimates are in some cases higher on the coordinator than executors, I think because of the padding. I think this is probably mostly OK - there's definitely more overhead on the coordinator if the fragments are tiny. * We probably want a "chicken bit" to fall back to the old behaviour in case there are regressions. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 07:39:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has uploaded a new patch set (#7) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. [WIP] IMPALA-7486: Prototype for admitting less mem on coord This patch computes two memory estimates in the frontend: an estimate for any host that is an executor (including a combined coordinator and executor) and an estimate for a dedicated coordinator. This is computed regardless of whether it is a dedicated coordinator or not. Admission control then, in the case when the coordinator is dedicated, uses the coordinator memory estimate for the coordinator node and the executor memory estimate for all other nodes. Complete: - Calculate the mem needed for the coord fragment + accounting for the runtime filter agg mem required + a safety buffer of 100MB - changes to get the mem_to_admit/limit for coord, accesible through the QuerySchedule - add/remove/modify checks in both reject and admit methods to seperate the coord and executor checks - Update memory accounting in admission controller - use coord's mem limit for the coord queryState - added BE unit tests for most cases - Updated the Admission debug page to show the coord mem values TODO: - add e2e tests: - Simple case (done) - query with num_node = 1 - Any additional test cases not covered by BE unit tests or need an equivalent e2e test - e2e tests are not written with dedicated coord, so all should work fine Testing: * Added unit test for admission/rejection in dedicated coordinator clusters. Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 --- M be/src/runtime/coordinator.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_admission_controller.py M www/admission_controller.tmpl 27 files changed, 2,172 insertions(+), 1,318 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/7 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 7: Fixed most of the test failures. -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 07:36:58 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7486: Prototype for admitting less mem on coord
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4631/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Jul 2019 07:36:49 + Gerrit-HasComments: No
