Hello Andrew Sherman, Lars Volker, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13740
to look at the new patch set (#2).
Change subject: [WIP] IMPALA-7486: Prototype for admitting less mem on coord
......................................................................
[WIP] IMPALA-7486: Prototype for admitting less mem on coord
Complete:
- Calculate the mem needed for the coord fragment + accounting
for the runtime filter agg mem required + a safety buffer of 100MB
- changes to get the mem_to_admit/limit for coord, accesible through
the QuerySchedule
- add/remove/modify checks in both reject and admit methods to
seperate the coord and executor checks
- Update memory accounting in admission controller
- use coord's mem limit for the coord queryState
- added BE unit tests for most cases
- Updated the Admission debug page to show the coord mem values
TODO:
- add e2e tests:
- Simple case (done)
- query with num_node = 1
- Any additional test cases not covered by BE unit tests
or need an equivalent e2e test
- e2e tests are not written with dedicated coord,
so all should work fine
Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
---
M be/src/runtime/coordinator.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-http-handler.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M tests/custom_cluster/test_admission_controller.py
M www/admission_controller.tmpl
14 files changed, 516 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/2
--
To view, visit http://gerrit.cloudera.org:8080/13740
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Gerrit-Change-Number: 13740
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Andrew Sherman <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Lars Volker <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
