Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15961
Change subject: [WIP] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf ...................................................................... [WIP] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf In order to support the new admission control service, we need to be able to return the results of an admission attempt, i.e. the query schedule, to the coordinator. To enable this, this patch moves all parts of the QuerySchedule class and related classes that are required by the coordinator into a new message QuerySchedulePB. The main admission control interface, SubmitForAdmission(), now returns a QuerySchedulePB. This patch is still a work in progress and needs additional clean up before its ready for a full review. Some notable changes: - Previously, QuerySchedule was used by Coordinator as a way to pass around references to parts of the TExecRequest to places like Coordinator::ExecSummary and Coordinator::BackendState. This has been replaced with the ExecParams class, which is a container for references to the TExecRequest and QuerySchedulePB along with convenience functions for accessing them. - Similarly, FragmentExecParams, which is part of QuerySchedule, contains references to the associated TPlanFragment, owned by the TExecRequest, which were used by the Coordinator when iterating over the schedule to initiate the query. Since FragmentExecParamsPB can't contain these references, they were replaced by a map between fragment idx and TPlanFragment in ExecParams. - AdmissionController::ReleaseQuery() and ReleaseQueryBackend() now take a query id as a parameter instead of a QuerySchedule. To facilitate this, AdmissionController now maintains a map from query ids of running queries to the resources that were allocated for them so that it can look the resources up when releasing them. This map will be necessary when implementing the admission control service to facilitate proper accounting of resouces in cases like coordinator failures. - Previously, Scheduler would create the FInstanceExecParams as children of the FragmentExecParams. After completely constructing the FragmentExecParams it would then translate them into the BackendExecParams which had references to the FInstanceExecParams, which are how Coordinator actually consumes the schedule. Since we can't include references in the protobuf, FInstanceExecParamsPBs are now created as children of the BackendExecParamsPBs so that they can be sent back to the Coordinator. This is facilitated by QuerySchedule::GetBackendExecParams() which creates the BackendExecParamsPBs as needed during scheduling. Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 --- M be/src/runtime/CMakeLists.txt M be/src/runtime/coordinator-backend-resource-state.cc M be/src/runtime/coordinator-backend-state-test.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h A be/src/runtime/exec-params.cc A be/src/runtime/exec-params.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-driver.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.h M be/src/util/uid-util.h M common/protobuf/CMakeLists.txt A common/protobuf/admission_control_service.proto 29 files changed, 1,017 insertions(+), 788 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/15961/1 -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall <[email protected]>
