Marcel Kornacker has uploaded a new patch set (#12). Change subject: IMPALA-3902: Scheduler improvements for running multiple fragment instances on a single backend ......................................................................
IMPALA-3902: Scheduler improvements for running multiple fragment instances on a single backend This is an extension of the scheduler and coordinator for multi-threaded execution. It mainly removes the assumption of having one instance per fragment per host. The approach taken here is to create parallel data structures and control flow functions, where necessary, and otherwise to leave the existing single-instance logic in place. The parallel structures' and functions' names are prefixed with "Mt" to facilitate the enventual clean-up. Not much of an attempt was made to factor out common functionality between the Mt- and the single-threaded version, because the single-threaded version will disappear in a follow-on patch and refactoring the existing code to fit into two parallel functions from which it's being called might end up obscuring the code more than helping it. Also, this code is relatively stable and having two parallel paths won't cause much extra work (in terms of having to apply the same changes/fixes twice) in the medium term. Changes to data structures: - QuerySchedule: per-instance and per-fragment structs with complete execution parameters (instead of partially relying on TQueryExecRequest); the per-instance execution parameter struct is a child of the per-fragment parameter struct - explicit fragment id, with range 0..#fragments-1 (instead of relying on an index into an array in TQueryExecRequest) Excluded: - runtime filter handling - anything related to RM Change-Id: I240445199e22f009f4e72fdb8754eb8d77e3d680 --- M be/src/common/global-types.h M be/src/exec/exec-node.cc M be/src/exec/union-node.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/runtime-filter-bank.cc A be/src/scheduling/query-resource-mgr.cc A be/src/scheduling/query-resource-mgr.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.h M be/src/scheduling/simple-scheduler.cc M be/src/scheduling/simple-scheduler.h M be/src/service/fragment-exec-state.cc M be/src/service/fragment-mgr.cc M be/src/service/impala-server.cc M be/src/service/query-exec-state.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/container-util.h M be/src/util/uid-util-test.cc M be/src/util/uid-util.h M common/thrift/ExecStats.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Planner.thrift M common/thrift/Types.thrift M fe/src/main/java/com/cloudera/impala/common/TreeNode.java M fe/src/main/java/com/cloudera/impala/planner/Planner.java M fe/src/main/java/com/cloudera/impala/service/Frontend.java M fe/src/test/java/com/cloudera/impala/planner/PlannerTestBase.java 32 files changed, 2,035 insertions(+), 559 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/4054/12 -- To view, visit http://gerrit.cloudera.org:8080/4054 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I240445199e22f009f4e72fdb8754eb8d77e3d680 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Marcel Kornacker <mar...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Henry Robinson <he...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Marcel Kornacker <mar...@cloudera.com> Gerrit-Reviewer: Matthew Jacobs <m...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com>