Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14058 )
Change subject: IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator ...................................................................... IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator This patch fixes a bug where if an insert or CTAS query has no fragments scheduled on the coordinator and a mem limit is to be enforced on the query (either through query option or automatically through estimates) then the same limit is also applied to the coordinator backend even though it does not execute anything. Highlights: - coord_backend_mem_to_admit_/mem_limit will always refer to the memory to admit/limit for the coordinator regardless of which fragments are scheduled on it. - There will always be a BackendExecParams added for the coordinator because coordinator always spawns a QueryState object with a mem_tracker for tracking runtime filter mem and the result set cache. For the case where this BackendExecParams is empty (no instances scheduled) it would ensure that some minimal amount of memory is accounted for by the admission controller and the right mem limit is applied to the QueryState spawned by the coordinator - added changes to Coordinator and Coordinator::BackendState classes to handle an empty BackendExecParams object Testing: The following cases need to be tested where the kind of fragments schduled on the coordinator backend are: 1. Coordinator fragment + other exec fragments 2. Coordinator fragment only 3. other exec fragments only (eg. insert into values OR insert into select 1) 4. No fragments, but coordinator still creates a QueryState Case 1 is covered by tests working with non-dedicated coordinators. Rest are covered by test_dedicated_coordinator_planner_estimates and test_sanity_checks_dedicated_coordinator in test_admission_controller.py Change-Id: Ic4fba02bb7b20553a20634f8c5989d43ba2c0721 Reviewed-on: http://gerrit.cloudera.org:8080/14058 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-query/queries/QueryTest/dedicated-coord-mem-estimates.test M tests/custom_cluster/test_admission_controller.py 13 files changed, 167 insertions(+), 125 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic4fba02bb7b20553a20634f8c5989d43ba2c0721 Gerrit-Change-Number: 14058 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
