[
https://issues.apache.org/jira/browse/IMPALA-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699507#comment-17699507
]
ASF subversion and git services commented on IMPALA-11858:
----------------------------------------------------------
Commit c810c51fa70e2718df76f91c861ebe3ff07da428 in impala's branch
refs/heads/master from Abhishek Rawat
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c810c51fa ]
IMPALA-11858: Cap per backend memory estimate to its memory limit for admission
Admission controller caps memory estimates for a given query to
its physical memory. The memory estimates should instead be capped to
the backend's memory limit for admission, which is computed during
daemon initialization in ExecEnv::Init().
With this patch, for a given query schedule, the Coordinator backend's
memory limit is used for capping memory to admit on coordinator and min
of all executor backend's memory limit is used for capping mem to admit
on executors. A config option 'clamp_query_mem_limit_backend_mem_limit'
is also added to revert to the old behavior where queries requesting
more memory than backend's admission limit get rejected.
The memory requested by a query when MEM_LIMIT or MEM_LIMIT_EXECUTORS is
set is also capped to the memory limit for admission on the backends.
Also fixed the issue related to excessive logging in query profiles
when using global admission controller. If the query was queued
the remote admission controller client was logging 'Queued' status in
profile every time it checked the query status and it hadn't changed.
Testing:
- Updated existing unit tests in admission-controller-test.cc
- Added new checks in existing tests in executor-group-test.cc
- Updated custom_cluster tests in test_admission_controller.py
- Ran exhaustive tests
Change-Id: I3b1f6e530785ef832dbc831d7cc6793133f3335c
Reviewed-on: http://gerrit.cloudera.org:8080/19533
Reviewed-by: Abhishek Rawat <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> admissiond incorrectly caps memory limit to its process memory
> --------------------------------------------------------------
>
> Key: IMPALA-11858
> URL: https://issues.apache.org/jira/browse/IMPALA-11858
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Assignee: Abhishek Rawat
> Priority: Critical
>
> When admission controller is running as a separate daemon it incorrectly caps
> memory limit for the query to its process limit. This is also incorrect
> behavior when admission controller is running in coordinator as executors
> could have different memory limit compared to coordinator.
> https://github.com/apache/impala/blob/master/be/src/scheduling/schedule-state.cc#L312#L313
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
