[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630188#comment-16630188
]
Antal Bálint Steinbach commented on YARN-8468:
----------------------------------------------
Hi [~cheersyang], [~haibochen] ,
I spent some hours to dig deeper and try to do this change. I would say it is
partly possible to do it. Let me tell you my findings and decide based on that.
Sorry, for the long post.
There are 2 things here.
h3. *Normalization on 2 levels*
as Weiwei called it (*#1*) RMAppManager({{ApplicationClientProtocol}}),
DefaultAMSProcessor and (*#2*) {{FS/CS#allocate}}
It turned out that it makes sense to do both because they are doing slightly
different things. I found this after I removed #2 and there were some tests
failing which were building unconsciously on this feature.
#1 _RMServerUtils.normalizeAndValidateRequests(...)_
Throws an exception if the allocation is higher then the allowed one
#2 _ask.setCapability(SchedulerUtils.getNormalizedResource(...))_
Sets the required allocation to the allowed maximum if it is higher than the
allowed maximum or increments the required allocation to be higher than the
minimum allocation if it is lower then the minimum allocation.
The solution can be to merge them into level #1 as you suggested but then we
have to figure out if we would like to keep the exception throwing or the
fixing behavior or both (throw on higher, fix on lower)
h3. *Protect Yarnscheduler* api
I would say we cannot do this.
* We have to calculate maxResourceAllocation for
_SchedulerUtils.getNormalizedResource(...)_ for that we need the queueName.
There is no workaround for that.
* We would like to calculate maxResourceAllocation once per allocate(),
therefore we have to calculate it before iterating over asks and do
normalization like here (RMAppManager):
{code:java}
Resource maxAllocation = scheduler.getMaximumResourceCapability(queue);
for (ResourceRequest amReq : amReqs) {
SchedulerUtils.normalizeAndValidateRequest(amReq, maxAllocation,
queue, scheduler, isRecovery, rmContext, null);
amReq.setCapability(scheduler.getNormalizedResource(
amReq.getCapability(), maxAllocation));{code}
If I understand correctly your suggestion was to skip
_scheduler.getNormalizedResource_ and use directly the static
_SchedulerUtils.getNormalizedResource(...)_ method. In this way, we don't have
to change YarnScheduler API because we don't have to pass around
maxResourceAllocation.
The problem with this that scheduler.getNormalizedResource is overridden in
FairScheduler.
FS:
{code:java}
@Override
public Resource getNormalizedResource(Resource requestedResource,
Resource maxResourceCapability) {
return SchedulerUtils.getNormalizedResource(requestedResource,
DOMINANT_RESOURCE_CALCULATOR,
minimumAllocation,
maxResourceCapability,
incrAllocation);
}{code}
AbstractYarnScheduler:
{code:java}
@Override
public Resource getNormalizedResource(Resource requestedResource,
Resource maxResourceCapability) {
return SchedulerUtils.getNormalizedResource(requestedResource,
getResourceCalculator(),
getMinimumResourceCapability(),
maxResourceCapability,
getMinimumResourceCapability());
}{code}
This means that if I would like to call
_SchedulerUtils.getNormalizedResource(...)_ from for example RMAppManager I
still need the scheduler to get parameters like this:
{code:java}
Resource maxAllocation = scheduler.getMaximumResourceCapability(queue);
for (ResourceRequest amReq : amReqs) {
SchedulerUtils.normalizeAndValidateRequest(amReq, maxAllocation,
queue, scheduler, isRecovery, rmContext, null);
Resource normalizedCapability = SchedulerUtils.getNormalizedResource(
amReq.getCapability(), scheduler.getResourceCalculator(),
scheduler.getMinimumResourceCapability(), maxAllocation,
scheduler.getIncrementAllocation());
amReq.setCapability(normalizedCapability);
}{code}
This means I have to introduce a new method in the YarnScheduler API called
getIncrementAllocation, which is against our starting point. Also, we can
delete the _getNormalizedResource_ method because it will be never used, that
is another change in the API.
Furthermore, _scheduler.getNormalizedResource_ is used in several tests as a
mocked method. If we replace that call to a static method call we will be in
trouble in those tests also.
> Enable the use of queue based maximum container allocation limit and
> implement it in FairScheduler
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Antal Bálint Steinbach
> Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch,
> YARN-8468.014.patch, YARN-8468.015.patch, YARN-8468.016.patch,
> YARN-8468.017.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
> The goal of this ticket is to allow this value to be set on a per queue basis.
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
> Suggested solution:
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability(String queueName) in both
> FSParentQueue and FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * Enforce the use of queue based maximum allocation limit if it is
> available, if not use the general scheduler level setting
> ** Use it during validation and normalization of requests in
> scheduler.allocate, app submit and resource request
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
