[
https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407294#comment-13407294
]
Robert Joseph Evans commented on MAPREDUCE-4327:
------------------------------------------------
Andrew, Sorry this has taken me so long to get to this. Thanks to both you and
Arun for taking this up. It is something that is going to be really great when
it is done.
I have a few comments on the code.
# ResourceComparator.java needs to have the Apache License Header in it.
# I don't really like having the resource comparator class configuration be
specific to the scheduler. I would prefer to see it be available for both the
fifo and the capacity scheduler. This is very minor, but it enforces
consistency between the schedulers.
# In a few places like SchedulerNode, we still operate on each of the resources
separately. {code}
this.availableResource.setMemory(node.getTotalCapability().getMemory());
this.availableResource.setCores(node.getTotalCapability().getCores());
{code} It would really be nice to be able to abstract some of that away, like
with the comparisons, so that if we add in new resources in the future, we do
not need to change the code again. (This is also very minor)
# To answer your question about computeSlotMillis, I would say no, but I am
open to others opinions on this. This counter is there to try an maintain
backwards compatibility. It was intended to indicate how much total resources
the job used, i.e. how many milliseconds this job held a slot that no-one else
could use. Because there are no real slots any more, I would prefer to see
this metric deprecated, and replaced with something that breaks it down by the
resources involved. But that is probably for a separate JIRA, because it is a
potentially complex question.
# I would like better protections against someone passing in a 0 or null for
the number of CPU cores in YARN. For MR I see a new default for the number of
CORES being requested, but I don't see an equivalent in just YARN. This is
mostly because CPU cores is being added in and I can see other applications,
like the distributed shell, not being updated, which could result in all kinds
of issues. It would be great that if no CPU request is given we default to 1.
# Could we change the name of ResourceMemoryCpuComparator to something more
like DefaultMultiResourceComparator? I think
ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it
is the direction we are headed in.
# Do we want to be able to schedule only part of a core (make the resource a
float not an int)? For a Map or Reduce task we typically are only going to
want 1 CPU, but some things like the MR AM, unless it is a very big job, 0.5
cores is overkill for what it does.
This is just a cursory look but I like what I see.
To chime in on some of your questions
bq. Are you planning to change the definition of a queue's capacity?
I think this could be something very useful, but should probably be done on a
separate JIRA.
bq. Do you plan to change how spare capacity is allocated?
What do you mean by spare capacity? Do you mean capacity that is not currently
in use? If so I would love to see a patch that does this, so that I can run
gridmix on it both ways and see what the results are.
bq. Are you planning to support priorities or weights within the queues?
I would also like to see something like this happen, but from discussions I
have had in the past, at least for the MRV1 case we would need something like
preemption to be able to avoid some potential deadlocks. I could be wrong
about that here, because the resource allocation now behaves differently in
with respect to a priority, but either way I think that discussion is something
that should be done on a separate JIRA to avoid blocking this coming in.
Thanks again to both you and Arun for doing this.
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-4327
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch,
> MAPREDUCE-4327-v4.patch, MAPREDUCE-4327.patch
>
>
> With YARN being a general purpose system, it would be useful for several
> applications (MPI et al) to specify not just memory but also CPU (cores) for
> their resource requirements. Thus, it would be useful to the
> CapacityScheduler to account for both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
