[
https://issues.apache.org/jira/browse/YARN-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081242#comment-15081242
]
Lei Guo commented on YARN-3870:
-------------------------------
I am not against to combine this JIRA and YARN-371, There is common ground
between these two JIRAs. And more likely the final technical solution will be
single solution to cover both, though it's not necessary. Maybe we can view
YARN-371 as a technical speculation and YARN-3870 as one related use case (if
YARN-371 is resolved, YARN-3870 should be covered).
>From another angle, YARN-3870 could be resolved via approaches without ID. The
>scheduling is more care about the current snapshot of resource requests from
>applications. It's not mandatory to have the ID, as long as the snapshot can
>provide detailed resource request information, scheduler can do fine
>scheduling. The ID will mainly help to prevent/handle issues from asynchronous
>protocol.
> Providing raw container request information for fine scheduling
> ---------------------------------------------------------------
>
> Key: YARN-3870
> URL: https://issues.apache.org/jira/browse/YARN-3870
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications, capacityscheduler, fairscheduler,
> resourcemanager, scheduler, yarn
> Reporter: Lei Guo
> Assignee: Karthik Kambatla
>
> Currently, when AM sends container requests to RM and scheduler, it expands
> individual container requests into host/rack/any format. For instance, if I
> am asking for container request with preference "host1, host2, host3",
> assuming all are in the same rack rack1, instead of sending one raw container
> request to RM/Scheduler with raw preference list, it basically expand it to
> become 5 different objects with host1, host2, host3, rack1 and any in there.
> When scheduler receives information, it basically already lost the raw
> request. This is ok for single container request, but it will cause trouble
> when dealing with multiple container requests from the same application.
> Consider this case:
> 6 hosts, two racks:
> rack1 (host1, host2, host3) rack2 (host4, host5, host6)
> When application requests two containers with different data locality
> preference:
> c1: host1, host2, host4
> c2: host2, host3, host5
> This will end up with following container request list when client sending
> request to RM/Scheduler:
> host1: 1 instance
> host2: 2 instances
> host3: 1 instance
> host4: 1 instance
> host5: 1 instance
> rack1: 2 instances
> rack2: 2 instances
> any: 2 instances
> Fundamentally, it is hard for scheduler to make a right judgement without
> knowing the raw container request. The situation will get worse when dealing
> with affinity and anti-affinity or even gang scheduling etc.
> We need some way to provide raw container request information for fine
> scheduling purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
