Konstantinos Karanasos commented on YARN-8250:

Just went through your comments. I am skeptical about adding a new 
implementation of the Container Scheduler, as [~asuresh] and [~leftnoteasy] 
also pointed out.

It seems we can achieve the required behavior just with some new 
policies/functions rather than a new implementation.

>From what I understand, the main points are (1) how to start guaranteed 
>containers (do we start them and then kill opportunistic, or do we kill 
>opportunistic and then start them) and (2) how/when to start opportunistic 
>containers (at regular intervals or when another container finishes).

*How to start guaranteed:*

For (1), just to make sure I understand: why can't we know the resources 
utilized by each opportunistic container? But even if we don't know, can't we 
just use the allocated resources of the container in our calculation? I 
understand that this might lead to more opportunistic containers killed than 
might be needed, but for sure it will be a good first version (and much better 
than what we have today). At the same time, if we are not very aggressive with 
starting opportunistic containers, it should not lead to too many containers 

So my take is to have a first version that does the starting as it is today 
(first queue, then kill), even if that leads to more opportunistic containers 
killed. And be less aggressive in starting opportunistic for that matter.

*How to start opportunistic:*

Why not keep existing behavior and add the new one too? That is, we still let 
opportunistic start when there is a container finish event, but do so not with 
utilized but allocated resources. This will keep the current behavior. On top 
of that, if over-allocation is enabled, we can have a thread that every so 
often checks utilized resources and starts opportunistic containers on 
over-allocated resources.



> Create another implementation of ContainerScheduler to support NM 
> overallocation
> --------------------------------------------------------------------------------
>                 Key: YARN-8250
>                 URL: https://issues.apache.org/jira/browse/YARN-8250
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>            Priority: Major
>         Attachments: YARN-8250-YARN-1011.00.patch, 
> YARN-8250-YARN-1011.01.patch, YARN-8250-YARN-1011.02.patch
> YARN-6675 adds NM over-allocation support by modifying the existing 
> ContainerScheduler and providing a utilizationBased resource tracker.
> However, the implementation adds a lot of complexity to ContainerScheduler, 
> and future tweak of over-allocation strategy based on how much containers 
> have been launched is even more complicated.
> As such, this Jira proposes a new ContainerScheduler that always launch 
> guaranteed containers immediately and queues opportunistic containers. It 
> relies on a periodical check to launch opportunistic containers. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to