[
https://issues.apache.org/jira/browse/YARN-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadayuki Furuhashi updated YARN-4039:
-------------------------------------
Attachment: YARN-4039.2.patch
> New AM instances waste resource by waiting only for resource availability
> when all available resources are already used
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-4039
> URL: https://issues.apache.org/jira/browse/YARN-4039
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0
> Reporter: Sadayuki Furuhashi
> Assignee: Sadayuki Furuhashi
> Attachments: YARN-4039.1.patch, YARN-4039.2.patch
>
>
> Problem:
> In FairScheduler, maxRunningApps doesn't work well if we can't predict size
> of an application in a queue because small maxRunningApps can't use all
> resources if many small applications are issued, where large maxRunningApps
> wastes resources if large applications run.
> Background:
> We're using FairScheduler. In following scenario, AM instances wastes
> resources significantly:
> * A queue has X MB of capacity.
> * An application requests 32 containers where a container requires (X / 32)
> MB of memory
> ** In this case, a single application occupies entire resource of the queue.
> * Many those applications are issued (10 applications)
> * Ideal behavior is that applications run one by one to maximize throughput.
> * However, all applications run simultaneously. As the result, AM instances
> occupy resources and prevent other tasks from starting. At worst case, most
> of resources are occupied by waiting AMs and applications progress very
> slowly.
> A solution is setting maxRunningApps to 1 or 2. However, it doesn't work well
> if following workload exists at the same queue:
> * An application requests 2 containers where a container requires (X / 32) MB
> of memory
> * Many those applications are issued (say, 10 applications)
> * Ideal behavior is that all applications run simultaneously to maximize
> concurrency and throughput.
> * However, number of applications are limited by maxRunningApps. At worst
> case, most of resources are idling.
> This problem happens especially with Hive because we can't estimate size of a
> MapReduce application.
> Solution:
> AM doesn't have to start if there are waiting resource requests because the
> AM can't grant resource requests even if it starts.
> Patch:
> I attached a patch that implements this behavior. But this implementation has
> this trade-off:
> * When AM is registered to FairScheduler, its demand is 0 because even AM
> attempt is not created. Starting this AM doesn't change resource demand of a
> queue. So, if many AMs are issued to a queue at the same time, all AMs will
> be RUNNING. But we want to prevent it.
> * When a AM starts, demand of the AM is only AM attempt. Then AM requires
> more resources. Until AM requires resources, demand of the queue is low. But
> starting AM during this time will start unnecessary AMs.
> * So, this patch doesn't start immediately when AM is registered. Instead, it
> starts AM only every continuous-scheduling-sleep-ms.
> * Setting large continuous-scheduling-sleep-ms will prevent wasting AMs. But
> increases latency.
> Therefore, this patch is enabled only if new option "demand-block-am-enabled"
> is true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
