Bouncing this thread again. Any other thoughts please? On 17 September 2015 at 23:21, Laxman Ch <[email protected]> wrote:
> No Naga. That wont help. > > I am running two applications (app1 - 100 vcores, app2 - 100 vcores) with > same user which runs in same queue (capacity=100vcores). In this scenario, > if app1 triggers first occupies all the slots and runs longs then app2 will > starve longer. > > Let me reiterate my problem statement. I wanted "to control the amount of > resources (vcores, memory) used by an application SIMULTANEOUSLY" > > On 17 September 2015 at 22:28, Naganarasimha Garla < > [email protected]> wrote: > >> Hi Laxman, >> For the example you have stated may be we can do the following things : >> 1. Create/modify the queue with capacity and max cap set such that its >> equivalent to 100 vcores. So as there is no elasticity, given application >> will not be using the resources beyond the capacity configured >> 2. yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent so >> that each active user would be assured with the minimum guaranteed >> resources . By default value is 100 implies no user limits are imposed. >> >> Additionally we can think of* >> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" >> *which >> will enforce strict cpu usage for a given container if required. >> >> + Naga >> >> On Thu, Sep 17, 2015 at 4:42 PM, Laxman Ch <[email protected]> wrote: >> >>> Yes. I'm already using cgroups. Cgroups helps in controlling the >>> resources at container level. But my requirement is more about controlling >>> the concurrent resource usage of an application at whole cluster level. >>> >>> And yes, we do configure queues properly. But, that won't help. >>> >>> For example, I have an application with a requirement of 1000 vcores. >>> But, I wanted to control this application not to go beyond 100 vcores at >>> any point of time in the cluster/queue. This makes that application to run >>> longer even when my cluster is free but I will be able meet the guaranteed >>> SLAs of other applications. >>> >>> Hope this helps to understand my question. >>> >>> And thanks Narasimha for quick response. >>> >>> On 17 September 2015 at 16:17, Naganarasimha Garla < >>> [email protected]> wrote: >>> >>>> Hi Laxman, >>>> Yes if cgroups are enabled and " >>>> yarn.scheduler.capacity.resource-calculator" configured to >>>> DominantResourceCalculator then cpu and memory can be controlled. >>>> Please Kindly furhter refer to the official documentation >>>> http://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html >>>> >>>> But may be if say more about problem then we can suggest ideal >>>> configuration, seems like capacity configuration and splitting of the queue >>>> is not rightly done or you might refer to Fair Scheduler if you want more >>>> fairness for container allocation for different apps. >>>> >>>> On Thu, Sep 17, 2015 at 4:10 PM, Laxman Ch <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> In YARN, do we have any way to control the amount of resources >>>>> (vcores, memory) used by an application SIMULTANEOUSLY. >>>>> >>>>> - In my cluster, noticed some large and long running mr-app occupied >>>>> all the slots of the queue and blocking other apps to get started. >>>>> - I'm using Capacity schedulers (using hierarchical queues and >>>>> preemption disabled) >>>>> - Using Hadoop version 2.6.0 >>>>> - Did some googling around this and gone through configuration docs >>>>> but I'm not able to find anything that matches my requirement. >>>>> >>>>> If needed, I can provide more details on the usecase and problem. >>>>> >>>>> -- >>>>> Thanks, >>>>> Laxman >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Laxman >>> >> >> > > > -- > Thanks, > Laxman > -- Thanks, Laxman
