[ https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bibin A Chundatt updated YARN-3627: ----------------------------------- [~ywskycn] Thank you for discussion. Adding his point also in comments Set MaxResource equal to cluster capacity (instead of <0,0>) is not practical in real clusters. For example, if we only have two queues, one (Q1) for ad-hoc jobs, and another one (Q2) for production jobs. If we let Q1 and Q2 take over the entire cluster, one possible here: at a given time, Q1 takes 100%, but we have some critical production jobs coming in to Q2. In that case, Q2 needs to wait a while for preempting some resources from Q1. It takes some times. So mostly, we don't set the maximum resource for each queue to 100%, but 70-80% instead, to leave some space. > Preemption not triggered in Fair scheduler when maxResources is set on parent > queue > ----------------------------------------------------------------------------------- > > Key: YARN-3627 > URL: https://issues.apache.org/jira/browse/YARN-3627 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler > Environment: Suse 11 SP3, 2 NM > Reporter: Bibin A Chundatt > > Consider the below scenario of fair configuration > > Root (10Gb cluster resource) > --Q1 (maxResources 4gb) > ----Q1.1 (maxResources 4gb) > ----Q1.2 (maxResources 4gb) > --Q2 (maxResources 6GB) > > No applications are running in Q2 > > Submit one application with to Q1.1 with 50 maps & 4Gb gets allocated to Q1.1 > Now submit application to Q1.2 the same will be starving for memory always. > > Preemption will never get triggered since > yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the > cluster utilization is below .8. > > *Fairscheduler.java* > {code} > private boolean shouldAttemptPreemption() { > if (preemptionEnabled) { > return (preemptionUtilizationThreshold < Math.max( > (float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(), > (float) rootMetrics.getAllocatedVirtualCores() / > clusterResource.getVirtualCores())); > } > return false; > } > {code} > Are we supposed to configure in running cluster maxResources <0mb and 0 > cores > so that all queues can take full cluster resources always if > available?? -- This message was sent by Atlassian JIRA (v6.3.4#6332)