[ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253273#comment-16253273 ]
Anup Tiwari commented on YARN-4781: ----------------------------------- Hi [~sunilg]/ [~leftnoteasy], I think i am also getting above issue in our scenario like one big query is taking all resources of a queue lets say Q1. And when i am launching another query in Q1, almost always it is hanging in ACCEPTED state and sometime in RUNNING state but just got 1 container but not progressing. We are using Apache Hadoop Distribution version 2.8.0 and Apache Hive 2.1.1. After reading this JIRA, as i understood we are not yet supporting "fair" ordering policy within same queue. But i am bit confused in what exactly "yarn.scheduler.capacity.<queue-path>.ordering-policy" property does when set to "fair". As i can read on net above property used to specify ordering policies on a per-queue basis, set the following property to fifo or fair. The default setting is fifo. Can you or anyone explain how capacity scheduler works when orderin-policy set to "fair" with example? I have also gone through https://issues.apache.org/jira/browse/YARN-3463 but didn't understood. [~milesc], Can you bit explain your solution? Because i think we would end creating too many queues and also how each queue uses all resources of cluster when nothing is running? And also as far as i know, if we have to remove existing queues then it can't be done on runtime, it requires yarn restart. Let me know if i am missing anything. Thanks in advance. > Support intra-queue preemption for fairness ordering policy. > ------------------------------------------------------------ > > Key: YARN-4781 > URL: https://issues.apache.org/jira/browse/YARN-4781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler > Reporter: Wangda Tan > Assignee: Wangda Tan > > We introduced fairness queue policy since YARN-3319, which will let large > applications make progresses and not starve small applications. However, if a > large application takes the queue’s resources, and containers of the large > app has long lifespan, small applications could still wait for resources for > long time and SLAs cannot be guaranteed. > Instead of wait for application release resources on their own, we need to > preempt resources of queue with fairness policy enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org