[
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253273#comment-16253273
]
Anup Tiwari commented on YARN-4781:
-----------------------------------
Hi [~sunilg]/ [~leftnoteasy],
I think i am also getting above issue in our scenario like one big query is
taking all resources of a queue lets say Q1. And when i am launching another
query in Q1, almost always it is hanging in ACCEPTED state and sometime in
RUNNING state but just got 1 container but not progressing. We are using Apache
Hadoop Distribution version 2.8.0 and Apache Hive 2.1.1.
After reading this JIRA, as i understood we are not yet supporting "fair"
ordering policy within same queue.
But i am bit confused in what exactly
"yarn.scheduler.capacity.<queue-path>.ordering-policy" property does when set
to "fair".
As i can read on net above property used to specify ordering policies on a
per-queue basis, set the following property to fifo or fair. The default
setting is fifo.
Can you or anyone explain how capacity scheduler works when orderin-policy set
to "fair" with example?
I have also gone through https://issues.apache.org/jira/browse/YARN-3463 but
didn't understood.
[~milesc], Can you bit explain your solution? Because i think we would end
creating too many queues and also how each queue uses all resources of cluster
when nothing is running? And also as far as i know, if we have to remove
existing queues then it can't be done on runtime, it requires yarn restart. Let
me know if i am missing anything.
Thanks in advance.
> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>
> Key: YARN-4781
> URL: https://issues.apache.org/jira/browse/YARN-4781
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> We introduced fairness queue policy since YARN-3319, which will let large
> applications make progresses and not starve small applications. However, if a
> large application takes the queue’s resources, and containers of the large
> app has long lifespan, small applications could still wait for resources for
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to
> preempt resources of queue with fairness policy enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
