[
https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942214#comment-13942214
]
Carlo Curino commented on YARN-1051:
------------------------------------
Arun, I think the current design point is closer to what you describe than it
seems (we changed a fair bit from the early conversation we had).
We created two new types of queues: InvetoryQueue and SessionQueue, which
respectively inherit form ParentQueue and LeafQueue...
The distribution of resources among SessionQueues performed by the
InventoryQueue (inherited by ParentQueue, but without a requirement
of capacity of children to sum-up to 100%) corresponds very much to the
priority mechanism you refer to (a SessionQueue with high nominal
capacity and low utilization is favored, etc.). So in principle, we could
change InventoryQueue to track apps directly with a priority list.
On the other hand, since we envision each SessionQueue to potentially being
used to submit multiple jobs (change of perspective from early
design, e.g., hive, pig multi-job queries, or pipelines), inheriting from the
LeafQueue and imposing the classic FIFO internal behavior + delay
scheduling + other limits seems, allows to:
1) make the notion of "Session" to be a rather consistent extension of a
"Queue", a session is a queue with some time-evolving properties (e.g.,
capacity).
2) reuse lots of tracking structures and well tested code.
Supporting this notion of session by assigning individual priorities to jobs
that share a session, and having multiple sessions per inventory seems harder
to get right and maintain.
We would also be happy to meet and talk this through, and then report in JIRA
the result of our conversation.
> YARN Admission Control/Planner: enhancing the resource allocation model with
> time.
> ----------------------------------------------------------------------------------
>
> Key: YARN-1051
> URL: https://issues.apache.org/jira/browse/YARN-1051
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler, resourcemanager, scheduler
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf,
> techreport.pdf
>
>
> In this umbrella JIRA we propose to extend the YARN RM to handle time
> explicitly, allowing users to "reserve" capacity over time. This is an
> important step towards SLAs, long-running services, workflows, and helps for
> gang scheduling.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
