Vinod Kumar Vavilapalli created YARN-4724:
---------------------------------------------
Summary: [Umbrella] Recognizing services: Special handling of
preemption, container reservations etc.
Key: YARN-4724
URL: https://issues.apache.org/jira/browse/YARN-4724
Project: Hadoop YARN
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
See overview doc at YARN-4692, copying the sub-section to track all related
efforts.
{quote}
Though it is desirable to not special-case services anywhere in YARN, there
are a few key areas where such special recognition is not only unavoidable but
very necessary. For example, preemption and reservation of long running
containers have different implications from regular ones.
Preemption of resources in YARN today works by killing of containers.
Obviously, preempting long running containers is different and costlier for the
apps. For many long-lived applications, preemption by killing will likely be
not even an option that they can tolerate.
[Task] Preemption also means that scheduler should avoid allocating long
running containers on borrowed resources.
On the other hand, today’s scheduler creates reservations when they cannot fit
a container on a machine with free resources. When making such reservations for
containers of long running services, the scheduler shouldn’t queue the
reservation behind other services running on a node otherwise the reservation
may get stuck unfulfilled forever.
[Task] Preemption and reservations logic thus need to understand if an
application has long running containers and make decisions accordingly.
[Task] There is an existing JIRA YARN-1039 (Add parameter for YARN resource
requests to indicate "long lived") which was filed to address some of this
special recognition of service containers. The options were between a boolean
flag and a long representing the lifecycle, though in practice I think we will
need to have both.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
