[
https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414834#comment-15414834
]
Hitesh Sharma edited comment on YARN-5501 at 8/10/16 7:11 AM:
--------------------------------------------------------------
Thanks [~atris] for the comments. These are good points but the answers depend
upon the implementation of the container.
1) Yes, there will be some overhead of maintaining the pooled containers but
that's a trade off to optimize for launch latencies. Containers can however
implement some custom behaviors to lower the overhead. As an e.g. if the
container supports PAUSE and RESUME semantics ([YARN-5292]) then a pooled
container could be PAUSED. Some other container could however chose to resize
the allocation to a minimum and resize as per the actual resource request.
2) I'm not sure if I follow the comment here. Pooled containers are useful to
lower the launch latencies and that's independent of the actual container run
time.
3) That would be implementation specific. A pooled container is in itself a
resource and when acquired by a request would need to be adjusted accordingly.
We will be posting some more design and implementation details which will
hopefully help clarify the ideas here.
was (Author: hrsharma):
Thanks [~atris] for the comments. These are good points but the answers depend
upon the implementation of the container.
1) Yes, there will be some overhead of maintaining the pooled containers but
that's a trade off to optimize for launch latencies. Containers can however
implement some custom behaviors to lower the overhead. As an e.g. if the
container supports PAUSE and RESUME semantics [YARN-5292] then a pooled
container could be PAUSED. Some other container could however chose to resize
the allocation to a minimum and resize as per the actual resource request.
2) I'm not sure if I follow the comment here. Pooled containers are useful to
lower the launch latencies and that's independent of the actual container run
time.
3) That would be implementation specific. A pooled container is in itself a
resource and when acquired by a request would need to be adjusted accordingly.
We will be posting some more design and implementation details which will
hopefully help clarify the ideas here.
> Container Pooling in YARN
> -------------------------
>
> Key: YARN-5501
> URL: https://issues.apache.org/jira/browse/YARN-5501
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Arun Suresh
>
> This JIRA proposes a method for reducing the container launch latency in
> YARN. It introduces a notion of pooling *Unattached Pre-Initialized
> Containers*.
> Proposal in brief:
> * Have a *Pre-Initialized Container Factory* service within the NM to create
> these unattached containers.
> * The NM would then advertise these containers as special resource types
> (this should be possible via YARN-3926).
> * When a start container request is received by the node manager for
> launching a container requesting this specific type of resource, it will take
> one of these unattached pre-initialized containers from the pool, and use it
> to service the container request.
> * Once the request is complete, the pre-initialized container would be
> released and ready to serve another request.
> This capability would help reduce container launch latencies and thereby
> allow for development of more interactive applications on YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
