[
https://issues.apache.org/jira/browse/TEZ-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372361#comment-14372361
]
Gopal V commented on TEZ-2217:
------------------------------
The performance bug occurs within the queue, that is a straight-forward failure
to utilize available resources properly.
I can work-around this issue by upping the max-delay for releasing containers,
but that's where the pre-emption scenario is relevant - that approach isn't
good on a busy cluster with a lot of applications. I don't want to hurt the
performance or concurrency considerations.
The AM-RM allocate responses contain the pre-emption messages, which should be
a good enough indicator that a certain fraction of currently held resources
will be removed soon.
The pre-emption period is the gap between that event and the event of
termination, in which period the min-held rules will not be enforced - to
protect the ones actually doing work, idle containers should be allowed to
expire (still following the min-max delay curve).
I suspect the pre-emption contracts do not expose that sunset period for the
ill-fated containers, which makes it slightly harder to do this directly off
the message - perhaps there is one I can't find ?
That makes the system both performant on idle cluster, but handles the
mid-query bottlenecking that is common on busy clusters - particularly if the
container expiry is smaller than the sunset period for pre-empted containers.
> The min-held-containers constraint is not enforced during query runtime
> ------------------------------------------------------------------------
>
> Key: TEZ-2217
> URL: https://issues.apache.org/jira/browse/TEZ-2217
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Gopal V
> Assignee: Bikas Saha
> Attachments: TEZ-2217.txt.bz2
>
>
> The min-held containers constraint is respected during query idle times, but
> is not respected when a query is actually in motion.
> The AM releases unused containers during dag execution without checking for
> min-held containers.
> {code}
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing
> container, containerId=container_1424502260528_1348_01_000013,
> containerExpiryTime=1426891313264, idleTimeoutMin=5000
> 2015-03-20 15:41:53,475 INFO [DelayedContainerManager]
> rm.YarnTaskSchedulerService: Releasing unused container:
> container_1424502260528_1348_01_000013
> {code}
> This is actually useful only after the AM has received a soft pre-emption
> message, doing it on an idle cluster slows down one of the most common query
> patterns in BI systems.
> {code}
> create temporary table smalltable as ...;
> select ... bigtable JOIN smalltable ON ...;
> {code}
> The smaller query in the beginning throws away the pre-warmed capacity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
