[
https://issues.apache.org/jira/browse/TEZ-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CruiseCheung updated TEZ-4260:
------------------------------
Description:
When tasks in two vertice require differrent memory size containers and
tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign
unfittable containers to tasks, cause out of memory error.
Drill down it, we found
[https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110]
should call hc.canFit() method
{code:java}
if (hc.canFit(request.getCapability()) && csig == null ||
signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
{code}
was:
When tasks in two vertice require differrent memory size containers and
tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign
unfittable containers to tasks, cause out of memory error.
Drill down it, we found
[https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110]
should call hc.canFit() method
{code:java}
if (hc.canFit(request.getCapability()) && csig == null ||
signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
{code}
> DagAwareYarnTaskScheduler may assign unfittable container to task
> -----------------------------------------------------------------
>
> Key: TEZ-4260
> URL: https://issues.apache.org/jira/browse/TEZ-4260
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.2
> Reporter: CruiseCheung
> Priority: Minor
>
> When tasks in two vertice require differrent memory size containers and
> tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign
> unfittable containers to tasks, cause out of memory error.
> Drill down it, we found
> [https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110]
> should call hc.canFit() method
> {code:java}
> if (hc.canFit(request.getCapability()) && csig == null ||
> signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
