[
https://issues.apache.org/jira/browse/TEZ-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949368#comment-14949368
]
Jason Lowe commented on TEZ-2872:
---------------------------------
This is similar to a scenario MapReduce encountered before, see MAPREDUCE-4730.
One could argue the problem can be solved by fixing HADOOP-8942 since the lack
of flow control in the RPC layer is what allows the responses to accumulate
without limit and hit the OOM. While that would prevent the OOM condition, it
would leave the AM in a state where its outbound network is saturated, unable
to respond properly to other RPC requests. Not exactly an ideal scenario from
the perspective of others trying to talk to the AM.
We may want to consider adding a rate throttle to getTask processing for
situations like this. It would be nice if we could cheaply calculate the size
of the ContainerTask response and only send out some configurable MB/sec of
responses, but ContainerTask is a writable, not a protobuf, so we don't know
the size of it until we actually serialize it to a buffer in the depths of the
RPC layer. Maybe we could scrape the ContainerTask for all the additional
resources and user payloads squirreled away everywhere and do a quick
guesstimate on the total size as a proxy. Or we could just do a simple RPC
count and only allow so many per second.
> Tez AM can be overwhelmed by TezTaskUmbilicalProtocol.getTask responses
> -----------------------------------------------------------------------
>
> Key: TEZ-2872
> URL: https://issues.apache.org/jira/browse/TEZ-2872
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jason Lowe
>
> When a large job runs on a large cluster with a large user payload then the
> AM can end up hitting OOM conditions. For example, Pig-on-Tez can require a
> significant user payload (approaching 1MB) for vertices, inputs, and outputs
> in the DAG. This can cause the ContainerTask response to be rather large per
> task, which can lead to a situation where the AM is generating output faster
> than the network interface can process it. If there are enough containers
> asking for tasks then this leads to an OOM condition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
