[
https://issues.apache.org/jira/browse/TEZ-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143288#comment-15143288
]
Siddharth Seth commented on TEZ-3114:
-------------------------------------
Flow control would definitely be nice to have.
1000 + source tasks ? EventMetaData is a fairly expensive object which could be
simplified a lot - vertexNames - ids, taskAttemptId with Ids.
TaskAttemptId are shared. The vertex names could be interned - this is all
short term though. That's also a lot of unnecessary data transfer over the wire.
That said, EventMetaData instances should be candidates for GC once
LogicalIOPorcessorRuntimeTask is done with them - and forwarded them over to
the actual Input/Ouput/Processor. From the heap dump, do you know what is
holding on to these instances ?
> Shuffle OOM due to EventMetaData flood
> --------------------------------------
>
> Key: TEZ-3114
> URL: https://issues.apache.org/jira/browse/TEZ-3114
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Jason Lowe
>
> A task encountered an OOM during shuffle, and investigation of the heap dump
> showed a lot of memory being consumed by almost 3.5 million EventMetaData
> objects. Auto-parallelism had reduced the number of tasks in the vertex to 1
> and there were 2000 upstream tasks to shuffle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
