[
https://issues.apache.org/jira/browse/TEZ-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149040#comment-15149040
]
Rohini Palaniswamy commented on TEZ-3114:
-----------------------------------------
bq. Will do, although I'm confused as to what's special about this Scope.VERTEX
property compared to all the other existing ones like
TEZ_TASK_MAX_EVENTS_PER_HEARTBEAT.
Scope.AM applies to all vertices and cannot be controlled per vertex.
Scope.VERTEX applies to per vertex and can have different values for different
vertices. For eg: One vertex can have speculation enabled and another disabled.
So if it is a system level setting, makes sense to have it on the AM level and
something that we want to let user control need to make it as scope vertex.
> Shuffle OOM due to EventMetaData flood
> --------------------------------------
>
> Key: TEZ-3114
> URL: https://issues.apache.org/jira/browse/TEZ-3114
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: TEZ-3114.001.patch, TEZ-3114.002.patch
>
>
> A task encountered an OOM during shuffle, and investigation of the heap dump
> showed a lot of memory being consumed by almost 3.5 million EventMetaData
> objects. Auto-parallelism had reduced the number of tasks in the vertex to 1
> and there were 2000 upstream tasks to shuffle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
