[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316061#comment-14316061
]
Siddharth Seth commented on TEZ-776:
------------------------------------
That said, I think Point2 from
https://issues.apache.org/jira/browse/TEZ-776?focusedCommentId=14000047&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14000047
(now Option 4a) will be the most efficient in terms of memory and CPU.
- API on the plugin to track events, along with tasks to whcih they need to go.
- Implement by all VertexManagerPlugins within Tez - we know exactly how the
routing is to be performed.
- Fallback to current routing for VertexManagerPlugins which don't implement
this API.
Option 4b would be to just special case the ScatterGatherPlugin and other Tez
plugins, without adding an interface to the VertexManagerPlugin API.
Will add comments on the doc tomorrow.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
