[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316061#comment-14316061
]
Siddharth Seth edited comment on TEZ-776 at 2/26/15 7:11 AM:
-------------------------------------------------------------
That said, I think Point2 from
https://issues.apache.org/jira/browse/TEZ-776?focusedCommentId=14000047&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14000047
(now Option 4a) will be the most efficient in terms of memory and CPU.
- API on the plugin to track events, along with tasks to whcih they need to go.
- Implement by all EdgeManagerPlugins within Tez - we know exactly how the
routing is to be performed.
- Fallback to current routing for EdgeManagerPlugins which don't implement this
API.
Option 4b would be to just special case the ScatterGatherPlugin and other Tez
plugins, without adding an interface to the EdgeManagerPlugin API.
(Fixed references to be EdgePlugins instead of VertexPlugins)
Will add comments on the doc tomorrow. It'll be useful to have some numbers on
expected impact on memory usage / CPU.
was (Author: sseth):
That said, I think Point2 from
https://issues.apache.org/jira/browse/TEZ-776?focusedCommentId=14000047&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14000047
(now Option 4a) will be the most efficient in terms of memory and CPU.
- API on the plugin to track events, along with tasks to whcih they need to go.
- Implement by all VertexManagerPlugins within Tez - we know exactly how the
routing is to be performed.
- Fallback to current routing for VertexManagerPlugins which don't implement
this API.
Option 4b would be to just special case the ScatterGatherPlugin and other Tez
plugins, without adding an interface to the VertexManagerPlugin API.
Will add comments on the doc tomorrow. It'll be useful to have some numbers on
expected impact on memory usage / CPU.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.patch,
> events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
