[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328663#comment-14328663
]
Bikas Saha commented on TEZ-776:
--------------------------------
EdgeManagerPlugin have been created for the specific goal of allowing logic in
code to replace memory needed to store a routing table and EdgeManagerPlugin
impls for broadcast/scatter-gather/1-1 etc do precisely that. So I am not clear
why we would go down the path of creating a generic compact routing table
representation.
There is overhead from the list based representations in some of the API's but
that is not where the memory is being used. The API's were designed for bulk
routing where all routing for an event happens in one shot upon event receipt.
Though I agree that an interface might have been cheaper memory wise than
lists. More on this in the next comment.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.patch, events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
