[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335970#comment-14335970
]
Siddharth Seth commented on TEZ-776:
------------------------------------
bq. So I am not clear why we would go down the path of creating a generic
compact routing table representation.
A more compact representation attempts to take a middle ground between CPU and
memory overheads - instead of being at either extreme. Most generic compact
representations, however, would also be far too large in terms of the memory
footprint.
bq. There is overhead from the list based representations in some of the API's
but that is not where the memory is being used. The API's were designed for
bulk routing where all routing for an event happens in one shot upon event
receipt. Though I agree that an interface might have been cheaper memory wise
than lists. More on this in the next comment.
The list based interfaces has nothing to do with the memory overhead (except
for very short lived objects, which may trigger GC more often). The point on
moving away from List based APIs was to be more efficient - 1) in terms of the
way code is written, and 2) in case Tez decides to infer anything from the
return value. IAC, that can be discussed in the jira which will change this.
Looked at the v1 patch earlier today - assuming v2 is similar. (Wasn't able to
look at the patch earlier, apologies). Comments follow.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.patch,
> events-problem-solutions.txt
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
