[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518504#comment-14518504
]
Hitesh Shah commented on TEZ-776:
---------------------------------
Question on the below code snippet:
{code}
EventRouteMetadata routeMeta = edgeManager
.routeCompositeDataMovementEventToDestination(srcTaskIndex,
taskIndex);
if (routeMeta != null) {
if (routeMeta.getNumEvents() + listToAdd.size() >
listMaxSize) {
return false;
}
{code}
It seems that there would be a problem if routeMeta.getNumEvents() was greater
than listMaxSize in any scenario ( maybe with auto-reduce where a whole bunch
of partitions get re-mapped to a single reducer) ?
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
> TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
> TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch,
> TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch,
> TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
