[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519181#comment-14519181
]
Rajesh Balamohan commented on TEZ-776:
--------------------------------------
- In Edge, "routeInputSourceTaskFailedEventToDestination" is checked for
enabling onDemandRouting. Would it an issue if the custom edgeManagers do not
have routeInputSourceTaskFailedEventToDestination or prepareForRouting() etc
missed out?
- In Edge.java, would be the following code be an issue if we have bigger
PartitionRange (corner case) in ShuffleVertexManager's
CustomShuffleEdgeManager?
{noformat}
if (routeMeta.getNumEvents() + listToAdd.size() > listMaxSize) {
return false;
}
{noformat}
- Common code in the
ShuffleVertexManager.CustomShuffleEdgeManager.prepareRouting() can be moved to
separate method "private int[] createIndex(int taskIndex, int partitionRange)"?
That can possibly reduce the code in prepareRouting and
createTargetIndicesForRemainder might not be needed in that case
- In OneToOneEdgeManager,commonRouteMeta can be init-ed in prepareForRouting()
similar to other managers (mainly for consistency)
- Should we throw exceptions in routeDataMovementEventToDestination in
OneToOneEdgeManager when sourceTaskIndex != destinationTaskIndex?. Or there can
be instances when null is acceptable?
- eventIndicesCreated, sourceIndicesRemainder can be removed in
ShuffleVertexManager
- BroadcastEdgeManager - cachedEventsLock, cachedEvents may not be relevant
- EdgeManagerPlugin - unwanted imports can be removed
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
> TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
> TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch,
> TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch,
> TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
