[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520228#comment-14520228
]
Bikas Saha commented on TEZ-776:
--------------------------------
bq. In Edge, "routeInputSourceTaskFailedEventToDestination" is checked for
enabling onDemandRouting
It uses the new signature of "routeCompositeDataMovementEventToDestination".
The logic is that if the plugin has not explicitly used the new API then it
must be using legacy API (since they were abstract). In some sense its a
reverse check to check if the plugin is a legacy API plugin. Of course the
check is not exhaustive but its not meant to be. This is to enabled older
versions of hive to run without change with newer versions of tez and limit the
legacy routing to only those vertices that use legacy plugins.
bq. if (routeMeta.getNumEvents() + listToAdd.size() > listMaxSize) {
Wil fix. I somehow missed it (I thought I had left a TODO there but clearly I
did not and forgot about it :P)
bq. Common code in the
ShuffleVertexManager.CustomShuffleEdgeManager.prepareRouting()
Will check if thats possible.
bq. In OneToOneEdgeManager,commonRouteMeta can be init-ed in prepareForRouting()
I did that so that I could make it final and allow compiler optimizations. I
forgot to add the final :P Added the final now.
bq. Should we throw exceptions in routeDataMovementEventToDestination in
OneToOneEdgeManager when sourceTaskIndex != destinationTaskIndex?. Or there can
be instances when null is acceptable?
It is by design. Not all consumers get events from every producer. Similar to
ScatterGather#routeDataMovementEventToDestination
bq. eventIndicesCreated, sourceIndicesRemainder can be removed in
ShuffleVertexManager
bq. BroadcastEdgeManager - cachedEventsLock, cachedEvents may not be relevant
bq. EdgeManagerPlugin - unwanted imports can be removed
Done.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
> TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
> TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch,
> TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch,
> TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
