[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527677#comment-14527677
]
Siddharth Seth commented on TEZ-776:
------------------------------------
- Minor: {code}Target input indices. The number must match the number of
events{code}
count / size of array may make this a little clearer. 'number' is a little
vague.
- Precondition checks to verify this ?
- BroadcastEdgeManger - commonRouteMeta setup via prepareForRouting. Not sure
access this structure at a later point is thread safe. This goes away anyway if
Broadcast/OneToOne are left unchanged.
- Bunch of repeated code between OneToOne, Broadcast, ScatterGather etc in
Edge.java. Looks like it's all the same (exploding the EventRouteMetadata)
- Not sure if the thread safety applies to ScatterGather as well. That seems to
be making changes within a lock though. Seems fairly complicated, assuming
that's all for caching and efficiency ?
- There's several methods on EdgeManagerPluginContextOnDemand which don't need
to be implemented/extended (The method on EdgeManagerPluginContext should be
sufficient). - e.g. initialize(), getContext, some of the routing methods
- I'm still concerned about the access to taskEvents (taskEvents.size() and
taskEvents.get()). This is an array list getting populated in one thread, and
accessed in 30 others without a lock. ArrayList isn't supposed to be thread
safe afaik. Will let someone else chime in here.
On TEZ-2409. I think it'll be better to get that done here itself. It's
probably 10 more lines, and removes the changes on Broadcast/OneToOne. 2409
becomes a blocker for 0.7 anyway - and would end up reverting undoing changes
made here. The overall functionality is already tested by the various jobs that
we run.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.10.patch, TEZ-776.11.patch,
> TEZ-776.2.patch, TEZ-776.3.patch, TEZ-776.4.patch, TEZ-776.5.patch,
> TEZ-776.6.A.patch, TEZ-776.6.B.patch, TEZ-776.7.patch, TEZ-776.8.patch,
> TEZ-776.9.patch, TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
> TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch,
> TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch,
> With_Patch_AM_hotspots.png, With_Patch_AM_profile.png,
> Without_patch_AM_CPU_Usage.png, events-problem-solutions.txt,
> with_patch_jmc_output_of_AM.png, without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
