[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353345#comment-14353345
]
Siddharth Seth commented on TEZ-776:
------------------------------------
Thanks for the analysis [~rajesh.balamohan]. Without the changes, this
particular job is not expected to run - and I expected the CPU usage is due to
tons of GC until the JVM finally gives up.
I had some run some numbers a couple of days ago to understand the similarity
in numbers posted earlier - with a no-op job, ScatterGather / OneToOne edge.
Note: There were a fair number of issues with big jobs; I'll open follow up
jiras for these.
Some things that came out of this - we're also performing unnecessary MxN
operations to route for ScatterGather edges. So today, we have MxN CPU and
memory overheads. What's worse in the current state of master is that all of
this happens on the central dispatcher thread - since that's the one used to
route events.
We can, however, do a lot better than MxN CPU for ScatterGather, Broadcast and
OneToOne - any edge which has some repeatability in the routing pattern.
For ScatterGather for instance, the EdgePlugin would store M events with the
targetIndex set on the event the moment it is seen. When a task asks for
events, the events are returned as is - with a set of source indices as a first
event. The task takes care of generating the final DataMovementEvent that will
be consumed.
For completeness, putting in the numbers that were observed.
One node dedicated to the AM - with a 16GB heap, Container size=2GB,
#attemptsPerTask=1
20K X 2K ScatterGather - Current implementation - Failed in 78s due to OOMs in
container
20K X 2K ScatterGather - ODR - Failed in 30s due to OOMs in containers
OneToOne was unstable and hung fairly often, managed to get only a single run
with ODR, and 2 runs with the current implementation.
50K X 60K OneToOne - Current implementation - CPUTime: 394530, 400420
50K X 50K OneToOne - ODR - CPUTime: 848170
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
> TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch,
> TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
