[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354316#comment-14354316
]
Rohini Palaniswamy commented on TEZ-776:
----------------------------------------
Ran the
https://issues.apache.org/jira/secure/attachment/12703385/TEZ-776.ondemand.5.patch
on a pig script with the below plan which required 16G AM before. It succeeded
without OOM on 4G container.
v1 93378 (LOAD, FILTER) -> v2 1000 (GROUP)
v3 4431 (LOAD, FILTER) -> v4 1000 (GROUP)
v2,v4 -> v5 1000 (JOIN) -> v6 1000 (GROUP)
org.apache.tez.common.counters.DAGCounter
NUM_SUCCEEDED_TASKS=101809
TOTAL_LAUNCHED_TASKS=101809
OTHER_LOCAL_TASKS=44
DATA_LOCAL_TASKS=75055
RACK_LOCAL_TASKS=22710
I was monitoring the top usage on the AM couple of times. It did not even use
the 4G, but I was only checking before and around 30K tasks had completed. Did
not check after that.
Haven't got the time to go through the code in the patch or the long
conversation history in the jira. But looks good from the test run perspective.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
> TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch,
> TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
