[
https://issues.apache.org/jira/browse/PIG-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366391#comment-15366391
]
Rohini Palaniswamy commented on PIG-4626:
-----------------------------------------
TEZ-3286 contains the fix and has been checked into Tez 0.8.4 and 0.9.0. This
changes the behavior too. So we need to fix and set the setting at vertex level
as well now to avoid OOM with newer Tez releases.
> [Pig on Tez] OOM in case of multiple outputs and POPartialAgg
> -------------------------------------------------------------
>
> Key: PIG-4626
> URL: https://issues.apache.org/jira/browse/PIG-4626
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.17.0
>
>
> tez.task.scale.memory.reserve-fraction is 0.3 by default which assumes 70%
> of memory is available for io.sort.mb. If map is configured as 1G Xmx and
> io.sort.mb as 256MB, and there is group by on 3 different keys it has 3
> different outputs and Tez's WeightedScalingMemoryDistributor allocates 256MB
> to each of the outputs which leaves very less memory for Pig and POPartialAgg
> (which requires 20% of memory) and sometime leads to OOM. This also causes
> SpillableMemoryManager to be invoked often as its threshold is set to 70% of
> memory.
> Need to set tez.task.scale.memory.reserve-fraction as 0.5 for all pig jobs
> and 0.6 in case there is POPartialAgg in combiner plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
