[
https://issues.apache.org/jira/browse/PIG-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984949#action_12984949
]
Scott Carey commented on PIG-1815:
----------------------------------
{quote}
I haven't followed the exact call sequence that leads to it, but it looks like
a PhysicalPlan instance is created with reference to a copy of the previous
Reducer$Context, and since this is a inner class of PigCombiner$Combine (a
subclass of Reducer) it has a reference (this$0) to it.
{quote}
I don't know if this is new info to anyone reading this, but there are two
types of inner classes, those with the 'this' reference to the enclosing class
and those without it. Those that don't hold references to their enclosing
class are those marked 'static'. If this inner class can be made static, it
would remove that reference (and make the object a little smaller).
In general, you can help the GC and memory footprint by making sure inner
classes are 'static' whenever they don't need to have the implicit 'this'
reference.
> pig task retains used instances of PhysicalPlan
> -----------------------------------------------
>
> Key: PIG-1815
> URL: https://issues.apache.org/jira/browse/PIG-1815
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0, 0.9.0
>
> Attachments: PIG-1815.1.patch, yourkit_combiner_hprof.jpg
>
>
> map tasks of a pig query ran out of memory because there were too many
> (thousands) instances of combiner PhysicalPlan in memory. Each physical plan
> (except the last?) was linked to older one as shown in the yourkit snapshot
> that I am attaching.
> This problem was noticed with 0.8 because of the split combination feature,
> that resulted in each map having larger inputs. The query also had large
> physical plan because of multi-query, it had 17 MR jobs merged into one
> during the multi-query optimization phase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
