[
https://issues.apache.org/jira/browse/PIG-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667778#action_12667778
]
Olga Natkovich commented on PIG-636:
------------------------------------
+1 on the patch. The only minor thing that I saw that needs to be changes is
size of a single tuple bag should be changed to 1 rather than 0.
> PERFORMANCE: Use lightweight bag implementations which do not register with
> SpillableMemoryManager with Combiner
> ----------------------------------------------------------------------------------------------------------------
>
> Key: PIG-636
> URL: https://issues.apache.org/jira/browse/PIG-636
> Project: Pig
> Issue Type: Improvement
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
> Attachments: PIG-636.patch
>
>
> Currently whenever Combiner is used in pig, in the map, the
> POPrecombinerLocalRearrange operator puts the single "value" tuple
> corresponding to a key into a DataBag and passes this to the foreach which is
> being combined. This will generate as many bags as there are input records.
> These bags all will have a single tuple and hence are small and should not
> need to be spilt to disk. However since the bags are created through the
> BagFactory mechanism, each bag creation is registered with the
> SpillableMemoryManager and a weak reference to the bag is stored in a linked
> list. This linked list grows really big over time causing unnecessary Garbage
> collection runs. This can be avoided by having a simple lightweight
> implementation of the DataBag interface to store the single tuple in a bag.
> Also these SingleTupleBags should be created without registering with the
> spillableMemoryManager. Likewise the bags created in POCombinePackage are
> supposed to fit in Memory and not spill. Again a NonSpillableDataBag
> implementation of DataBag interface which does not register with the
> SpillableMemoryManager would help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
