[ 
https://issues.apache.org/jira/browse/PIG-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667778#action_12667778
 ] 

Olga Natkovich commented on PIG-636:
------------------------------------

+1 on the patch. The only minor thing that I saw that needs to be changes is 
size of a single tuple bag should be changed to 1 rather than 0.

> PERFORMANCE: Use lightweight bag implementations which do not register with 
> SpillableMemoryManager with Combiner
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-636
>                 URL: https://issues.apache.org/jira/browse/PIG-636
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: types_branch
>
>         Attachments: PIG-636.patch
>
>
> Currently whenever Combiner is used in pig, in the map, the 
> POPrecombinerLocalRearrange operator puts the single "value" tuple 
> corresponding to a key into a DataBag and passes this to the foreach which is 
> being combined. This will generate as many bags as there are input records. 
> These bags all will have a single tuple and hence are small and should not 
> need to be spilt to disk. However since the bags are created through the 
> BagFactory mechanism, each bag creation is registered with the 
> SpillableMemoryManager and a weak reference to the bag is stored in a linked 
> list. This linked list grows really big over time causing unnecessary Garbage 
> collection runs. This can be avoided by having a simple lightweight 
> implementation of the DataBag interface to store the single tuple in a bag. 
> Also these SingleTupleBags should be created without registering with the 
> spillableMemoryManager. Likewise the bags created in POCombinePackage are 
> supposed to fit in Memory and not spill. Again a NonSpillableDataBag 
> implementation of DataBag interface which does not register with the 
> SpillableMemoryManager would help.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to