[ 
https://issues.apache.org/jira/browse/PIG-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759689#action_12759689
 ] 

Olga Natkovich commented on PIG-975:
------------------------------------

Ying, what Pradeep is asking for is more like a safety switch - to give users a 
way to go back to the old implementation if they run into problem with new. 
Once we verify that the new code is as stable as the old, we would remove the 
switch. We would also not expose it to users unless they do run into trouble.

> Need a databag that does not register with SpillableMemoryManager and spill 
> data pro-actively
> ---------------------------------------------------------------------------------------------
>
>                 Key: PIG-975
>                 URL: https://issues.apache.org/jira/browse/PIG-975
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.2.0
>            Reporter: Ying He
>            Assignee: Ying He
>             Fix For: 0.2.0
>
>         Attachments: internalbag.xls, PIG-975.patch, PIG-975.patch2, 
> PIG-975.patch3
>
>
> POPackage uses DefaultDataBag during reduce process to hold data. It is 
> registered with SpillableMemoryManager and prone to OutOfMemoryException.  
> It's better to pro-actively managers the usage of the memory. The bag fills 
> in memory to a specified amount, and dump the rest the disk.  The amount of 
> memory to hold tuples is configurable. This can avoid out of memory error.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to