[
https://issues.apache.org/jira/browse/PIG-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759645#action_12759645
]
Pradeep Kamath commented on PIG-975:
------------------------------------
I think it might be a good idea to have a config parameter (maybe a java -D
property) which can allow users to choose between spillableBagForReduce and
NonSpillableBagForReduce with the Non spillable one being the default. This way
if for some reason users find the spillablebag better for their query they can
use it.
> Need a databag that does not register with SpillableMemoryManager and spill
> data pro-actively
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-975
> URL: https://issues.apache.org/jira/browse/PIG-975
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Ying He
> Assignee: Ying He
> Fix For: 0.2.0
>
> Attachments: internalbag.xls, PIG-975.patch, PIG-975.patch2,
> PIG-975.patch3
>
>
> POPackage uses DefaultDataBag during reduce process to hold data. It is
> registered with SpillableMemoryManager and prone to OutOfMemoryException.
> It's better to pro-actively managers the usage of the memory. The bag fills
> in memory to a specified amount, and dump the rest the disk. The amount of
> memory to hold tuples is configurable. This can avoid out of memory error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
