[
https://issues.apache.org/jira/browse/PIG-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209003#comment-15209003
]
Rohini Palaniswamy commented on PIG-4847:
-----------------------------------------
Also wanted to note that I wanted to get rid of collection threshold
notifications as when big sort buffers are involved we will keep hitting it.
But retained them and kept them same as usage threshold due to the below
comment in the code.
{code}
// we want to set both collection and usage threshold alerts to be
// safe. In some local tests after a point only collection threshold
// notifications were being sent though usage threshold notifications
// were sent early on.
{code}
Another thing that would happen with this patch is it might try to free more
memory as threshold size is higher now. spillFileSizeThreshold of 5MB should
avoid spill of small bags. But if there is unnecessary spill of other bigger
bags we might have to cap toFree at a maximum fixed size instead of 50% of
thresholdsize so that toFree is not too big for bigger heaps.
{code}
toFree = info.getUsage().getUsed() - memoryThresholdSize +
(long)(memoryThresholdSize * 0.5);
{code}
> POPartialAgg processing and spill improvements
> ----------------------------------------------
>
> Key: PIG-4847
> URL: https://issues.apache.org/jira/browse/PIG-4847
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.16.0
>
> Attachments: PIG-4847-1.patch, PIG-4847-2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
