[
https://issues.apache.org/jira/browse/PIG-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583025#action_12583025
]
Pi Song commented on PIG-170:
-----------------------------
In certain scenarios, for example, a lot of small intermediate bags are used to
calculate something and outputs are stored in a big bag. The big bag will
become the first node after
{noformat}
Collections.sort(spillables, new Comparator<WeakReference<Spillable>>()
{noformat}
and the below code will no longer work.
{noformat}
public void registerSpillable(Spillable s) {
synchronized(spillables) {
// Cleaing the entire list is too expensive. Just trim off the
front while
// we can.
WeakReference<Spillable> first = spillables.peek();
while (first != null && first.get() == null) {
spillables.remove();
first = spillables.peek();
}
spillables.add(new WeakReference<Spillable>(s));
}
}
{noformat}
I
> Memory manager spills bags in the wrong order
> ---------------------------------------------
>
> Key: PIG-170
> URL: https://issues.apache.org/jira/browse/PIG-170
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Amir Youssefi
> Attachments: PIG-170_0_20080327.patch
>
>
> For optimal performance, we want to spill the largest bags first. This is not
> what is happening right now and could be causing some of our memory issues.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
