Ankit Modi updated PIG-1106:

    Attachment: frjoin-nonspill.patch

This patch does not have any tests. Creating a test would be creating a big 
file about 250 MB and testing it.

I have ran some tests in similar fashion.

> FR join should not spill
> ------------------------
>                 Key: PIG-1106
>                 URL: https://issues.apache.org/jira/browse/PIG-1106
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Olga Natkovich
>            Assignee: Ankit Modi
>             Fix For: 0.7.0
>         Attachments: frjoin-nonspill.patch
> Currently, the values for the replicated side of the data are placed in a 
> spillable bag (POFRJoin near line 275). This does not make sense because the 
> whole point of the optimization is that the data on one side fits into 
> memory. We already have a non-spillable bag implemented 
> (NonSpillableDataBag.java) and we need to change FRJoin code to use it. And 
> of course need to do lots of testing to make sure that we don't spill but die 
> instead when we run out of memory

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to