Alan Gates updated PIG-556:

    Attachment: PIG-556.patch

The attached patch addresses several issues.  It addresses the lack of 
progress() calls in FindQuantiles.  It also fixes two other problems:

# On order bys the sampling job was running the default number of reduces even 
though all data was sent to one reduce.  This was fixed to only run one reduce 
for that job.
# The progress reporter being passed to EvalFuncs is null.  I believe this is 
because it is not getting properly serialized and sent across from front end to 
back end.  I changed it so that the progress reporter is set each time the 
EvalFunc is called.  I also set set EvalFunc.setReporter to be final so that 
these set calls can hopefully be inlined by the java optimizer.

> FindQuantiles function does not report progress
> -----------------------------------------------
>                 Key: PIG-556
>                 URL: https://issues.apache.org/jira/browse/PIG-556
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>             Fix For: types_branch
>         Attachments: PIG-556.patch
> FindQuantiles does not call progress.  Most of the time this isn't an issue 
> as it finishes quickly.  But for very large (multi-terabyte) order bys, it 
> does not finish in the required five minutes.  This causes the whole job to 
> fail.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to