Alan Gates commented on PIG-545:

I ran the pigmix L9 (order by of single field) and L10 (order by of multiple 
fields).  L9 went from 14 minutes to 8, so this patch holds huge promise.  But 
L10 went from 8 minutes to 11, so it doesn't seem to be working well in the 
multiple field case.  (It could also be related to the fact that L10 uses 
descending on one of the columns, I don't know if the new partitioner can 
handle that or not.)  I also ran our end to end order by tests on it, and all 
passed, except bigdata_1, which fails with an IndexOutOfBounds exception in the 
new WeightedRangePartitioner class.  

As for the caveat that it needs to know the number of reducers up front, I 
believe in cases where the user doesn't say parallel, that we can determine the 
parallelism of the reduces using JobClient.getDefaultReduces().  We need to 
double check that this will give us the right information in both the hod and 
non-hod cases.

> PERFORMANCE: Sampler for order bys does not produce a good distribution
> -----------------------------------------------------------------------
>                 Key: PIG-545
>                 URL: https://issues.apache.org/jira/browse/PIG-545
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Alan Gates
>            Assignee: Amir Youssefi
>             Fix For: types_branch
>         Attachments: WRP.patch
> In running tests on actual data, I've noticed that the final reduce of an 
> order by has skewed partitions.  Some reduces finish in a few seconds while 
> some run for 20 minutes.  Getting a better distribution should lead to much 
> better performance for order by.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to