Re: Pig Relation Sorting, labeling, partitioning

[Alan Gates]

Item 2 is no currently easy to do in Pig in a parallel fashion.  This  
is because you don't know how many records each map task is going to  
get so you don't know which number to start on in map 2 and greater.   
You could write a complex two pass algorithm that were first count the  
number of tuples and then do the splits again, but it would involve  
implementing your own Slicer and LoadFunc.

If your data set is small enough that you can do the labeling on a  
single machine you can do something like:

A = load 'data';
B = group A all parallel 1;
C = foreach B {
       D = order A by sortkey;
       generate flatten(lablerUDF(D));
}

where lablerUDF is a UDF you write that walks through the bag and  
appends the position to each tuple.  If you are doing this on trunk I  
was strongly suggest using the new accumulator interface for this UDF  
as it will make is much more efficient.  But again, this depends on  
pulling all of your data onto one machine, which defeats the purpose  
of parallel systems like Hadoop.

Alan.

On Nov 20, 2009, at 12:03 PM, Desai Dharmendra wrote:

I am using PIG and this is what I am trying to do this:

1) Sort a relation A into B by a field x. The smallest value of x is  
first.
Just use SORT.

2) Label each tuple in B with a number denoting its order in the  
sorted
relation. So the first tuple would be labeled with a 1, the second  
tuple
with a 2, the third with a 3 and so on. Not certain how to do this.

3) Derive a relation C where each row is a bag of tuples. The first  
row
contains the first n1 tuples from relation B, the second row  
contains the
tuples from B labeled (n1 + 1) to n2 from, the third row contains  
the tuples
from B labeled (n2 + 1) to n3 and so on to n100. This step is simple  
(just
use filter) once we've labeled each tuple in B with a number.

The question: how do I do step 2)?

thanks