pig-user  

Performance/coding question

Prashanth Pappu
Thu, 19 Jun 2008 12:20:53 -0700

I have a PIG script that simply generates a lot of 'counts' over very large
data. For example,

a = load 'data' as (x,y,z);

b1 = filter a by x==1;
b1_group = group b1 all;
b1_count = foreach b1_group generate COUNT(b1);

b2 = filter a by y==1;
b2_group = group b2 all;
b2_count = foreach b2_group generate COUNT(b2);

...etc

Suppose that we need to generate counts b1 to b1000. Now, PIG generates 1000
different hadoop jobs (one for each count). While each job finishes fast
enough, the per job overhead considerably slows down the script. So, I have
two questions

(a) If I want to generate many counts by simply filtering the rows of the
data - is there a better way to code this script?
(b)  Are there any PIG optimizations (current or planned) that will cause
PIG to  generate  fewer number of jobs? Because, clearly, one can write a
single java map-reduce job to accomplish the task.

Thanks,
Prashanth