Group, and pass the grouped sets to your batch-processing UDF?
so:
data:
id1 bucket1
id2 bucket2
id3 bucket2
id4 bucket1
bucketized = group data by bucket_id;
bucket1, { (id1, id4) }
bucket2, { (id2, id3) }
batch_processed = foreach bucketized generate MyUDF(data);
D
On Wed, Sep 12, 2012 at 11:55 AM, Terry Siu <[email protected]> wrote:
> Hi all,
>
> I'm wondering if anyone has experience with my following scenario:
>
> I have a HBase table loaded with millions of records. I want to load these
> records into Pig, process each batch of 1000 by calling an external API, and
> then associate the results with the Tuples in each batch. I've having
> difficulty figuring out how to do this in Pig (via a UDF) if this is even
> possible or a valid scenario for Pig. In a nutshell, this is what I want:
>
> [Input]
> ID1,A,B
> ID2,C,D
> ID3,E,F
> ID4,G,H
> ...
> IDn,Y,Z
>
> For example purposes, let's say batch size is 2. So, I'd like ID1 and ID2 to
> be batched together, call the external API, and then have the returned Tuples
> to include the data from the API call. Similarly, ID3 and ID4 are batched
> together, call the external API, and the returned Tuples have the data from
> API. So, I'd like my output to be:
>
> [Output]
> ID1,A,B,1,2
> ID2,C,D,3,4
> ID3,E,F,5
> ID4,G,H,6,7,8
> ...
> IDn,Y,Z
>
> Yes, I can call the API per record, but I want to reduce the # of API calls,
> thus, I'd like to batch of set of records and then call the API.
>
> Hope this makes sense. Is this possible via Pig with UDF?
>
> -Terry
>
> PS: I did try implementing an accumulator by grouping my records via a single
> constant value, hoping that the accumulate() and getValue() are called per
> batch with no luck.
>
