Have you taken a look at the Algebraic and Accumulator interfaces? They provide exactly these sorts of benefits.
2012/8/23 Yang <[email protected]> > if I group records into a huge bag, and hand over to a Udf, would the input > tuple actually > create a bag with all the records? that way it may generate a OOM ?? > > if indeed there is such an issue, I could probably implement the logic in > plain pig, instead of Udf. > but many times, logic is so complex that only Udf could do it. > > > Thanks > Yang >
