I am able to group the tuples by date. But the problem I am facing is how do I ensure that when I finally STORE it, it is stored in separate folders?
On Tue, Jan 10, 2012 at 11:27 AM, Daniel Dai <[email protected]> wrote: > You can use custom partitioner. Check > http://pig.apache.org/docs/r0.9.1/basic.html#partitionby. > > Daniel > > On Mon, Jan 9, 2012 at 9:39 PM, IGZ Nick <[email protected]> wrote: > > > Hi, > > > > What I would like to do is to store outputs to different directories > based > > on record value. Essentially I want to read the date from a field and > store > > the output in yyyy/mm/dd directory structure. How should I go about > this? I > > want to use AvroStorage for storing the stuff. I want to specify STORE > xyz > > INTO '$location' USING MyStorage(); where $location would be the base > > output directory. MyStorage() would be the modified version of > AvroStorage > > which stores the values in $location/yyyy/mm/dd/part-abc files, reading > the > > yyyymmdd from a particular field in the input records. > > > > What is the way to achieve this with minimal changes? > > > > Nick > > >
