Hey all, I've starting using SequenceFiles more and more (in particular the elephant bird load and storage functions) and am wondering what's the best approach is for marshaling between a schema from pig (which can have some arbitrary number of fields) and a sequence files (which must have two fields; key and value).
When I've got two fields its trivial... %declare SEQFILE_STORAGE 'com.twitter.elephantbird.pig.store.SequenceFileStorage'; %declare TEXT_CONVERTER 'com.twitter.elephantbird.pig.util.TextConverter'; %declare LONG_CONVERTER 'com.twitter.elephantbird.pig.util.LongWritableConverter'; a = load 'x' as (f1:chararray, f2:chararray); store a into 'y' using $SEQFILE_STORAGE( '-c $TEXT_CONVERTER', '-c $TEXT_CONVERTER'); but what's the best way to handle something with 3+ fields? a = load 'x' as (f1:chararray, f2:chararray, f3:chararray); I can see two options... 1) A simple writeable convertor to convert to something like f1 and a composite f2, f3 field 2) Packing the fields myself using something like "a = foreach a generate f1, TOTUPLE(f2, f3)" But both are super clumsy and require unpacking when i reread things. Am I missing something obvious here? Cheers, Mat
