pig-user  

loading using a custom input formatter

Manish Shah
Thu, 05 Jun 2008 16:49:58 -0700

We have a custom input formatter that we use for regular map/reduce jobs. Is there a way to make use of this input formatter in pig? We've looked at most of the docs, and havent found much. The issue we have is that we arent loading data from a single file. Also the number of files is not determinable so we cant just write separate load commands in our pig latin.

The input formatter we have takes care of giving back records that conform to key/value semantics for hadoop map/reduce functions. Is there a reason it couldnt be used to generate tuples from the resultant records?

- Manish
Co-Founder Rapleaf.com
http://www.rapleaf.com/pub/Manish-Shah