In what way are you gathering results? Solutions typically involve a choice of: * don't -- just read directories * openIterator (which is the same thing, really) * use a single reducer * hadoop fs -cat /path/to/output/* > myoutput * use HAR * write your own
Pig (and hadoop) don't store your results in a single file because that would force all reducers to coordinate writing their outputs; this way they function completely independently. -D On Mon, Jan 31, 2011 at 4:13 PM, <[email protected]> wrote: > Is there a way to tell pig in map red embedded mode > to store all my results in single results file > instead of all the parts that it creates > that I have to merge afterwards > > if it is not possible then > what is the recommended way to gather the results (using openIterator ?) > > thanks > Anindita >
