Thank you Cheolsoo. I am currently adding prints in a few classes under the executionengine/mapReduceLayer package and recompiled the source code to use the jars with the prints. The prints show up, but not the content that I am looking for which is the data received by the reducer from the mapper (or the combiner if there is one in the plan). Still investigating :)
Thanks, Keren On Fri, Aug 1, 2014 at 7:58 PM, Cheolsoo Park <[email protected]> wrote: > >> I am running a Pig job, and would like to see the intermediate outputs > flowing > out from the combiners into the reducers for this query. > > Actually, this won't help you since you're looking for intermediate files > between mappers and reducers. This feature is for debugging intermediate > files between MR jobs. > > Sorry for the confusion. > > > On Fri, Aug 1, 2014 at 7:55 PM, Cheolsoo Park <[email protected]> > wrote: > > > You can enable the following options on the command line- > > pig -Dpig.delete.temp.files=false -Dpig.temp.dir=/foo > > > > It will keep intermediate files in /foo on hdfs. These properties must be > > passed on the command line, or Main will delete all the temporary files > at > > exit. > > > > To load temporary files, search for pig.reduce.output.dirs in job conf. > > The value will be something like /foo/temp610061619/tmp-1397105382. The > > file format is what you configure in your Pig job for temporary files > such > > TFile, Sequence, etc. If you're using TFile, you can do the following- > > > > a = LOAD '/foo/temp610061619/tmp-1397105382' USING > > org.apache.pig.impl.io.TFileStorage(); > > DUMP a; > > > > This feature was introduced in 0.13. > > > > > > On Fri, Aug 1, 2014 at 12:26 AM, Keren Ouaknine <[email protected]> > wrote: > > > >> Hello, > >> > >> I am running a Pig job, and would like to see the intermediate outputs > >> flowing out from the combiners into the reducers for this query. Is > there > >> a > >> flag I could activate (some kind of verbose or debug mode) that would > >> output these values? > >> > >> Thanks, > >> Keren > >> > >> -- > >> Keren Ouaknine > >> www.kereno.com > >> > > > > > -- Keren Ouaknine www.kereno.com
