Some of the discrepancy could be due to compression of map input/output format. E.g. The mapper output bytes will show the compressed size, while the reduce input will show the uncompressed size. Or something along those lines. But I'm also questioning the accuracy of the reporting and suspect that some if it is due to all the disk activity that happens while processing spills in the mapper and the copy/shuffle/sort phase in the reducer. It would certainly be nice if all the byte counts were reported in a way that they're comparable.
-- Stefan > From: Bryan Duxbury <br...@rapleaf.com> > Reply-To: <core-user@hadoop.apache.org> > Date: Tue, 17 Mar 2009 17:26:52 -0700 > To: <core-user@hadoop.apache.org> > Subject: Massive discrepancies in job's bytes written/read > > Hey all, > > In looking at the stats for a number of our jobs, the amount of data > that the UI claims we've read from or written to HDFS is vastly > larger than the amount of data that should be involved in the job. > For instance, we have a job that combines small files into big files > that we're operating on around 2TB worth of data. The outputs in HDFS > (via hadoop dfs -du) matches the expected size, but the jobtracker UI > claims that we've read and written around 22TB of data! > > By all accounts, Hadoop is actually *doing* the right thing - we're > not observing excess data reading or writing anywhere. However, this > massive discrepancy makes the job stats essentially worthless for > understanding IO in our jobs. > > Does anyone know why there's such an enormous difference? Have others > experienced this problem? > > -Bryan
