Binglin, Thanks a lot for the info, I will check the format.
2012/2/17 Binglin Chang <decst...@gmail.com> > As far as I know, TeraOutputFormat don't support compression > > > On Fri, Feb 17, 2012 at 6:16 PM, <bejoy.had...@gmail.com> wrote: > >> ** >> Juwei >> Is there any error messages on your TaskTracker logs related to >> compression like 'Codec not found' or so ? >> Regards >> Bejoy K S >> >> From handheld, Please excuse typos. >> ------------------------------ >> *From: * Juwei Shi <shiju...@gmail.com> >> *Date: *Fri, 17 Feb 2012 17:48:08 +0800 >> *To: *<mapreduce-user@hadoop.apache.org> >> *ReplyTo: * mapreduce-user@hadoop.apache.org >> *Subject: *Re: reduce output compression of Terasort >> >> We use LZO, so the value is >> mapred.output.compression.codec = com.hadoop.compression.lzo.LzoCodec >> >> No compressed file in HDFS. >> >> >> >> 2012/2/17 Bejoy Ks <bejoy.had...@gmail.com> >> >>> Hi Juwei >>> What is the value for mapred.output.compression.codec? It'd be >>> better to determine whether the output files are compressed by getting the >>> codec of the same and not just from the size of files. >>> >>> Regards >>> Bejoy.K.S >>> >>> >>> On Fri, Feb 17, 2012 at 12:07 PM, Juwei Shi <shiju...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am benchmarking the cluster using the Terasort package of Hadoop >>>> 0.20.2. I enabled compression for both map output (* >>>> mapred.compress.map.output*) and reduce output (*mapred.output.compress >>>> *). I checked the parameter in Job.xml, both are true. I can see that >>>> the compression for Map output works, but it seems that the compression for >>>> reduce output does not work. The output of the job on HDFS is also 1TB. >>>> >>>> Thanks! >>>> >>>> - Juwei >>>> >>> >>> >> >> >>
