Thanks Yongqiang. My last message was wrong - we had mapred.compress.map.output=false. Will try to get LZO on our cluster and see.
Zheng From: He Yongqiang [mailto:[email protected]] Sent: Friday, October 23, 2009 1:28 PM To: [email protected] Subject: Re: DefaultCodec vs. LZO I did a search, and find the below two links, hope they are helpful. http://blog.oskarsson.nu/2009/03/hadoop-feat-lzo-save-disk-space-and.html http://mail.jabber.org/pipermail/standards/2005-October/008768.html I think Hadoop’s DefaultCodec is using native Zlib when possible. Thanks, Yongqiang On 09-10-23 下午12:43, "Zheng Shao" <[email protected]> wrote: We are using Hadoop 0.20 and mapred.map.output.compression.Codec is set to DefaultCodec. We tried LZO but the performance seems very similar to DefaultCodec. I heard of a lot of good words about LZO. So did anybody compare LZO with DefaultCodec? Is there a big difference? We are running CentOS release 5.2 (Final). Thanks, Zheng
