Thanks Himanshu. Is there a part 2? -- Michael
--- On Tue, 2/16/10, himanshu chandola <himanshu_cool...@yahoo.com> wrote: From: himanshu chandola <himanshu_cool...@yahoo.com> Subject: Re: LZO compression for Map output in Hadoop 0.20+? To: common-user@hadoop.apache.org Date: Tuesday, February 16, 2010, 11:35 PM You might want to check out this: http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Morpheus: Do you believe in fate, Neo? Neo: No. Morpheus: Why Not? Neo: Because I don't like the idea that I'm not in control of my life. ----- Original Message ---- From: jiang licht <licht_ji...@yahoo.com> To: common-user@hadoop.apache.org Sent: Wed, February 17, 2010 12:26:48 AM Subject: LZO compression for Map output in Hadoop 0.20+? New to Hadoop (now using 0.20.1), I want to know how to choose and set up compression methods for Map output, especially how to configure and use LZO compression? Specifically, please share your experience for the following 2 scenarios. Thanks! (1) Is there a global setting in some hadoop configuration files for naming a compression method (e.g. LZO) such that it will be used to compress Map output by default? and how? (2) How to use a compression method (e.g. LZO) in java code (I noticed that in javadoc, org.apache.hadoop.mapred is labeld Deprecated)? Thanks! -- Michael
