Yes, seems pig.tmpfilecompression.codec does not support CMX now, following is my testing: SET pig.tmpfilecompression true; SET pig.tmpfilecompression.codec cmx; Error output: Caused by: java.io.IOException: Invalid temporary file compression codec [cmx]. Expected compression codecs for org.apache.pig.impl.io.TFileStorage are GZ,GZIP,LZO.
Is there anybody has any suggestion on this? Thanks At 2014-11-18 18:47:18, "Krishna Kalyan" <[email protected]> wrote: >Oh your question is regarding compression codec for temporary files?. >I am not sure about that. > > > >On Tue, Nov 18, 2014 at 4:13 PM, Krishna Kalyan <[email protected]> >wrote: > >> Hi lulynn, >> If you are using biginsights follow the steps to enable compression. >> >> set output.compression.enabled true; >> set output.compression.codec org.apache.hadoop.io.compress.GzipCodec; >> data = LOAD >> '/bigdata/sample_data/NOAA_Weather_csv/2011/999999-53019-2011.csv.gz' >> using PigStorage(',') as (projectname:chararray); >> STORE data INTO '/comCodecGzip'; >> >> Cheers, >> Krishna >> >> On Tue, Nov 18, 2014 at 2:04 PM, lulynn_2008 <[email protected]> wrote: >> >>> BTW, cmx is "com.ibm.biginsights.compress.CmxCodec", the related jar is >>> ibm-compression.jar. >>> >>> >>> >>> At 2014-11-18 15:49:53, "lulynn_2008" <[email protected]> wrote: >>> >>> Hi All, >>> I am trying to use CMX as temp file compression >>> >>> codec, i.e >>> SET pig.tmpfilecompression true; >>> SET pig.tmpfilecompression.codec cmx; >>> >>> but following errors happened: >>> Caused by: java.io.IOException: Invalid temporary file compression codec >>> []. Expected compression codecs are gz and lzo >>> >>> from pig cookbook I found following line >>> >>> "pig.tmpfilecompression.codec - Specifies which compression codec to use. >>> Currently, Pig accepts "gz" and "lzo" as possible values. However, because >>> LZO is under GPL license (and disabled by default) you will need to >>> configure your cluster to use the LZO codec to take advantage of this >>> feature. For details, see >>> http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ. " >>> >>> >>> Is there any workaround? or there are roadmaps for adding cmx as a >>> >>> supported codec ? I was using pig 0.12.0. >>> >>> Thanks >>> >>> >>> >>> >>
