[
https://issues.apache.org/jira/browse/PIG-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang fang chen updated PIG-4341:
--------------------------------
Attachment: PIG-4341.patch
1. Add following settings in hadoop core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
2. Launch pig grunt with properties:
-Dpig.tmpfilecompression=true -Dpig.tmpfilecompression.codec=cmx
-Dpig.additional.jars=ibm-compression.jar:libcmxbiginsights.so
With patch in HADOOP-11418, test case passed with following massage:
2014-12-22 17:37:24,453 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 2
2014-12-22 17:37:24,550 [main] INFO
org.apache.hadoop.io.file.tfile.Compression - Trying to load Lzo codec class:
com.ibm.biginsights.compress.CmxCodec
2014-12-22 17:37:24,554 [main] INFO
com.ibm.biginsights.compress.NativeCodeLoader - Loaded the native library
2014-12-22 17:37:24,554 [main] INFO org.apache.hadoop.io.compress.CodecPool -
Got brand-new decompressor [.cmx]
> Add CMX support to pig.tmpfilecompression.codec
> -----------------------------------------------
>
> Key: PIG-4341
> URL: https://issues.apache.org/jira/browse/PIG-4341
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.13.0
> Reporter: fang fang chen
> Fix For: 0.15.0
>
> Attachments: PIG-4341.patch
>
>
> Pig has supported compression(GZ, GZIP, LZO). But latest pig has not
> supported CMX codec yet. cmx is "com.ibm.biginsights.compress.CmxCodec". This
> information also could be found from latest release pig-0.13.0 documentation:
> http://pig.apache.org/docs/r0.13.0/perf.html.
> Besides, I once tested CMX codec with pig-0.13.0 before. Following was the
> current settings:
> SET pig.tmpfilecompression true;
> SET pig.tmpfilecompression.codec cmx;
> Error:
> Caused by: java.io.IOException: Invalid temporary file compression codec
> [cmx]. Expected compression codecs for org.apache.pig.impl.io.TFileStorage
> are GZ,GZIP,LZO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
