[
https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344854#comment-14344854
]
fang fang chen commented on HADOOP-11418:
-----------------------------------------
Sure, thanks for looking at this JIRA, Akira. I generated new patch based on
current trunk. And the regression test has been made with pig before, here is
the process and log:
1. Add following settings in hadoop core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
2. Launch pig grunt with properties:
-Dpig.tmpfilecompression=true -Dpig.tmpfilecompression.codec=cmx
-Dpig.additional.jars=ibm-compression.jar:libcmxbiginsights.so
Together with patch in PIG-4341, test case passed with following massage:
2014-12-22 17:37:24,453 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 2
2014-12-22 17:37:24,550 [main] INFO org.apache.hadoop.io.file.tfile.Compression
- Trying to load Lzo codec class: com.ibm.biginsights.compress.CmxCodec
2014-12-22 17:37:24,554 [main] INFO
com.ibm.biginsights.compress.NativeCodeLoader - Loaded the native library
2014-12-22 17:37:24,554 [main] INFO org.apache.hadoop.io.compress.CodecPool -
Got brand-new decompressor [.cmx]
> Property "io.compression.codec.lzo.class" does not work with other value
> besides default
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-11418
> URL: https://issues.apache.org/jira/browse/HADOOP-11418
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: fang fang chen
> Assignee: fang fang chen
> Attachments: HADOOP-11418-1.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for
> other codec besides default. Hadoop will always treat it as defaultClazz. I
> think it is a bug. Please let me know if this is a work as design thing.
> Thanks
> 77 private static final String defaultClazz =
> 78 "org.apache.hadoop.io.compress.LzoCodec";
> 82 public synchronized boolean isSupported() {
> 83 if (!checked) {
> 84 checked = true;
> 85 String extClazz =
> 86 (conf.get(CONF_LZO_CLASS) == null ? System
> 87 .getProperty(CONF_LZO_CLASS) : null);
> 88 String clazz = (extClazz != null) ? extClazz : defaultClazz;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
