Bryan Bende created NIFI-3191:
---------------------------------
Summary: HDFS Processors Should Allow Choosing LZO Compression
Key: NIFI-3191
URL: https://issues.apache.org/jira/browse/NIFI-3191
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 0.7.1, 1.1.0
Reporter: Bryan Bende
The HDFS processors have a CompressionType enum that maps the enum to a class
name, currently it has the following:
{code}
case NONE: return "NONE";
case DEFAULT: return DefaultCodec.class.getName();
case BZIP: return BZip2Codec.class.getName();
case GZIP: return GzipCodec.class.getName();
case LZ4: return Lz4Codec.class.getName();
case SNAPPY: return SnappyCodec.class.getName();
case AUTOMATIC: return "Automatically Detected";
{code}
The Lzo codec class is GPL (http://code.google.com/p/hadoop-gpl-compression and
https://github.com/twitter/hadoop-lzo) so it can not be shipped with NiFi or
Hadoop, but if someone adds it to their Hadoop distribution there should be a
way to select it from NiFi.
Seems LZO could be added to the enumeration with a mapping of:
{code}
case LZO: return "com.hadoop.compression.lzo.LzoCodec";
{code}
The libraries would still need to be linked through the LD_LIBRARY_PATH such as:
{code}
java.arg.15=-Djava.libaray.path=/path/to/your/lzocodec.so
{code}
and possibly point the additional classpath resource to the jar containing
LzoCodec.java.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
