[
https://issues.apache.org/jira/browse/HADOOP-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236744#comment-17236744
]
Ayush Saxena commented on HADOOP-17292:
---------------------------------------
This seems to break {{TestFSImage.testNativeCompression}}
[https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/305/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestFSImage/testNativeCompression/]
{noformat}
java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:67)
at
org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
at
org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131){noformat}
Looks could be because of this.
[~csun]/[~viirya] can you guys give a check...
> Using lz4-java in Lz4Codec
> --------------------------
>
> Key: HADOOP-17292
> URL: https://issues.apache.org/jira/browse/HADOOP-17292
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
> Affects Versions: 3.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for lz4 codec which has several disadvantages:
> It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and
> they have to be installed separately on each node of the clusters, container
> images, or local test environments which adds huge complexities from
> deployment point of view. In some environments, it requires compiling the
> natives from sources which is non-trivial. Also, this approach is platform
> dependent; the binary may not work in different platform, so it requires
> recompilation.
> It requires extra configuration of java.library.path to load the natives, and
> it results higher application deployment and maintenance cost for users.
> Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which
> is JNI-based implementation. It contains native binaries in jar file, and it
> can automatically load the native binaries into JVM from jar without any
> setup. If a native implementation can not be found for a platform, it can
> fallback to pure-java implementation of lz4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
