[
https://issues.apache.org/jira/browse/HADOOP-17125?focusedWorklogId=492567&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492567
]
ASF GitHub Bot logged work on HADOOP-17125:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 29/Sep/20 17:20
Start Date: 29/Sep/20 17:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2297:
URL: https://github.com/apache/hadoop/pull/2297#issuecomment-700859880
Having checked up on those dependencies myself, yes, we are already shipping
snappy-java 1.0.5 as a dependency of hadoop-common, by way of avro. Which makes
a strong case for keeping the snappy codec in hadoop-common, declaring the
snappy dependency as a compile time dependency, with the version we choose.
This ensures hbase &c pick it up, and, because it is there: we aren't creating
any more complications for people downstream than they get today.
@saintstack happy with that?
In which case, we actually go to the earlier patches, which just switch the
codec to using the native one, and the pom changes to import it. Plus a release
note 'you don't need native snappy no more'.
We could stay with the current code, which is resilient to someone removing
the JAR.
Good: resilient
bad: doesn't work so well if things (hadoop-client?) shade the snappy jar
references.
I don't know what to do there. Given the code is written, I'd go for "merge
it as is and see what happens", but the "go back to the minimal binding" could
well have lower maintenance costs down the line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 492567)
Time Spent: 20h 10m (was: 20h)
> Using snappy-java in SnappyCodec
> --------------------------------
>
> Key: HADOOP-17125
> URL: https://issues.apache.org/jira/browse/HADOOP-17125
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
> Affects Versions: 3.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20h 10m
> Remaining Estimate: 0h
>
> In Hadoop, we use native libs for snappy codec which has several
> disadvantages:
> * It requires native *libhadoop* and *libsnappy* to be installed in system
> *LD_LIBRARY_PATH*, and they have to be installed separately on each node of
> the clusters, container images, or local test environments which adds huge
> complexities from deployment point of view. In some environments, it requires
> compiling the natives from sources which is non-trivial. Also, this approach
> is platform dependent; the binary may not work in different platform, so it
> requires recompilation.
> * It requires extra configuration of *java.library.path* to load the
> natives, and it results higher application deployment and maintenance cost
> for users.
> Projects such as *Spark* and *Parquet* use
> [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based
> implementation. It contains native binaries for Linux, Mac, and IBM in jar
> file, and it can automatically load the native binaries into JVM from jar
> without any setup. If a native implementation can not be found for a
> platform, it can fallback to pure-java implementation of snappy based on
> [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
