GitHub user maropu opened a pull request:

    [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)

    ## What changes were proposed in this pull request?
    This pr updated `lz4-java` to the latest (v1.4.0) and removed custom 
`LZ4BlockInputStream`. We currently use custom `LZ4BlockInputStream` to read 
concatenated byte stream in shuffle. But, this functionality has been 
implemented in the latest lz4-java ( 
So, we might update the latest to remove the custom `LZ4BlockInputStream`.
    Major diffs between the latest release and v1.3.0 in the master are as 
    - fixed NPE in XXHashFactory similarly
    - Don't place resources in default package to support shading
    - Fixes ByteBuffer methods failing to apply arrayOffset() for array-backed
    - Try to load lz4-java from java.library.path, then fallback to bundled
    - Add ppc64le binary
    - Add s390x JNI binding
    - Add basic LZ4 Frame v1.5.0 support
    - enable aarch64 support for lz4-java
    - Allow unsafeInstance() for ppc64le archiecture
    - Add unsafeInstance support for AArch64
    - Support 64-bit JNI build on Solaris
    - Avoid over-allocating a buffer
    - Allow EndMark to be incompressible for LZ4FrameInputStream.
    - Concat byte stream
    ## How was this patch tested?
    Existing tests.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-21276

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18883
commit 531a2166114b2a363a26dbe864a76f01e2d4ce23
Author: Takeshi Yamamuro <>
Date:   2017-08-08T12:09:20Z

    Update lz4-java to the latest (v1.4.0)


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to