[
https://issues.apache.org/jira/browse/HADOOP-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-15822:
--------------------------------
Status: Patch Available (was: Open)
Compression flushing failure has to do with how the JNI wrapper code was
invoking the zstd library. When using a small output buffer sometimes
flushStream or endStream needs to be called successively to finish flushing
everything, but the JNI code would always invoke the compressStream method on a
null input buffer before invoking the flush/end call. Older versions of zstd
apparently were OK with this, but the new ones are not. This patch skips
calling compressStream if there is nothing in the input buffer to compress, so
the zstd library will see a contiguous sequence of end stream calls towards the
end of compression when using small output buffers.
The decompress direct test failure is a bug in the interface between the Java
layer and the JNI layer. The function takes a buffer pointer, a buffer length,
and a buffer offset, as arguments but the Java layer was using remaining()
instead of limit() to send down the size of the buffer. Occasionally during
the test remaining() can be smaller than position() and the zstd library
rightfully complains that we are asking it to use a buffer past the end of the
reported length. In addition the test would sometimes fail to flip the output
buffer which would break the test when that occurs.
These tests also were not running during precommit because the zstandard
libraries were missing from the build environment, so this patch adds the
libzstd package to the build environment Dockerfile.
> zstd compressor can fail with a small output buffer
> ---------------------------------------------------
>
> Key: HADOOP-15822
> URL: https://issues.apache.org/jira/browse/HADOOP-15822
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.0.0, 2.9.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Major
> Attachments: HADOOP-15822.001.patch
>
>
> TestZStandardCompressorDecompressor fails a couple of tests on my machine
> with the latest zstd library (1.3.5). Compression can fail to successfully
> finalize the stream when a small output buffer is used resulting in a failed
> to init error, and decompression with a direct buffer can fail with an
> invalid src size error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
