Stephen Jung (Stripe) created HADOOP-17096:

             Summary: ZStandardCompressor throws java.lang.InternalError: Error 
                 Key: HADOOP-17096
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 3.2.1
         Environment: Our repro is on ubuntu xenial LTS, with hadoop 3.2.1 
linking to libzstd 1.3.1. The bug is difficult to reproduce in an end-to-end 
environment (eg running an actual hadoop job with zstd compression) because 
it's very sensitive to the exact input and output characteristics. I reproduced 
the bug by turning one of the existing unit tests into a crude fuzzer, but I'm 
not sure upstream will accept that patch, so I've attached it separately on 
this ticket.

Note that the existing unit test for testCompressingWithOneByteOutputBuffer 
fails to reproduce this bug. This is because it's using the license file as 
input, and this file is too small. libzstd has internal buffering (in our 
environment it seems to be 128 kilobytes), and the license file is only 10 
kilobytes. Thus libzstd is able to consume all the input and compress it in a 
single call, then return pieces of its internal buffer one byte at a time. 
Since all the input is consumed in a single call, uncompressedDirectBufOff and 
uncompressedDirectBufLen are both set to zero and thus the bug does not 
            Reporter: Stephen Jung (Stripe)

A bug in index handling causes ZStandardCompressor.c to pass a malformed 
ZSTD_inBuffer to libzstd. libzstd then returns an "Error (generic)" that gets 
thrown. The crux of the issue is two variables, uncompressedDirectBufLen and 
uncompressedDirectBufOff. The hadoop code counts uncompressedDirectBufOff from 
the start of uncompressedDirectBuf, then uncompressedDirectBufLen is counted 
from uncompressedDirectBufOff. However, libzstd considers pos and size to both 
be counted from the start of the buffer. As a result, this line
 causes a malformed buffer to be passed to libzstd, where pos>size. Here's a 
longer description of the bug in case this abstract explanation is unclear:


Suppose we initialize uncompressedDirectBuf (via setInputFromSavedData) with 
five bytes of input. This results in uncompressedDirectBufOff=0 and 

Then we call compress(), which initializes a ZSTD_inBuffer 
 The definition of those libzstd structs is here - note that 
we set size=uncompressedDirectBufLen and pos=uncompressedDirectBufOff. The 
ZSTD_inBuffer gets passed to libzstd, compression happens, etc. When libzstd 
returns from the compression function, it updates the ZSTD_inBuffer struct to 
indicate how many bytes were consumed 
 Note that pos is advanced, but size is unchanged.

Now, libzstd does not guarantee that the entire input will be compressed in a 
single call of the compression function. (Some of the compression libraries 
used by hadoop, such as snappy, _do_ provide this guarantee, but libzstd is not 
one of them.) So the hadoop native code updates uncompressedDirectBufOff and 
uncompressedDirectBufLen using the updated ZSTD_inBuffer:

Now, returning to our example, we started with 5 bytes of uncompressed input. 
Suppose libzstd compressed 4 of those bytes, leaving one unread. This would 
result in a ZSTD_inBuffer struct with size=5 (unchanged) and pos=4 (four bytes 
were consumed). The hadoop native code would then set 
uncompressedDirectBufOff=4, but it would also set uncompressedDirectBufLen=1 
(five minus four equals one).

Since some of the input was not consumed, we will eventually call compress() 
again. Then we instantiate another ZSTD_inBuffer struct, this time with size=1 
and pos=4. This is a bug - libzstd expects size and pos to both be counted from 
the start of the buffer, therefore pos>size is unsound. So it returns an error 
which gets escalated as a java.lang.InternalError.

I will be submitting a pull request on github with a fix for this bug. The key 
is that the hadoop code should handle offsets the same way libzstd does, ie 
uncompressedDirectBufLen should be counted from the start of 
uncompressedDirectBuf, not from uncompressedDirectBufOff.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to