Nikhil S created FLINK-41577:
--------------------------------

             Summary: `NoFetchingInput` reads ahead via `Input#optional` and 
bypasses buffered bytes in `readBytes/skip/read(byte[])`, corrupting Kryo 
deserialization
                 Key: FLINK-41577
                 URL: https://issues.apache.org/jira/browse/FLINK-41577
             Project: Flink
          Issue Type: Bug
          Components: API / Type Serialization System
    Affects Versions: 2.3.0, 2.2.0
            Reporter: Nikhil S
         Attachments: NoFetchingInputRepro.java, NoFetchingInputTest.patch

{{NoFetchingInput}} must not read past what Kryo requests, because callers 
share the underlying stream and treat its position as the end of the record.
h3. Two problems:
 # It does not override {{{}Input#optional(int){}}}. Kryo's bulk reads for 
{{{}short[]{}}}, {{{}char[]{}}}, {{float[]}} (and other fixed-width arrays) 
call {{{}optional(){}}}, which fills the whole 8-byte buffer. This reads ahead 
of the requested bytes and leaves them in Kryo's buffer.
 # {{{}readBytes(byte[],int,int){}}}, {{skip(int)}} and 
{{read(byte[],int,int)}} read straight from the stream and ignore bytes already 
buffered, so those bytes are skipped or returned out of order. {{require()}} 
can also pass a zero or negative {{count}} to {{fill()}} when the buffer is 
partly consumed.

h3. Effects: 

A type with a {{char[]}} field followed by a {{byte[]}} field deserializes 
silently wrong or throws. In a framed stream the position ends past the record 
boundary, which breaks the next record's framing.
h3. Reproducer:

{{NoFetchingInputRepro.java}} (standalone, Kryo 5.6.2 with Flink's 
{{{}NoFetchingInput{}}}).
  (1) One object of 36 bytes consumes 37 bytes from the stream.
  (2) 481 of 1560 char[]-then-byte[] combinations fail; the first is 
pre.length=0 a.length=1 b.length=1 -> EOFException: No more bytes left.
It is deterministic: single-threaded, and it depends only on the bytes and the 
buffer offsets.

Not reproduced with Kryo 2.24 (Flink 1.x) using the same cases.
h3. Suggested fix:

Override {{optional()}} so it never prefetches. Make {{{}readBytes{}}}, 
{{skip}} and {{read}} drain the buffer first, then bulk-read the remainder from 
the stream. (Falling through to Kryo's defaults reads in 8-byte chunks, about 
25x slower for large {{byte[]}} in my benchmark; see also 
[FLINK-37435|https://issues.apache.org/jira/browse/FLINK-37435?].)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to