Nikhil S created FLINK-41577:
--------------------------------
Summary: `NoFetchingInput` reads ahead via `Input#optional` and
bypasses buffered bytes in `readBytes/skip/read(byte[])`, corrupting Kryo
deserialization
Key: FLINK-41577
URL: https://issues.apache.org/jira/browse/FLINK-41577
Project: Flink
Issue Type: Bug
Components: API / Type Serialization System
Affects Versions: 2.3.0, 2.2.0
Reporter: Nikhil S
Attachments: NoFetchingInputRepro.java, NoFetchingInputTest.patch
{{NoFetchingInput}} must not read past what Kryo requests, because callers
share the underlying stream and treat its position as the end of the record.
h3. Two problems:
# It does not override {{{}Input#optional(int){}}}. Kryo's bulk reads for
{{{}short[]{}}}, {{{}char[]{}}}, {{float[]}} (and other fixed-width arrays)
call {{{}optional(){}}}, which fills the whole 8-byte buffer. This reads ahead
of the requested bytes and leaves them in Kryo's buffer.
# {{{}readBytes(byte[],int,int){}}}, {{skip(int)}} and
{{read(byte[],int,int)}} read straight from the stream and ignore bytes already
buffered, so those bytes are skipped or returned out of order. {{require()}}
can also pass a zero or negative {{count}} to {{fill()}} when the buffer is
partly consumed.
h3. Effects:
A type with a {{char[]}} field followed by a {{byte[]}} field deserializes
silently wrong or throws. In a framed stream the position ends past the record
boundary, which breaks the next record's framing.
h3. Reproducer:
{{NoFetchingInputRepro.java}} (standalone, Kryo 5.6.2 with Flink's
{{{}NoFetchingInput{}}}).
(1) One object of 36 bytes consumes 37 bytes from the stream.
(2) 481 of 1560 char[]-then-byte[] combinations fail; the first is
pre.length=0 a.length=1 b.length=1 -> EOFException: No more bytes left.
It is deterministic: single-threaded, and it depends only on the bytes and the
buffer offsets.
Not reproduced with Kryo 2.24 (Flink 1.x) using the same cases.
h3. Suggested fix:
Override {{optional()}} so it never prefetches. Make {{{}readBytes{}}},
{{skip}} and {{read}} drain the buffer first, then bulk-read the remainder from
the stream. (Falling through to Kryo's defaults reads in 8-byte chunks, about
25x slower for large {{byte[]}} in my benchmark; see also
[FLINK-37435|https://issues.apache.org/jira/browse/FLINK-37435?].)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)