Mykyta Bozhenko created KAFKA-21280:
---------------------------------------
Summary: SslTransportLayer.transferFrom sends stale buffer
contents after a failed file read
Key: KAFKA-21280
URL: https://issues.apache.org/jira/browse/KAFKA-21280
Project: Kafka
Issue Type: Bug
Components: network
Affects Versions: 4.3.1, 2.5.0
Reporter: Mykyta Bozhenko
Assignee: Mykyta Bozhenko
When TLS is enabled, {{SslTransportLayer.transferFrom}} copies records from the
segment into {{fileChannelBuffer}} and writes that buffer to the TLS stream.
The buffer is cleared before each read:
{code:java}
if (!fileChannelBuffer.hasRemaining()) {
fileChannelBuffer.clear();
...
int bytesRead = fileChannel.read(fileChannelBuffer, pos);
if (bytesRead <= 0)
break;
fileChannelBuffer.flip();
}
int networkBytesWritten = write(fileChannelBuffer);
{code}
If the read throws, or returns no bytes, the buffer is left cleared instead of
empty: {{hasRemaining()}} is true and the buffer still holds the previous
chunk. When something was already written in the same call, the {{IOException}}
is swallowed ({{if (totalBytesWritten > 0) return totalBytesWritten}}). The
next call then skips the read, writes the stale buffer to the socket as the
next chunk of the file, advances the position, and fails on the following read
the same way. A read error on the segment never surfaces; the rest of the
requested region is sent as copies of an earlier chunk.
Reproduced on trunk (03e6606430): one controller and three brokers, JDK 17,
Linux 6.8, the leader's log directory on {{dm-dust}}. The tail of a 10 MiB
segment is made unreadable while its head stays in the page cache, then a
consumer reads the partition from the beginning over the SSL listener. On the
broker's network thread:
{code}
pread64(176, ..., 32768, 2850816) = 16384
pread64(176, ..., 32768, 2867200) = -1 EIO (Input/output error)
{code}
followed by 168 more {{-1 EIO}} at offsets advancing by 32768, covering the
whole 5.5 MiB region. The broker logs nothing about it: no exception, no
disconnect, the log directory stays online and the leader is unchanged. The
consumer gets 5456 of 20000 records, then:
{code}
Record batch for partition t1-0 at offset 5456 is invalid, cause: Record is
corrupt (stored crc = 1243940787, computed crc = 114489509)
{code}
The same scenario over the PLAINTEXT listener surfaces the error: {{sendfile}}
returns {{EIO}} and the broker closes the connection with
{{java.io.IOException: Input/output error}}.
Impact:
* Clients and followers on a TLS listener receive bytes that are not in the
file. The CRC check catches it, so a readable record on healthy replicas is
reported as corrupt, and the partition cannot be consumed past it from this
leader.
* Disk read errors on TLS listeners never reach the code that handles I/O
errors, so KAFKA-16304 cannot work for TLS until this is fixed.
* The stale bytes come from an earlier transfer on the same connection, so
nothing from another connection is exposed.
The {{bytesRead <= 0}} branch leaves the buffer in the same state;
{{FileRecords.writeTo}}'s size check usually rejects a truncated file before
that is reached.
Proposed fix: leave the buffer empty when a read does not succeed (for example
set its limit to 0 when the read throws or returns no bytes), so the next call
reads again and the original {{IOException}} propagates.
Disclosure per the ASF generative tooling guidance: the reproduction and the
analysis were produced with AI assistance; I have reviewed and am responsible
for all of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
