[
https://issues.apache.org/jira/browse/KAFKA-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao resolved KAFKA-21280.
-----------------------------
Fix Version/s: 4.5.0
Resolution: Fixed
merged the PR to trunk
> SslTransportLayer.transferFrom sends stale buffer contents after a failed
> file read
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-21280
> URL: https://issues.apache.org/jira/browse/KAFKA-21280
> Project: Kafka
> Issue Type: Bug
> Components: network
> Affects Versions: 2.5.0, 4.3.1
> Reporter: Mykyta Bozhenko
> Assignee: Mykyta Bozhenko
> Priority: Major
> Fix For: 4.5.0
>
>
> When TLS is enabled, {{SslTransportLayer.transferFrom}} copies records from
> the segment into {{fileChannelBuffer}} and writes that buffer to the TLS
> stream. The buffer is cleared before each read:
> {code:java}
> if (!fileChannelBuffer.hasRemaining()) {
> fileChannelBuffer.clear();
> ...
> int bytesRead = fileChannel.read(fileChannelBuffer, pos);
> if (bytesRead <= 0)
> break;
> fileChannelBuffer.flip();
> }
> int networkBytesWritten = write(fileChannelBuffer);
> {code}
> If the read throws, or returns no bytes, the buffer is left cleared instead
> of empty: {{hasRemaining()}} is true and the buffer still holds the previous
> chunk. When something was already written in the same call, the
> {{IOException}} is swallowed ({{if (totalBytesWritten > 0) return
> totalBytesWritten}}). The next call then skips the read, writes the stale
> buffer to the socket as the next chunk of the file, advances the position,
> and fails on the following read the same way. A read error on the segment
> never surfaces; the rest of the requested region is sent as copies of an
> earlier chunk.
> Reproduced on trunk (03e6606430): one controller and three brokers, JDK 17,
> Linux 6.8, the leader's log directory on {{dm-dust}}. The tail of a 10 MiB
> segment is made unreadable while its head stays in the page cache, then a
> consumer reads the partition from the beginning over the SSL listener. On the
> broker's network thread:
> {code}
> pread64(176, ..., 32768, 2850816) = 16384
> pread64(176, ..., 32768, 2867200) = -1 EIO (Input/output error)
> {code}
> followed by 168 more {{-1 EIO}} at offsets advancing by 32768, covering the
> whole 5.5 MiB region. The broker logs nothing about it: no exception, no
> disconnect, the log directory stays online and the leader is unchanged. The
> consumer gets 5456 of 20000 records, then:
> {code}
> Record batch for partition t1-0 at offset 5456 is invalid, cause: Record is
> corrupt (stored crc = 1243940787, computed crc = 114489509)
> {code}
> The same scenario over the PLAINTEXT listener surfaces the error:
> {{sendfile}} returns {{EIO}} and the broker closes the connection with
> {{java.io.IOException: Input/output error}}.
> Impact:
> * Clients and followers on a TLS listener receive bytes that are not in the
> file. The CRC check catches it, so a readable record on healthy replicas is
> reported as corrupt, and the partition cannot be consumed past it from this
> leader.
> * Disk read errors on TLS listeners never reach the code that handles I/O
> errors, so KAFKA-16304 cannot work for TLS until this is fixed.
> * The stale bytes come from an earlier transfer on the same connection, so
> nothing from another connection is exposed.
> The {{bytesRead <= 0}} branch leaves the buffer in the same state;
> {{FileRecords.writeTo}}'s size check usually rejects a truncated file before
> that is reached.
> Proposed fix: leave the buffer empty when a read does not succeed (for
> example set its limit to 0 when the read throws or returns no bytes), so the
> next call reads again and the original {{IOException}} propagates.
> Disclosure per the ASF generative tooling guidance: the reproduction and the
> analysis were produced with AI assistance; I have reviewed and am responsible
> for all of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
