cheeeee opened a new pull request, #23928:
URL: https://github.com/apache/kafka/pull/23928
Follow-up to #23445, under the same ticket (resolved, fix version
4.5.0).
Since #23445, a failed send of fetched records is passed to a handler
that re-reads the region on the scheduler, and the log directory is
offlined only if the re-read fails too. The re-read is needed because
over PLAINTEXT `sendfile` reads the file and writes the socket in one
call, so a disk error and a socket error are the same bare
`IOException`. `SslTransportLayer.transferFrom` does not have that
problem: it reads the file with `FileChannel.read`, then encrypts and
writes, so it knows which of the two failed. As discussed on #23445,
this passes that on instead of finding it out again:
* `SslTransportLayer.transferFrom` throws a new `FileReadException`, an
`IOException` with the position of the read, when `FileChannel.read`
or `FileChannel.size` fails. A `ClosedChannelException`, from a
deletion or a shutdown, is rethrown as is.
* `TransferableChannel.reportsFileReadFailures()` says whether a
transport does that: true for TLS, false, the default, for
PLAINTEXT.
* `FileRecords.writeTo` passes a `FileReadException` to the handler,
and any other failure only if the transport cannot tell the two
apart. A failure of its own `channel.size()` call before the transfer
becomes a `FileReadException` as well.
* `LocalLog` reports a `FileReadException` as a failure of the log
directory without re-reading the region, on the scheduler and at most
one report at a time per log directory, as for the re-read. Reports
are tracked apart from re-reads, so that a re-read running in the
same directory does not drop a known failure. Other failures are
re-read as before.
Consequences worth stating:
* On TLS listeners a disconnect no longer re-reads the region that was
being sent. In the test notes, consumers killed mid-fetch over TLS
for 5 minutes made the scheduler re-read 151 MB without this change
and nothing with it.
* On TLS listeners a read error now offlines the log directory even if
the region could be read again by the time of a re-read, e.g. after a
transient media error. Before, such an error was attributed to the
socket. This is the criterion `LocalLog` applies to any other
`IOException`.
* A failed `FileChannel.size()` call on the segment, over either
transport, is now reported as a read failure without a re-read.
* PLAINTEXT listeners still re-read: `sendfile` cannot tell the two
apart.
* KRaft snapshots, sent through `UnalignedFileRecords`, are still not
covered.
Generated-by: Claude (Anthropic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
