junrao commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4316498614
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
@Override
public int writeTo(TransferableChannel destChannel, int offset, int
length) throws IOException {
- long newSize = Math.min(channel.size(), end) - start;
int oldSize = sizeInBytes();
- if (newSize < oldSize)
- throw new KafkaException(String.format(
- "Size of FileRecords %s has been truncated during write:
old size %d, new size %d",
- file.getAbsolutePath(), oldSize, newSize));
-
long position = start + offset;
int count = Math.min(length, oldSize - offset);
- // safe to cast to int since `count` is an int
- return (int) destChannel.transferFrom(channel, position, count);
+ try {
+ long newSize = Math.min(channel.size(), end) - start;
+ if (newSize < oldSize)
+ throw new KafkaException(String.format(
+ "Size of FileRecords %s has been truncated during
write: old size %d, new size %d",
+ file.getAbsolutePath(), oldSize, newSize));
+ // safe to cast to int since `count` is an int
+ return (int) destChannel.transferFrom(channel, position, count);
+ } catch (IOException e) {
+ // The transfer reads the file and writes the socket in one call,
so a storage failure and a socket
+ // failure both arrive here as a bare IOException. Let the handler
find out which one it was. A closed
+ // channel means the file was not read, so there is nothing to
find out.
+ if (transferFailureHandler != null && !(e instanceof
ClosedChannelException))
+ transferFailureHandler.onTransferFailure(this, position,
count, e);
+ throw e;
+ }
+ }
+
+ /**
+ * Read the given region of the backing file, after {@link #writeTo}
failed on it, to tell a storage failure from
Review Comment:
Claude suggested the following optimzation.
```
1. Re-read only the first chunk (FileRecords.readFailure). With both
transports, an IOException only escapes from a transferFrom call that sent
nothing:
SSL: explicitly, through if (totalBytesWritten > 0) return
totalBytesWritten;.
Plaintext: sendfile reports partial progress before an error.
So the bytes that failed to read are at position, within the first
REREAD_BUFFER_SIZE bytes. Reading only that much:
cuts the extra disk reads caused by ordinary disconnects, which are most
transfer failures;
shortens how long a slow or hung disk holds a scheduler thread;
keeps the per-directory slot free more often, so a real failure elsewhere in
the directory is less likely to be skipped while a harmless check runs. That
was the reviewer's finding 1. It's still best effort, but the client retrying
the bad offset gives another chance.
If you do this, update the Javadoc, which says "the transfer may fail
anywhere within it without reporting progress". I haven't checked every
fallback path inside the JDK's transferTo. The buffered fallback reports
partial progress too, and the mmap one fails with an InternalError rather than
an IOException, so it never reaches this handler. A quick check before relying
on this would be worthwhile.
```
Gemini seems to confirm the validity of this optimization.
```
When transferTo() throws an IOException due to a disk issue (such as a bad
sector or read error), the physical defect is located within the very first OS
page or disk block (typically 4KB to 8KB) starting at the position provided in
that specific method call.
It does not mean the error occurred at the exact single byte of position,
but it does mean the error is at the immediate start of the requested read
chunk.
Here is how the underlying mechanics dictate this behavior:
1. Partial Transfers Prevent "Future" Exceptions
If you request a large transfer (e.g., 1GB) and the disk has a bad sector
500MB into the file, transferTo will not throw an exception on your initial
call.
Instead, the underlying OS system call (like Linux sendfile or macOS
copyfile) will successfully transfer the healthy data up to the error, and Java
will return the number of successfully transferred bytes (e.g., 524288000).
You are then expected to loop and call transferTo again with the position
updated to the new offset. Only on this subsequent call, where the bad sector
is now sitting immediately at the new position, will the OS fail immediately
and cause Java to throw the IOException.
2. Disk Block Resolution
When the exception is finally thrown, the exact corrupted byte might not be
the very first byte at position because storage hardware and operating systems
do not read files byte-by-byte. They read in blocks or memory pages (usually
4,096 bytes).
If position points to the start of a 4KB block, and the disk hardware
encounters a cyclic redundancy check (CRC) failure at the 2,000th byte of that
block, the hardware fails the read for the entire block. The OS returns an EIO
(Input/Output Error) to Java, which throws the exception. The error is
somewhere inside that first block.
State of the Transfer upon Exception
Because of how OS-level POSIX I/O operates, if a transferTo call throws an
IOException due to a disk read error:
Previous calls: Any bytes counted in the return values of previous
transferTo calls were successfully read and sent to the target.
Current call: The target WritableByteChannel received exactly zero bytes
from the specific transferTo call that threw the exception.
Location: The disk failure is guaranteed to be within the first hardware
block (usually within 4KB) of the position argument that caused the crash.
```
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
@Override
public int writeTo(TransferableChannel destChannel, int offset, int
length) throws IOException {
- long newSize = Math.min(channel.size(), end) - start;
int oldSize = sizeInBytes();
- if (newSize < oldSize)
- throw new KafkaException(String.format(
- "Size of FileRecords %s has been truncated during write:
old size %d, new size %d",
- file.getAbsolutePath(), oldSize, newSize));
-
long position = start + offset;
int count = Math.min(length, oldSize - offset);
- // safe to cast to int since `count` is an int
- return (int) destChannel.transferFrom(channel, position, count);
+ try {
+ long newSize = Math.min(channel.size(), end) - start;
+ if (newSize < oldSize)
+ throw new KafkaException(String.format(
+ "Size of FileRecords %s has been truncated during
write: old size %d, new size %d",
+ file.getAbsolutePath(), oldSize, newSize));
+ // safe to cast to int since `count` is an int
+ return (int) destChannel.transferFrom(channel, position, count);
+ } catch (IOException e) {
+ // The transfer reads the file and writes the socket in one call,
so a storage failure and a socket
+ // failure both arrive here as a bare IOException. Let the handler
find out which one it was. A closed
+ // channel means the file was not read, so there is nothing to
find out.
+ if (transferFailureHandler != null && !(e instanceof
ClosedChannelException))
+ transferFailureHandler.onTransferFailure(this, position,
count, e);
+ throw e;
+ }
+ }
+
+ /**
+ * Read the given region of the backing file, after {@link #writeTo}
failed on it, to tell a storage failure from
+ * a socket failure. The whole region is read rather than its first byte
because, depending on the platform, the
+ * transfer may fail anywhere within it without reporting progress. Only a
failed read of an open channel counts
+ * as a storage failure: the end of the file may be reached because the
log was truncated concurrently (e.g. when
+ * the replica becomes a follower), and the channel may have been closed
concurrently, by a deletion or a
+ * shutdown; neither says anything about the medium.
+ *
+ * @param position The position of the region in the file
Review Comment:
Probably worth adding that this is an absolute file position.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +528,48 @@ else if (segment.baseOffset() ==
maxOffsetMetadata.segmentBaseOffset && !maxOffs
);
}
+ /**
+ * The records of a fetch are read from the segment only when they are
written to the network, after this log
+ * has returned them, so an IOException from that read cannot be caught
here. Have the records report a failed
+ * write instead, so that the log directory is marked offline if the
segment cannot be read, as for any other
+ * IOException in this log.
+ */
+ private FetchDataInfo withTransferFailureHandler(FetchDataInfo
fetchDataInfo) {
+ if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+ return fetchDataInfo;
+ return
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+ }
+
+ /**
+ * Called on the network thread when writing fetched records failed, which
may be a socket or a storage failure.
+ * Re-reading the region tells them apart, so it is done on the scheduler
rather than on the network thread, at
+ * most once at a time per log dir, so that a burst of failed sends or a
slow failing disk does not take up the
+ * threads of the scheduler, which other tasks of the broker share.
+ */
+ private void checkReadAfterTransferFailure(FileRecords records, long
position, int count, IOException transferFailure) {
Review Comment:
Claude suggested the following change.
```
2. Clear the per-directory flag in one place
(LocalLog.checkReadAfterTransferFailure). finishReadCheck is called in three
places, and an Error thrown by scheduleOnce skips all of them, leaving the
directory unchecked until a restart. An example is OutOfMemoryError: unable to
create native thread when the executor starts a worker. Use a try/finally with
a hand-off flag instead:
if (!logDirFailureChannel.tryStartReadCheck(logDir))
return;
boolean handedOff = false;
try {
ScheduledFuture<?> check =
scheduler.scheduleOnce("check-read-after-transfer-failure", () -> {
try {
records.readFailure(position, count).ifPresent(...);
} finally {
logDirFailureChannel.finishReadCheck(logDir);
}
});
// A scheduler that is not running ignores the task
handedOff = check != null && !check.isCancelled();
} catch (RuntimeException e) {
logger.warn("Could not schedule a read check of {} after a failed
transfer", records.file().getAbsolutePath(), e);
} finally {
if (!handedOff)
logDirFailureChannel.finishReadCheck(logDir);
}
```
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
* instance to enable slicing a range of the log records.
*/
public class FileRecords extends AbstractRecords implements Closeable {
+ // Size of the buffer used to re-read a region that failed to transfer,
see readFailure
+ static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+ /**
+ * Notified when {@link #writeTo} fails, typically to find out with {@link
#readFailure} whether the backing file
+ * could not be read. It is called on the thread that failed to write, a
network thread, so it must not block.
+ */
+ @FunctionalInterface
+ public interface TransferFailureHandler {
Review Comment:
The plan sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]