junrao commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4316498614


##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
 
     @Override
     public int writeTo(TransferableChannel destChannel, int offset, int 
length) throws IOException {
-        long newSize = Math.min(channel.size(), end) - start;
         int oldSize = sizeInBytes();
-        if (newSize < oldSize)
-            throw new KafkaException(String.format(
-                    "Size of FileRecords %s has been truncated during write: 
old size %d, new size %d",
-                    file.getAbsolutePath(), oldSize, newSize));
-
         long position = start + offset;
         int count = Math.min(length, oldSize - offset);
-        // safe to cast to int since `count` is an int
-        return (int) destChannel.transferFrom(channel, position, count);
+        try {
+            long newSize = Math.min(channel.size(), end) - start;
+            if (newSize < oldSize)
+                throw new KafkaException(String.format(
+                        "Size of FileRecords %s has been truncated during 
write: old size %d, new size %d",
+                        file.getAbsolutePath(), oldSize, newSize));
+            // safe to cast to int since `count` is an int
+            return (int) destChannel.transferFrom(channel, position, count);
+        } catch (IOException e) {
+            // The transfer reads the file and writes the socket in one call, 
so a storage failure and a socket
+            // failure both arrive here as a bare IOException. Let the handler 
find out which one it was. A closed
+            // channel means the file was not read, so there is nothing to 
find out.
+            if (transferFailureHandler != null && !(e instanceof 
ClosedChannelException))
+                transferFailureHandler.onTransferFailure(this, position, 
count, e);
+            throw e;
+        }
+    }
+
+    /**
+     * Read the given region of the backing file, after {@link #writeTo} 
failed on it, to tell a storage failure from

Review Comment:
   Claude suggested the following optimzation.
   
   ```
   1. Re-read only the first chunk (FileRecords.readFailure). With both 
transports, an IOException only escapes from a transferFrom call that sent 
nothing:
   
   SSL: explicitly, through if (totalBytesWritten > 0) return 
totalBytesWritten;.
   Plaintext: sendfile reports partial progress before an error.
   So the bytes that failed to read are at position, within the first 
REREAD_BUFFER_SIZE bytes. Reading only that much:
   
   cuts the extra disk reads caused by ordinary disconnects, which are most 
transfer failures;
   shortens how long a slow or hung disk holds a scheduler thread;
   keeps the per-directory slot free more often, so a real failure elsewhere in 
the directory is less likely to be skipped while a harmless check runs. That 
was the reviewer's finding 1. It's still best effort, but the client retrying 
the bad offset gives another chance.
   If you do this, update the Javadoc, which says "the transfer may fail 
anywhere within it without reporting progress". I haven't checked every 
fallback path inside the JDK's transferTo. The buffered fallback reports 
partial progress too, and the mmap one fails with an InternalError rather than 
an IOException, so it never reaches this handler. A quick check before relying 
on this would be worthwhile.
   ```
   
   Gemini seems to confirm the validity of this optimization.
   
   ```
   When transferTo() throws an IOException due to a disk issue (such as a bad 
sector or read error), the physical defect is located within the very first OS 
page or disk block (typically 4KB to 8KB) starting at the position provided in 
that specific method call.
   
   It does not mean the error occurred at the exact single byte of position, 
but it does mean the error is at the immediate start of the requested read 
chunk.
   
   Here is how the underlying mechanics dictate this behavior:
   
   1. Partial Transfers Prevent "Future" Exceptions
   If you request a large transfer (e.g., 1GB) and the disk has a bad sector 
500MB into the file, transferTo will not throw an exception on your initial 
call.
   
   Instead, the underlying OS system call (like Linux sendfile or macOS 
copyfile) will successfully transfer the healthy data up to the error, and Java 
will return the number of successfully transferred bytes (e.g., 524288000).
   
   You are then expected to loop and call transferTo again with the position 
updated to the new offset. Only on this subsequent call, where the bad sector 
is now sitting immediately at the new position, will the OS fail immediately 
and cause Java to throw the IOException.
   
   2. Disk Block Resolution
   When the exception is finally thrown, the exact corrupted byte might not be 
the very first byte at position because storage hardware and operating systems 
do not read files byte-by-byte. They read in blocks or memory pages (usually 
4,096 bytes).
   
   If position points to the start of a 4KB block, and the disk hardware 
encounters a cyclic redundancy check (CRC) failure at the 2,000th byte of that 
block, the hardware fails the read for the entire block. The OS returns an EIO 
(Input/Output Error) to Java, which throws the exception. The error is 
somewhere inside that first block.
   
   State of the Transfer upon Exception
   Because of how OS-level POSIX I/O operates, if a transferTo call throws an 
IOException due to a disk read error:
   
   Previous calls: Any bytes counted in the return values of previous 
transferTo calls were successfully read and sent to the target.
   
   Current call: The target WritableByteChannel received exactly zero bytes 
from the specific transferTo call that threw the exception.
   
   Location: The disk failure is guaranteed to be within the first hardware 
block (usually within 4KB) of the position argument that caused the crash.
   ```
   



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
 
     @Override
     public int writeTo(TransferableChannel destChannel, int offset, int 
length) throws IOException {
-        long newSize = Math.min(channel.size(), end) - start;
         int oldSize = sizeInBytes();
-        if (newSize < oldSize)
-            throw new KafkaException(String.format(
-                    "Size of FileRecords %s has been truncated during write: 
old size %d, new size %d",
-                    file.getAbsolutePath(), oldSize, newSize));
-
         long position = start + offset;
         int count = Math.min(length, oldSize - offset);
-        // safe to cast to int since `count` is an int
-        return (int) destChannel.transferFrom(channel, position, count);
+        try {
+            long newSize = Math.min(channel.size(), end) - start;
+            if (newSize < oldSize)
+                throw new KafkaException(String.format(
+                        "Size of FileRecords %s has been truncated during 
write: old size %d, new size %d",
+                        file.getAbsolutePath(), oldSize, newSize));
+            // safe to cast to int since `count` is an int
+            return (int) destChannel.transferFrom(channel, position, count);
+        } catch (IOException e) {
+            // The transfer reads the file and writes the socket in one call, 
so a storage failure and a socket
+            // failure both arrive here as a bare IOException. Let the handler 
find out which one it was. A closed
+            // channel means the file was not read, so there is nothing to 
find out.
+            if (transferFailureHandler != null && !(e instanceof 
ClosedChannelException))
+                transferFailureHandler.onTransferFailure(this, position, 
count, e);
+            throw e;
+        }
+    }
+
+    /**
+     * Read the given region of the backing file, after {@link #writeTo} 
failed on it, to tell a storage failure from
+     * a socket failure. The whole region is read rather than its first byte 
because, depending on the platform, the
+     * transfer may fail anywhere within it without reporting progress. Only a 
failed read of an open channel counts
+     * as a storage failure: the end of the file may be reached because the 
log was truncated concurrently (e.g. when
+     * the replica becomes a follower), and the channel may have been closed 
concurrently, by a deletion or a
+     * shutdown; neither says anything about the medium.
+     *
+     * @param position The position of the region in the file

Review Comment:
   Probably worth adding that this is an absolute file position.



##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +528,48 @@ else if (segment.baseOffset() == 
maxOffsetMetadata.segmentBaseOffset && !maxOffs
         );
     }
 
+    /**
+     * The records of a fetch are read from the segment only when they are 
written to the network, after this log
+     * has returned them, so an IOException from that read cannot be caught 
here. Have the records report a failed
+     * write instead, so that the log directory is marked offline if the 
segment cannot be read, as for any other
+     * IOException in this log.
+     */
+    private FetchDataInfo withTransferFailureHandler(FetchDataInfo 
fetchDataInfo) {
+        if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+            return fetchDataInfo;
+        return 
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+    }
+
+    /**
+     * Called on the network thread when writing fetched records failed, which 
may be a socket or a storage failure.
+     * Re-reading the region tells them apart, so it is done on the scheduler 
rather than on the network thread, at
+     * most once at a time per log dir, so that a burst of failed sends or a 
slow failing disk does not take up the
+     * threads of the scheduler, which other tasks of the broker share.
+     */
+    private void checkReadAfterTransferFailure(FileRecords records, long 
position, int count, IOException transferFailure) {

Review Comment:
   Claude suggested the following change.
   
   ```
   2. Clear the per-directory flag in one place 
(LocalLog.checkReadAfterTransferFailure). finishReadCheck is called in three 
places, and an Error thrown by scheduleOnce skips all of them, leaving the 
directory unchecked until a restart. An example is OutOfMemoryError: unable to 
create native thread when the executor starts a worker. Use a try/finally with 
a hand-off flag instead:
   
   
   
   if (!logDirFailureChannel.tryStartReadCheck(logDir))
       return;
   boolean handedOff = false;
   try {
       ScheduledFuture<?> check = 
scheduler.scheduleOnce("check-read-after-transfer-failure", () -> {
           try {
               records.readFailure(position, count).ifPresent(...);
           } finally {
               logDirFailureChannel.finishReadCheck(logDir);
           }
       });
       // A scheduler that is not running ignores the task
       handedOff = check != null && !check.isCancelled();
   } catch (RuntimeException e) {
       logger.warn("Could not schedule a read check of {} after a failed 
transfer", records.file().getAbsolutePath(), e);
   } finally {
       if (!handedOff)
           logDirFailureChannel.finishReadCheck(logDir);
   }
   
   ```



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
  * instance to enable slicing a range of the log records.
  */
 public class FileRecords extends AbstractRecords implements Closeable {
+    // Size of the buffer used to re-read a region that failed to transfer, 
see readFailure
+    static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+    /**
+     * Notified when {@link #writeTo} fails, typically to find out with {@link 
#readFailure} whether the backing file
+     * could not be read. It is called on the thread that failed to write, a 
network thread, so it must not block.
+     */
+    @FunctionalInterface
+    public interface TransferFailureHandler {

Review Comment:
   The plan sounds good to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to