junrao commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4288647057


##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
  * instance to enable slicing a range of the log records.
  */
 public class FileRecords extends AbstractRecords implements Closeable {
+    // Size of the buffer used to re-read a region that failed to transfer, 
see readFailure
+    static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+    /**
+     * Notified when {@link #writeTo} fails, typically to find out with {@link 
#readFailure} whether the backing file
+     * could not be read. It is called on the thread that failed to write, a 
network thread, so it must not block.
+     */
+    @FunctionalInterface
+    public interface TransferFailureHandler {
+        void onTransferFailure(FileRecords records, long position, int count, 
IOException transferFailure);
+    }
+
     private final boolean isSlice;
     private final int start;
     private final int end;
+    // Notified when writing to a channel fails, or null

Review Comment:
   channel => network channel?



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -146,7 +164,26 @@ public FileRecords slice(int position, int size) {
         int availableBytes = availableBytes(position, size);
         int startPosition = this.start + position;
 
-        return new FileRecords(file, channel, startPosition, startPosition + 
availableBytes);
+        return new FileRecords(file, channel, startPosition, startPosition + 
availableBytes, transferFailureHandler);
+    }
+
+    /**
+     * Return a copy of this slice whose {@link #writeTo} notifies the given 
handler when writing to a channel fails.

Review Comment:
   writing to a channel => writing to a network channel ?



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
 
     @Override
     public int writeTo(TransferableChannel destChannel, int offset, int 
length) throws IOException {
-        long newSize = Math.min(channel.size(), end) - start;
         int oldSize = sizeInBytes();
-        if (newSize < oldSize)
-            throw new KafkaException(String.format(
-                    "Size of FileRecords %s has been truncated during write: 
old size %d, new size %d",
-                    file.getAbsolutePath(), oldSize, newSize));
-
         long position = start + offset;
         int count = Math.min(length, oldSize - offset);
-        // safe to cast to int since `count` is an int
-        return (int) destChannel.transferFrom(channel, position, count);
+        try {
+            long newSize = Math.min(channel.size(), end) - start;
+            if (newSize < oldSize)
+                throw new KafkaException(String.format(
+                        "Size of FileRecords %s has been truncated during 
write: old size %d, new size %d",
+                        file.getAbsolutePath(), oldSize, newSize));
+            // safe to cast to int since `count` is an int
+            return (int) destChannel.transferFrom(channel, position, count);
+        } catch (IOException e) {
+            // The transfer reads the file and writes the socket in one call, 
so a storage failure and a socket
+            // failure both arrive here as a bare IOException. Let the handler 
find out which one it was. A closed
+            // channel means the file was not read, so there is nothing to 
find out.
+            if (transferFailureHandler != null && !(e instanceof 
ClosedChannelException))
+                transferFailureHandler.onTransferFailure(this, position, 
count, e);
+            throw e;
+        }
+    }
+
+    /**
+     * Read the given region of the backing file, after {@link #writeTo} 
failed on it, to tell a storage failure from
+     * a socket failure. The whole region is read rather than its first byte 
because, depending on the platform, the
+     * transfer may fail anywhere within it without reporting progress. Only a 
failed read of an open channel counts
+     * as a storage failure: the end of the file may be reached because the 
log was truncated concurrently (e.g. when
+     * the replica becomes a follower), and the channel may have been closed 
concurrently, by a deletion or a
+     * shutdown; neither says anything about the medium.
+     *
+     * @param position The position of the region in the file
+     * @param count The size of the region
+     * @return The failure of the read, naming the file and the position, or 
empty if the region could be read or
+     *         nothing can be concluded
+     */
+    public Optional<IOException> readFailure(long position, int count) {
+        ByteBuffer buffer = ByteBuffer.allocate(Math.min(count, 
REREAD_BUFFER_SIZE));
+        long regionEnd = position + count;
+        long readPosition = position;
+        try {
+            while (readPosition < regionEnd) {
+                buffer.clear();
+                buffer.limit((int) Math.min(buffer.capacity(), regionEnd - 
readPosition));
+                int bytesRead = channel.read(buffer, readPosition);
+                if (bytesRead <= 0)
+                    return Optional.empty();
+                readPosition += bytesRead;
+            }
+            return Optional.empty();
+        } catch (ClosedChannelException e) {
+            return Optional.empty();
+        } catch (IOException e) {
+            return Optional.of(new IOException("Failed to read " + 
file.getAbsolutePath() + " at position "
+                    + readPosition + " after failing to write " + count + " 
bytes from position " + position
+                    + " to a channel", e));

Review Comment:
   to a channel => to a network channel



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
  * instance to enable slicing a range of the log records.
  */
 public class FileRecords extends AbstractRecords implements Closeable {
+    // Size of the buffer used to re-read a region that failed to transfer, 
see readFailure
+    static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+    /**
+     * Notified when {@link #writeTo} fails, typically to find out with {@link 
#readFailure} whether the backing file
+     * could not be read. It is called on the thread that failed to write, a 
network thread, so it must not block.
+     */
+    @FunctionalInterface
+    public interface TransferFailureHandler {

Review Comment:
   For consistency, should we hook this up to SslTransportLayer.transferFrom() 
to handle disk IO error during read too? This could be done in a separate 
followup PR.



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java:
##########
@@ -256,6 +273,122 @@ public void testLogAppend() throws IOException {
         assertEquals(keyValues, recordsToKvs(fetchDataInfo.records.records()));
     }
 
+    @Test
+    public void testReadFailureAfterTransferFailureMarksLogDirOffline() throws 
IOException {
+        try {
+            appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 
0L);
+            FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+            assertNotNull(records.transferFailureHandler());
+            IOException readFailure = new IOException("Input/output error");
+            FileRecords unreadable = unreadableRecords(records.file(), 
readFailure);
+            IOException transferFailure = new IOException("Input/output 
error");
+
+            // The fetched records are read only when they are written to the 
network; a failed write is reported
+            // through their handler, which re-reads the region on the 
scheduler
+            records.transferFailureHandler().onTransferFailure(unreadable, 0, 
100, transferFailure);
+
+            verify(unreadable).readFailure(0, 100);
+            
assertTrue(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+            assertArrayEquals(new Throwable[] {transferFailure}, 
readFailure.getSuppressed());
+            assertThrows(KafkaStorageException.class, () -> readRecords(0L));
+        } finally {
+            log.closeQuietly();
+        }
+    }
+
+    @Test
+    public void testTransferFailureOnReadableRegionKeepsLogDirOnline() throws 
IOException {
+        appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 0L);
+        FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+
+        records.transferFailureHandler().onTransferFailure(records, 0, 
records.sizeInBytes(), new IOException("Broken pipe"));
+
+        assertFalse(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+        assertEquals(1, 
Utils.toList(readRecords(0L).records.records()).size());
+    }
+
+    /**
+     * A burst of failed writes on a segment schedules a single re-read, and 
another one only once it is done.
+     */
+    @Test
+    public void testOneReadCheckAtATimePerSegment() throws IOException {
+        List<Runnable> scheduled = new ArrayList<>();
+        Scheduler scheduler = mock(Scheduler.class);
+        when(scheduler.scheduleOnce(anyString(), 
any(Runnable.class))).thenAnswer(invocation -> {
+            scheduled.add(invocation.getArgument(1));
+            return mock(ScheduledFuture.class);
+        });
+        LocalLog otherLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir),
+            new LogConfig(new Properties()), scheduler);
+        try {
+            FileRecords records = fetchedRecords(otherLog);
+            FileRecords readable = spy(records);
+
+            for (int i = 0; i < 3; i++)
+                records.transferFailureHandler().onTransferFailure(readable, 
0, 10, new IOException("Broken pipe"));
+            assertEquals(1, scheduled.size());
+
+            scheduled.get(0).run();
+            verify(readable).readFailure(0, 10);
+            records.transferFailureHandler().onTransferFailure(readable, 0, 
10, new IOException("Broken pipe"));
+            assertEquals(2, scheduled.size());
+        } finally {
+            otherLog.closeQuietly();
+        }
+    }
+
+    /**
+     * A check the scheduler did not take, because it is not running or it 
rejected it, must not stop the next
+     * failed write from being checked.
+     */
+    @Test
+    public void testReadCheckNotTakenBySchedulerDoesNotBlockTheNextOne() 
throws IOException {
+        ScheduledFuture<?> ignored = mock(ScheduledFuture.class);
+        when(ignored.isCancelled()).thenReturn(true);
+        Scheduler scheduler = mock(Scheduler.class);
+        doReturn(ignored).doThrow(new 
RejectedExecutionException()).doReturn(ignored)
+            .when(scheduler).scheduleOnce(anyString(), any(Runnable.class));
+        LocalLog otherLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir),
+            new LogConfig(new Properties()), scheduler);
+        try {
+            FileRecords records = fetchedRecords(otherLog);
+
+            for (int i = 0; i < 3; i++)
+                records.transferFailureHandler().onTransferFailure(records, 0, 
10, new IOException("Broken pipe"));
+            verify(scheduler, times(3)).scheduleOnce(anyString(), 
any(Runnable.class));
+        } finally {
+            otherLog.closeQuietly();
+        }
+    }
+
+    @Test
+    public void testNoReadCheckOnceLogDirIsOffline() throws IOException {

Review Comment:
   Claude found the following issue.
   
   ```
   1. testNoReadCheckOnceLogDirIsOffline takes the shared log offline and leaks 
its files (LocalLogTest.java:364-378). dir comes from 
randomPartitionLogDir(tmpDir), so dir.getParent() is tmpDir, which is also the 
parent of the shared log's logDir. Marking it offline makes tearDown's 
log.close() throw KafkaStorageException before any segment is closed, and the 
exception is swallowed. The open FileChannel and index mmaps then stay open for 
the rest of the test JVM. Either:
   
   create otherLog under its own temp directory, e.g. 
TestUtils.randomPartitionLogDir(TestUtils.tempDirectory()), so the shared log 
isn't affected; or
   call log.closeQuietly() in the finally, as 
testReadFailureAfterTransferFailureMarksLogDirOffline does.
   ```



##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +531,47 @@ else if (segment.baseOffset() == 
maxOffsetMetadata.segmentBaseOffset && !maxOffs
         );
     }
 
+    /**
+     * The records of a fetch are read from the segment only when they are 
written to the network, after this log
+     * has returned them, so an IOException from that read cannot be caught 
here. Have the records report a failed
+     * write instead, so that the log directory is marked offline if the 
segment cannot be read, as for any other
+     * IOException in this log.
+     */
+    private FetchDataInfo withTransferFailureHandler(FetchDataInfo 
fetchDataInfo) {
+        if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+            return fetchDataInfo;
+        return 
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+    }
+
+    /**
+     * Called on the network thread when writing fetched records failed, which 
may be a socket or a storage failure.
+     * Re-reading the region tells them apart, so it is done on the scheduler 
rather than on the network thread, at
+     * most once at a time per segment file so that a burst of failed sends 
does not queue up re-reads.
+     */
+    private void checkReadAfterTransferFailure(FileRecords records, long 
position, int count, IOException transferFailure) {
+        String file = records.file().getAbsolutePath();
+        if (logDirFailureChannel.hasOfflineLogDir(parentDir) || 
!pendingReadChecks.add(file))

Review Comment:
   Claude recommended the following.
   
   ```
   Don't let all re-read checks run at once on the shared scheduler 
(LocalLog.java:556). scheduler is the broker's KafkaScheduler 
(background.threads, 10 by default). It also runs ISR expiration, log flush, 
high-watermark and recovery-point checkpoints, retention and async deletes. 
Checks are de-duplicated only per segment file, so a disk that fails slowly, 
with reads blocking for tens of seconds, can queue one check per failing 
segment across many partitions. That can tie up every scheduler thread and 
stall those tasks for healthy log dirs too.
   
   Fix: allow at most one check per log directory at a time, across all 
partitions. Marking the directory offline is all-or-nothing anyway, so a second 
check on the same directory adds nothing.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to