junrao commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4297151835


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +528,48 @@ else if (segment.baseOffset() == 
maxOffsetMetadata.segmentBaseOffset && !maxOffs
         );
     }
 
+    /**
+     * The records of a fetch are read from the segment only when they are 
written to the network, after this log
+     * has returned them, so an IOException from that read cannot be caught 
here. Have the records report a failed
+     * write instead, so that the log directory is marked offline if the 
segment cannot be read, as for any other
+     * IOException in this log.
+     */
+    private FetchDataInfo withTransferFailureHandler(FetchDataInfo 
fetchDataInfo) {
+        if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+            return fetchDataInfo;
+        return 
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+    }
+
+    /**
+     * Called on the network thread when writing fetched records failed, which 
may be a socket or a storage failure.
+     * Re-reading the region tells them apart, so it is done on the scheduler 
rather than on the network thread, at
+     * most once at a time per log dir, so that a burst of failed sends or a 
slow failing disk does not take up the
+     * threads of the scheduler, which other tasks of the broker share.
+     */
+    private void checkReadAfterTransferFailure(FileRecords records, long 
position, int count, IOException transferFailure) {
+        String logDir = parentDir;
+        if (logDirFailureChannel.hasOfflineLogDir(logDir) || 
!logDirFailureChannel.tryStartReadCheck(logDir))

Review Comment:
   It's probably better to push the hasOfflineLogDir check inside 
tryStartReadCheck.



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java:
##########
@@ -256,6 +273,136 @@ public void testLogAppend() throws IOException {
         assertEquals(keyValues, recordsToKvs(fetchDataInfo.records.records()));
     }
 
+    @Test
+    public void testReadFailureAfterTransferFailureMarksLogDirOffline() throws 
IOException {
+        try {
+            appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 
0L);
+            FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+            assertNotNull(records.transferFailureHandler());
+            IOException readFailure = new IOException("Input/output error");
+            FileRecords unreadable = unreadableRecords(records.file(), 
readFailure);
+            IOException transferFailure = new IOException("Input/output 
error");
+
+            // The fetched records are read only when they are written to the 
network; a failed write is reported
+            // through their handler, which re-reads the region on the 
scheduler
+            records.transferFailureHandler().onTransferFailure(unreadable, 0, 
100, transferFailure);
+
+            verify(unreadable).readFailure(0, 100);
+            
assertTrue(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+            assertArrayEquals(new Throwable[] {transferFailure}, 
readFailure.getSuppressed());
+            assertThrows(KafkaStorageException.class, () -> readRecords(0L));
+        } finally {
+            log.closeQuietly();
+        }
+    }
+
+    @Test
+    public void testTransferFailureOnReadableRegionKeepsLogDirOnline() throws 
IOException {
+        appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 0L);
+        FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+
+        records.transferFailureHandler().onTransferFailure(records, 0, 
records.sizeInBytes(), new IOException("Broken pipe"));
+
+        assertFalse(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+        assertEquals(1, 
Utils.toList(readRecords(0L).records.records()).size());
+    }
+
+    /**
+     * A burst of failed writes across the partitions of a log dir schedules a 
single re-read, and another one only
+     * once it is done. Another log dir is checked on its own.
+     */
+    @Test
+    public void testOneReadCheckAtATimePerLogDir() throws IOException {
+        List<Runnable> scheduled = new ArrayList<>();
+        Scheduler scheduler = mock(Scheduler.class);
+        when(scheduler.scheduleOnce(anyString(), 
any(Runnable.class))).thenAnswer(invocation -> {
+            scheduled.add(invocation.getArgument(1));
+            return mock(ScheduledFuture.class);
+        });
+        LogConfig config = new LogConfig(new Properties());
+        LocalLog firstLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir), 
config, scheduler);
+        LocalLog secondLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir), 
config, scheduler);
+        LocalLog otherDirLog = createLocalLogWithActiveSegment(
+            TestUtils.randomPartitionLogDir(TestUtils.tempDirectory()), 
config, scheduler);
+        try {
+            FileRecords first = fetchedRecords(firstLog);
+            FileRecords readable = spy(first);

Review Comment:
   readable => sth like spiedFirst ?



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
  * instance to enable slicing a range of the log records.
  */
 public class FileRecords extends AbstractRecords implements Closeable {
+    // Size of the buffer used to re-read a region that failed to transfer, 
see readFailure
+    static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+    /**
+     * Notified when {@link #writeTo} fails, typically to find out with {@link 
#readFailure} whether the backing file
+     * could not be read. It is called on the thread that failed to write, a 
network thread, so it must not block.
+     */
+    @FunctionalInterface
+    public interface TransferFailureHandler {

Review Comment:
   Thanks for the explanation. Agreed.
   
   Since SslTransportLayer.transferFrom() first reads from file channel and 
then sends the bytes to the network channel. We can categorize the disk IO 
error more accurately here. Perhaps we can add a new type of disk related 
IOException through TransferableChannel.transferFrom()?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to