cheeeee commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4300113709


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +528,48 @@ else if (segment.baseOffset() == 
maxOffsetMetadata.segmentBaseOffset && !maxOffs
         );
     }
 
+    /**
+     * The records of a fetch are read from the segment only when they are 
written to the network, after this log
+     * has returned them, so an IOException from that read cannot be caught 
here. Have the records report a failed
+     * write instead, so that the log directory is marked offline if the 
segment cannot be read, as for any other
+     * IOException in this log.
+     */
+    private FetchDataInfo withTransferFailureHandler(FetchDataInfo 
fetchDataInfo) {
+        if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+            return fetchDataInfo;
+        return 
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+    }
+
+    /**
+     * Called on the network thread when writing fetched records failed, which 
may be a socket or a storage failure.
+     * Re-reading the region tells them apart, so it is done on the scheduler 
rather than on the network thread, at
+     * most once at a time per log dir, so that a burst of failed sends or a 
slow failing disk does not take up the
+     * threads of the scheduler, which other tasks of the broker share.
+     */
+    private void checkReadAfterTransferFailure(FileRecords records, long 
position, int count, IOException transferFailure) {
+        String logDir = parentDir;
+        if (logDirFailureChannel.hasOfflineLogDir(logDir) || 
!logDirFailureChannel.tryStartReadCheck(logDir))

Review Comment:
   Done.
   



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java:
##########
@@ -256,6 +273,136 @@ public void testLogAppend() throws IOException {
         assertEquals(keyValues, recordsToKvs(fetchDataInfo.records.records()));
     }
 
+    @Test
+    public void testReadFailureAfterTransferFailureMarksLogDirOffline() throws 
IOException {
+        try {
+            appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 
0L);
+            FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+            assertNotNull(records.transferFailureHandler());
+            IOException readFailure = new IOException("Input/output error");
+            FileRecords unreadable = unreadableRecords(records.file(), 
readFailure);
+            IOException transferFailure = new IOException("Input/output 
error");
+
+            // The fetched records are read only when they are written to the 
network; a failed write is reported
+            // through their handler, which re-reads the region on the 
scheduler
+            records.transferFailureHandler().onTransferFailure(unreadable, 0, 
100, transferFailure);
+
+            verify(unreadable).readFailure(0, 100);
+            
assertTrue(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+            assertArrayEquals(new Throwable[] {transferFailure}, 
readFailure.getSuppressed());
+            assertThrows(KafkaStorageException.class, () -> readRecords(0L));
+        } finally {
+            log.closeQuietly();
+        }
+    }
+
+    @Test
+    public void testTransferFailureOnReadableRegionKeepsLogDirOnline() throws 
IOException {
+        appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 0L);
+        FileRecords records = assertInstanceOf(FileRecords.class, 
readRecords(0L).records);
+
+        records.transferFailureHandler().onTransferFailure(records, 0, 
records.sizeInBytes(), new IOException("Broken pipe"));
+
+        assertFalse(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+        assertEquals(1, 
Utils.toList(readRecords(0L).records.records()).size());
+    }
+
+    /**
+     * A burst of failed writes across the partitions of a log dir schedules a 
single re-read, and another one only
+     * once it is done. Another log dir is checked on its own.
+     */
+    @Test
+    public void testOneReadCheckAtATimePerLogDir() throws IOException {
+        List<Runnable> scheduled = new ArrayList<>();
+        Scheduler scheduler = mock(Scheduler.class);
+        when(scheduler.scheduleOnce(anyString(), 
any(Runnable.class))).thenAnswer(invocation -> {
+            scheduled.add(invocation.getArgument(1));
+            return mock(ScheduledFuture.class);
+        });
+        LogConfig config = new LogConfig(new Properties());
+        LocalLog firstLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir), 
config, scheduler);
+        LocalLog secondLog = 
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir), 
config, scheduler);
+        LocalLog otherDirLog = createLocalLogWithActiveSegment(
+            TestUtils.randomPartitionLogDir(TestUtils.tempDirectory()), 
config, scheduler);
+        try {
+            FileRecords first = fetchedRecords(firstLog);
+            FileRecords readable = spy(first);

Review Comment:
   Done.
   



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
  * instance to enable slicing a range of the log records.
  */
 public class FileRecords extends AbstractRecords implements Closeable {
+    // Size of the buffer used to re-read a region that failed to transfer, 
see readFailure
+    static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+    /**
+     * Notified when {@link #writeTo} fails, typically to find out with {@link 
#readFailure} whether the backing file
+     * could not be read. It is called on the thread that failed to write, a 
network thread, so it must not block.
+     */
+    @FunctionalInterface
+    public interface TransferFailureHandler {

Review Comment:
   Agreed. Now that #23832 rethrows the read error, wrapping it in a dedicated 
`IOException` at the `fileChannel.read` call would let `LocalLog` take the 
directory offline on TLS listeners without a re-read, and would also catch a 
media error that has cleared by the time of the re-read. I'd like to do it in a 
second PR for KAFKA-16304, after this PR and #23832 are merged:
   
   * it changes the same lines of `SslTransportLayer.transferFrom` as #23832, 
which fixes a bug present since 2.5.0 and could be backported on its own, so 
I'd rather keep that PR small;
   * doing it here would make this PR depend on #23832 in code, while the fix 
here works on its own over PLAINTEXT, and over TLS once #23832 is in;
   * over PLAINTEXT `sendfile` reads and sends in one call and reports a disk 
error and a socket error as the same `IOException`, so there is no read to 
wrap: the re-read stays the way to tell them apart there, and the new type only 
lets the TLS path skip it.
   
   Would that be ok with you, and does this order of the three PRs work for 
you: this PR and #23832 first, in either order, then the follow-up?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to