cheeeee commented on code in PR #23445:
URL: https://github.com/apache/kafka/pull/23445#discussion_r4289967588
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
* instance to enable slicing a range of the log records.
*/
public class FileRecords extends AbstractRecords implements Closeable {
+ // Size of the buffer used to re-read a region that failed to transfer,
see readFailure
+ static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+ /**
+ * Notified when {@link #writeTo} fails, typically to find out with {@link
#readFailure} whether the backing file
+ * could not be read. It is called on the thread that failed to write, a
network thread, so it must not block.
+ */
+ @FunctionalInterface
+ public interface TransferFailureHandler {
+ void onTransferFailure(FileRecords records, long position, int count,
IOException transferFailure);
+ }
+
private final boolean isSlice;
private final int start;
private final int end;
+ // Notified when writing to a channel fails, or null
Review Comment:
Done.
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -146,7 +164,26 @@ public FileRecords slice(int position, int size) {
int availableBytes = availableBytes(position, size);
int startPosition = this.start + position;
- return new FileRecords(file, channel, startPosition, startPosition +
availableBytes);
+ return new FileRecords(file, channel, startPosition, startPosition +
availableBytes, transferFailureHandler);
+ }
+
+ /**
+ * Return a copy of this slice whose {@link #writeTo} notifies the given
handler when writing to a channel fails.
Review Comment:
Done.
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -278,17 +315,61 @@ public int truncateTo(int targetSize) throws IOException {
@Override
public int writeTo(TransferableChannel destChannel, int offset, int
length) throws IOException {
- long newSize = Math.min(channel.size(), end) - start;
int oldSize = sizeInBytes();
- if (newSize < oldSize)
- throw new KafkaException(String.format(
- "Size of FileRecords %s has been truncated during write:
old size %d, new size %d",
- file.getAbsolutePath(), oldSize, newSize));
-
long position = start + offset;
int count = Math.min(length, oldSize - offset);
- // safe to cast to int since `count` is an int
- return (int) destChannel.transferFrom(channel, position, count);
+ try {
+ long newSize = Math.min(channel.size(), end) - start;
+ if (newSize < oldSize)
+ throw new KafkaException(String.format(
+ "Size of FileRecords %s has been truncated during
write: old size %d, new size %d",
+ file.getAbsolutePath(), oldSize, newSize));
+ // safe to cast to int since `count` is an int
+ return (int) destChannel.transferFrom(channel, position, count);
+ } catch (IOException e) {
+ // The transfer reads the file and writes the socket in one call,
so a storage failure and a socket
+ // failure both arrive here as a bare IOException. Let the handler
find out which one it was. A closed
+ // channel means the file was not read, so there is nothing to
find out.
+ if (transferFailureHandler != null && !(e instanceof
ClosedChannelException))
+ transferFailureHandler.onTransferFailure(this, position,
count, e);
+ throw e;
+ }
+ }
+
+ /**
+ * Read the given region of the backing file, after {@link #writeTo}
failed on it, to tell a storage failure from
+ * a socket failure. The whole region is read rather than its first byte
because, depending on the platform, the
+ * transfer may fail anywhere within it without reporting progress. Only a
failed read of an open channel counts
+ * as a storage failure: the end of the file may be reached because the
log was truncated concurrently (e.g. when
+ * the replica becomes a follower), and the channel may have been closed
concurrently, by a deletion or a
+ * shutdown; neither says anything about the medium.
+ *
+ * @param position The position of the region in the file
+ * @param count The size of the region
+ * @return The failure of the read, naming the file and the position, or
empty if the region could be read or
+ * nothing can be concluded
+ */
+ public Optional<IOException> readFailure(long position, int count) {
+ ByteBuffer buffer = ByteBuffer.allocate(Math.min(count,
REREAD_BUFFER_SIZE));
+ long regionEnd = position + count;
+ long readPosition = position;
+ try {
+ while (readPosition < regionEnd) {
+ buffer.clear();
+ buffer.limit((int) Math.min(buffer.capacity(), regionEnd -
readPosition));
+ int bytesRead = channel.read(buffer, readPosition);
+ if (bytesRead <= 0)
+ return Optional.empty();
+ readPosition += bytesRead;
+ }
+ return Optional.empty();
+ } catch (ClosedChannelException e) {
+ return Optional.empty();
+ } catch (IOException e) {
+ return Optional.of(new IOException("Failed to read " +
file.getAbsolutePath() + " at position "
+ + readPosition + " after failing to write " + count + "
bytes from position " + position
+ + " to a channel", e));
Review Comment:
Done.
##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java:
##########
@@ -256,6 +273,122 @@ public void testLogAppend() throws IOException {
assertEquals(keyValues, recordsToKvs(fetchDataInfo.records.records()));
}
+ @Test
+ public void testReadFailureAfterTransferFailureMarksLogDirOffline() throws
IOException {
+ try {
+ appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))),
0L);
+ FileRecords records = assertInstanceOf(FileRecords.class,
readRecords(0L).records);
+ assertNotNull(records.transferFailureHandler());
+ IOException readFailure = new IOException("Input/output error");
+ FileRecords unreadable = unreadableRecords(records.file(),
readFailure);
+ IOException transferFailure = new IOException("Input/output
error");
+
+ // The fetched records are read only when they are written to the
network; a failed write is reported
+ // through their handler, which re-reads the region on the
scheduler
+ records.transferFailureHandler().onTransferFailure(unreadable, 0,
100, transferFailure);
+
+ verify(unreadable).readFailure(0, 100);
+
assertTrue(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+ assertArrayEquals(new Throwable[] {transferFailure},
readFailure.getSuppressed());
+ assertThrows(KafkaStorageException.class, () -> readRecords(0L));
+ } finally {
+ log.closeQuietly();
+ }
+ }
+
+ @Test
+ public void testTransferFailureOnReadableRegionKeepsLogDirOnline() throws
IOException {
+ appendRecords(kvsToRecords(List.of(new KeyValue("abc", "ABC"))), 0L);
+ FileRecords records = assertInstanceOf(FileRecords.class,
readRecords(0L).records);
+
+ records.transferFailureHandler().onTransferFailure(records, 0,
records.sizeInBytes(), new IOException("Broken pipe"));
+
+ assertFalse(logDirFailureChannel.hasOfflineLogDir(logDir.getParent()));
+ assertEquals(1,
Utils.toList(readRecords(0L).records.records()).size());
+ }
+
+ /**
+ * A burst of failed writes on a segment schedules a single re-read, and
another one only once it is done.
+ */
+ @Test
+ public void testOneReadCheckAtATimePerSegment() throws IOException {
+ List<Runnable> scheduled = new ArrayList<>();
+ Scheduler scheduler = mock(Scheduler.class);
+ when(scheduler.scheduleOnce(anyString(),
any(Runnable.class))).thenAnswer(invocation -> {
+ scheduled.add(invocation.getArgument(1));
+ return mock(ScheduledFuture.class);
+ });
+ LocalLog otherLog =
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir),
+ new LogConfig(new Properties()), scheduler);
+ try {
+ FileRecords records = fetchedRecords(otherLog);
+ FileRecords readable = spy(records);
+
+ for (int i = 0; i < 3; i++)
+ records.transferFailureHandler().onTransferFailure(readable,
0, 10, new IOException("Broken pipe"));
+ assertEquals(1, scheduled.size());
+
+ scheduled.get(0).run();
+ verify(readable).readFailure(0, 10);
+ records.transferFailureHandler().onTransferFailure(readable, 0,
10, new IOException("Broken pipe"));
+ assertEquals(2, scheduled.size());
+ } finally {
+ otherLog.closeQuietly();
+ }
+ }
+
+ /**
+ * A check the scheduler did not take, because it is not running or it
rejected it, must not stop the next
+ * failed write from being checked.
+ */
+ @Test
+ public void testReadCheckNotTakenBySchedulerDoesNotBlockTheNextOne()
throws IOException {
+ ScheduledFuture<?> ignored = mock(ScheduledFuture.class);
+ when(ignored.isCancelled()).thenReturn(true);
+ Scheduler scheduler = mock(Scheduler.class);
+ doReturn(ignored).doThrow(new
RejectedExecutionException()).doReturn(ignored)
+ .when(scheduler).scheduleOnce(anyString(), any(Runnable.class));
+ LocalLog otherLog =
createLocalLogWithActiveSegment(TestUtils.randomPartitionLogDir(tmpDir),
+ new LogConfig(new Properties()), scheduler);
+ try {
+ FileRecords records = fetchedRecords(otherLog);
+
+ for (int i = 0; i < 3; i++)
+ records.transferFailureHandler().onTransferFailure(records, 0,
10, new IOException("Broken pipe"));
+ verify(scheduler, times(3)).scheduleOnce(anyString(),
any(Runnable.class));
+ } finally {
+ otherLog.closeQuietly();
+ }
+ }
+
+ @Test
+ public void testNoReadCheckOnceLogDirIsOffline() throws IOException {
Review Comment:
Done.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -526,6 +531,47 @@ else if (segment.baseOffset() ==
maxOffsetMetadata.segmentBaseOffset && !maxOffs
);
}
+ /**
+ * The records of a fetch are read from the segment only when they are
written to the network, after this log
+ * has returned them, so an IOException from that read cannot be caught
here. Have the records report a failed
+ * write instead, so that the log directory is marked offline if the
segment cannot be read, as for any other
+ * IOException in this log.
+ */
+ private FetchDataInfo withTransferFailureHandler(FetchDataInfo
fetchDataInfo) {
+ if (!(fetchDataInfo.records instanceof FileRecords fileRecords))
+ return fetchDataInfo;
+ return
fetchDataInfo.withRecords(fileRecords.withTransferFailureHandler(transferFailureHandler));
+ }
+
+ /**
+ * Called on the network thread when writing fetched records failed, which
may be a socket or a storage failure.
+ * Re-reading the region tells them apart, so it is done on the scheduler
rather than on the network thread, at
+ * most once at a time per segment file so that a burst of failed sends
does not queue up re-reads.
+ */
+ private void checkReadAfterTransferFailure(FileRecords records, long
position, int count, IOException transferFailure) {
+ String file = records.file().getAbsolutePath();
+ if (logDirFailureChannel.hasOfflineLogDir(parentDir) ||
!pendingReadChecks.add(file))
Review Comment:
Done: at most one check at a time per log directory, across partitions,
tracked by `LogDirFailureChannel`, which all logs of the broker share.
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -41,9 +42,23 @@
* instance to enable slicing a range of the log records.
*/
public class FileRecords extends AbstractRecords implements Closeable {
+ // Size of the buffer used to re-read a region that failed to transfer,
see readFailure
+ static final int REREAD_BUFFER_SIZE = 32 * 1024;
+
+ /**
+ * Notified when {@link #writeTo} fails, typically to find out with {@link
#readFailure} whether the backing file
+ * could not be read. It is called on the thread that failed to write, a
network thread, so it must not block.
+ */
+ @FunctionalInterface
+ public interface TransferFailureHandler {
Review Comment:
No separate hook is needed: `SslTransportLayer.transferFrom` reads the file
within the call that `FileRecords.writeTo` makes, so a read error there reaches
the same handler. What stops it today is KAFKA-21280: once part of the region
has been sent, `transferFrom` swallows the error and sends a stale buffer
instead. #23832 fixes that. On the cluster in the test notes, a TLS listener
with #23832 offlined the directory and all 20000 records were consumed; without
it the consumer failed on a corrupt record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]