cheeeee commented on PR #23445:
URL: https://github.com/apache/kafka/pull/23445#issuecomment-6160507349
I've rewritten the description to match the new approach. The previous one
described the earlier revision, which
only reported the failure and asked how to wire it to
`LogDirFailureChannel`; that question is settled by this
revision, so the "Scope" section is gone. The commits are squashed into one,
hence the force-push.
End to end, on this revision and on its base commit (c130b7382c), built with
`releaseTarGz` and run with JDK 17 in a
User-Mode Linux 6.8 guest with `dm-dust`: a single-node KRaft broker with
two log directories, the metadata log
directory and `/data/good` on a healthy device, `/tmp/bad/logs` on
`dm-dust`. 20000 records are produced, the tail of
the target segment is marked unreadable while its head and indexes stay in
the page cache, and one large fetch
spans into that tail. `sendfile` is traced with `strace` on all broker
threads.
| | before (c130b7382c) | after (this PR) |
|---|---|---|
| `sendfile` calls / of those `-EIO` | 414 / 104 | 4 / 1 |
| `DEBUG ... disconnected` with `IOException: Input/output error` | 104 | 0 |
| `ERROR` from `LogDirFailureChannel` | 0 | 1 |
| `KafkaStorageException` (`WARN Unexpected error`) | 0 | 1 |
| `/tmp/bad/logs` in `kafka-log-dirs` | `error: null`, 3 partitions |
`error: KafkaStorageException`, 0 partitions |
| target partition | `Leader: 1` | `Leader: none` |
| broker alive at the end | yes | yes |
Before, every one of the 104 failed sends was logged as a client disconnect
and the consumer kept retrying the
same region. After, the first failed send offlines the directory and nothing
reads from it again:
```
ERROR Failed to read /tmp/bad/logs/t1-0/00000000000000000000.log at position
638976 while writing 1101369 bytes
from position 638976 to a channel
(org.apache.kafka.storage.internals.log.LogDirFailureChannel)
WARN [SocketServer listenerType=BROKER, nodeId=1] Unexpected error from
/127.0.0.1 (...); closing connection
org.apache.kafka.common.errors.KafkaStorageException: Failed to read
/tmp/bad/logs/t1-0/00000000000000000000.log at
position 638976 while writing 1101369 bytes from position 638976 to a
channel
Suppressed: java.io.IOException: Input/output error
Caused by: java.io.IOException: Input/output error
WARN [ReplicaManager broker=1] Stopping serving replicas in dir
/tmp/bad/logs ... because the log directory has failed.
WARN Logs for partitions t1-2,t1-4,t1-0 are offline ... due to failure on
log directory /tmp/bad/logs
```
The trace also shows the Linux behaviour the re-read relies on: `sendfile`
reports progress up to the first
unreadable page and fails at it on the next call (`[538996] => [638976]`,
then `[638976] = -1 EIO`). Only the
plaintext path was exercised end to end; the SSL path and other platforms
are covered by the unit tests only.
Tests:
* `FileRecordsTest`: storage failure at the start and inside the region
(with the failed position in the message),
end of file, no progress and a closed channel (`ClosedChannelException`,
`ClosedByInterruptException`) during the
re-read, no handler (no re-read), slices keeping the handler.
* `LocalLogTest`: a read failure reported by fetched records offlines the
log directory, and the next read fails with
`KafkaStorageException`.
* Ran `FileRecordsTest`, `:storage:test`, `ReplicaManagerTest`,
`:raft:test`, `LogDirFailureTest`, plus checkstyle
and spotless for `clients` and `storage`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]