cheeeee commented on PR #23928:
URL: https://github.com/apache/kafka/pull/23928#issuecomment-6354481909
Test notes for 73fc46a880, on trunk d18f93f985.
Unit tests:
* `SslTransportLayerTest`, `FileRecordsTest` and `LocalLogTest` pass,
as do `:clients:test` (13757 tests), `:storage:test` (1355 tests),
checkstyle and spotless. 2 of 5 runs of `:storage:test` failed in
`TransactionsWithTieredStoreTest` (a timeout, and the test JVM
exiting), which passes alone 3 times out of 3 here and on trunk. On
trunk 2 of 3 runs failed in tiered storage integration tests too
(`TransactionsTest`, `RemoteTopicCrudTest`); see KAFKA-15772.
* Each new or changed test fails when the part of the change it covers
is undone. 15 such mutations, e.g. `transferFrom` wrapping a closed
file or a failed socket write, `reportsFileReadFailures()` returning
false for TLS, a report sharing the re-read's per-directory slot, or
a failed task not releasing it: each makes the test named for it
fail.
Cluster: 3 brokers and a controller (KRaft) in Docker on a Linux 6.8
VM. Broker 1 has two log directories, one of them on a `dm-dust`
device. Partition t1-0 has replicas 1, 2 and 3, broker 1 leads it, and
its replica on broker 1 is in the `dm-dust` directory. 20000 records of
512 bytes are produced, then a console consumer reads the partition
from the beginning with an 8 MiB fetch size, over PLAINTEXT or TLS.
`strace -yy` on broker 1 shows which thread reads which file at which
position.
1. Persistent media error: `dm-dust` makes reads of the tail of the
segment fail on the device. In every run the first read to fail is
at position 2867200.
* TLS, trunk: the network thread fails to read position 2867200.
The scheduler re-reads the region from position 0, 2867200 bytes,
and fails at 2867200. The directory goes offline.
* TLS, this change: the same failed read on the network thread, and
no read of the segment on the scheduler. The directory goes
offline, with "Failed to read .../00000000000000000000.log at
position 2867200 while writing 8388608 bytes from position 0 to a
network channel".
* PLAINTEXT, both: `sendfile` fails, the re-read on the scheduler
fails at the same position, and the directory goes offline.
* In all of them the leader moves to broker 2 and the consumer reads
all 20000 records.
2. Error gone on the next read: `strace` makes the first read of the
segment on each network thread of the listener fail with `EIO`, and
the following reads succeed.
* TLS, trunk: 3 failed reads, one on each network thread as the
consumer retried, each followed by a successful re-read of 8 MiB
on the scheduler. The directory stays online.
* TLS, this change: the first failed read offlines the directory,
without a re-read. The consumer reads all 20000 records from
broker 2.
* PLAINTEXT, both: as TLS on trunk, the directory stays online.
3. Disconnects, with a healthy disk: 3 TLS consumers read 4 partitions
led by broker 1, about 200 MB, and are killed after 2 to 8 seconds
and restarted, for 5 minutes, about 180 times in all.
* trunk: 4629 reads of segment files on the scheduler, 151 MB.
* this change: none.
* No directory went offline in either.
With this change the failed send is still logged as a disconnect, at
DEBUG, now with the `FileReadException`, and the ERROR comes from
`LogDirFailureChannel`. The timings depend on the host, a VM under
load, so I have left them out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
