wangliucheng created KAFKA-15506:
------------------------------------
Summary: follower receive KafkaStorageException before leader
raise disk error
Key: KAFKA-15506
URL: https://issues.apache.org/jira/browse/KAFKA-15506
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 3.3.2
Environment: Kafka Version: 3.3.2
Jdk Version: jdk1.8.0_301
Deployment mode: kraft
Reporter: wangliucheng
In my kafka environment, topic has 2 replicas, leader and follower unavailable
when disk error of leader
The follower detects disk error before the leader
Here is the logs:
*follower recive KafkaStorageException:*
{code:java}
[2023-08-17 08:40:15,516] ERROR [ReplicaFetcher replicaId=4, leaderId=1,
fetcherId=10] Error for partition __consumer_offsets-37 at offset 305860652
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.
{code}
*isr shrink 4,1 to 1:*
{code:java}
[2023-08-17 08:41:49,953] INFO [Partition __consumer_offsets-37 broker=1]
Shrinking ISR from 4,1 to 1. Leader: (highWatermark: 305860652, endOffset:
305860653). Out of sync replicas: (brokerId: 4, endOffset: 305860652).
(kafka.cluster.Partition)
{code}
*broker marking dir to offline:*
{code:java}
[2023-08-17 08:41:50,188] ERROR Error while appending records to
eb_raw_legendsec_flow_2-33 in dir /data09/kafka/log
(kafka.server.LogDirFailureChannel)
java.io.IOException: Read-only file system
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at
org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:92)
at
org.apache.kafka.common.record.FileRecords.append(FileRecords.java:188)
at kafka.log.LogSegment.append(LogSegment.scala:158)
at kafka.log.LocalLog.append(LocalLog.scala:436)
at kafka.log.UnifiedLog.append(UnifiedLog.scala:949)
at kafka.log.UnifiedLog.appendAsFollower(UnifiedLog.scala:778)
at
kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica(Partition.scala:1121)
at
kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica(Partition.scala:1128)
at
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:121)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:336)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:325)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:324)
at
kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
at
scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:359)
at
scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:355)
at
scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:309)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:324)
at
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:124)
at
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:123)
at scala.Option.foreach(Option.scala:437)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:123)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:106)
at
kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:97)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
