[
https://issues.apache.org/jira/browse/KAFKA-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Harris updated KAFKA-15339:
--------------------------------
Component/s: KafkaConnect
(was: connect)
> Transient I/O error happening in appending records could lead to the halt of
> whole cluster
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-15339
> URL: https://issues.apache.org/jira/browse/KAFKA-15339
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect, producer
> Affects Versions: 3.5.0
> Reporter: Haoze Wu
> Priority: Major
>
> We are running an integration test in which we start an Embedded Connect
> Cluster in the active 3.5 branch. However, because of transient disk error,
> we may encounter an IOException during appending records to one topic. As
> shown in the stack trace:
> {code:java}
> [2023-08-13 16:53:51,016] ERROR Error while appending records to
> connect-config-topic-connect-cluster-0 in dir
> /tmp/EmbeddedKafkaCluster8003464883598783225
> (org.apache.kafka.storage.internals.log.LogDirFailureChannel:61)
> java.io.IOException:
> at
> org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:92)
> at
> org.apache.kafka.common.record.FileRecords.append(FileRecords.java:188)
> at kafka.log.LogSegment.append(LogSegment.scala:161)
> at kafka.log.LocalLog.append(LocalLog.scala:436)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:853)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:664)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1281)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1269)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:977)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:965)
> at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:623)
> at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:680)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:180)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:76)
> at java.lang.Thread.run(Thread.java:748) {code}
> However, just because of failing to append the records to one partition. The
> fetcher for all the other partitions are removed, broker shutdown, and
> finally embedded connect cluster killed as whole.
> {code:java}
> [2023-08-13 17:35:37,966] WARN Stopping serving logs in dir
> /tmp/EmbeddedKafkaCluster6777164631574762227 (kafka.log.LogManager:70)
> [2023-08-13 17:35:37,968] ERROR Shutdown broker because all log dirs in
> /tmp/EmbeddedKafkaCluster6777164631574762227 have failed
> (kafka.log.LogManager:143)
> [2023-08-13 17:35:37,968] WARN Abrupt service halt with code 1 and message
> null (org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster:130)
> [2023-08-13 17:35:37,968] ERROR [LogDirFailureHandler]: Error due to
> (kafka.server.ReplicaManager$LogDirFailureHandler:135)
> org.apache.kafka.connect.util.clusters.UngracefulShutdownException: Abrupt
> service halt with code 1 and message null {code}
> I am wondering if we could add configurable retry around the root cause to
> tolerate the possible I/O faults so that if the retry is successful, the
> embedded connect cluster could still operate.
> Any comments and suggestions would be appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
