Candice Wan created KAFKA-7802: ---------------------------------- Summary: Connection to Broker Disconnected Taking Down the Whole Cluster Key: KAFKA-7802 URL: https://issues.apache.org/jira/browse/KAFKA-7802 Project: Kafka Issue Type: Bug Components: core Affects Versions: 2.1.0 Reporter: Candice Wan Attachments: thread_dump.log
We recently upgraded to 2.1.0. Since then, several times per day, we observe some brokers were disconnected when other brokers were trying to fetch the replicas. This issue took down the whole cluster, making all the producers and consumers not able to publish or consume messages. Here is an example of what we're seeing in the broker which was trying to send fetch request to the problematic one: 2019-01-09 08:05:10.445 [ReplicaFetcherThread-0-3] INFO o.a.k.clients.FetchSessionHandler - [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Error sending fetch request (sessionId=937967566, epoch=1599941) to node 3: java.io.IOException: Connection to 3 was disconnected before the response was read. 2019-01-09 08:05:10.445 [ReplicaFetcherThread-1-3] INFO o.a.k.clients.FetchSessionHandler - [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=1] Error sending fetch request (sessionId=506217047, epoch=1375749) to node 3: java.io.IOException: Connection to 3 was disconnected before the response was read. 2019-01-09 08:05:10.445 [ReplicaFetcherThread-0-3] WARN kafka.server.ReplicaFetcherThread - [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=1, maxWait=500, minBytes=1, maxBytes=10485760, fetchData=\{__consumer_offsets-11=(offset=421032847, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[178])}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=937967566, epoch=1599941)) java.io.IOException: Connection to 3 was disconnected before the response was read at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100) at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:99) at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:199) at kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) We also took the thread dump of the problematic broker (attached). We found all the kafka-request-handler were hanging and waiting for some locks, which seemed to be a resource leak there. FYI java version we are running is 11.0.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)