[
https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970917#comment-16970917
]
Peter Bukowinski edited comment on KAFKA-9044 at 11/9/19 7:11 PM:
------------------------------------------------------------------
I had another broker dropout on one of my clusters. This time, it was the
controller node that dropped out, but this node that is no longer in the
cluster is continuing to act as the controller!
[2019-11-09 02:00:43,638] INFO [Controller id=15] Newly added brokers: ,
deleted brokers: 15, bounced brokers: , all live brokers:
1,2,3,4,5,6,7,9,10,11,13,14,16,17,19,20,21,22,23,24,25,26,27,28,29,30
(kafka.controller.KafkaController)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Shutting
down (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Stopped
(kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Shutdown
completed (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,660] INFO [Controller id=15] Broker failure callback for
15 (kafka.controller.KafkaController)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0023.example2.com:9092 (id: 23 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0017.example2.com:9092 (id: 17 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0011.example2.com:9092 (id: 11 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0026.example2.com:9092 (id: 26 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0005.example2.com:9092 (id: 5 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0025.example2.com:9092 (id: 25 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0022.example2.com:9092 (id: 22 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0029.example2.com:9092 (id: 29 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0004.example2.com:9092 (id: 4 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0013.example2.com:9092 (id: 13 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0007.example2.com:9092 (id: 7 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0020.example2.com:9092 (id: 20 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0014.example2.com:9092 (id: 14 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0010.example2.com:9092 (id: 10 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0006.example2.com:9092 (id: 6 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0016.example2.com:9092 (id: 16 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0021.example2.com:9092 (id: 21 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0002.example2.com:9092 (id: 2 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0030.example2.com:9092 (id: 30 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0028.example2.com:9092 (id: 28 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0019.example2.com:9092 (id: 19 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0027.example2.com:9092 (id: 27 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0009.example2.com:9092 (id: 9 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0024.example2.com:9092 (id: 24 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0003.example2.com:9092 (id: 3 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0001.example2.com:9092 (id: 1 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:44,052] DEBUG [Controller id=15] Unregister
BrokerModifications handler for Vector(15) (kafka.controller.KafkaController)
[2019-11-09 02:00:44,053] INFO [Controller id=15] Updated broker epochs cache:
Map(5 -> 73464160830, 10 -> 73464164101, 24 -> 73464174816, 25 -> 73464175663,
14 -> 73464166613, 20 -> 73464171591, 29 -> 73464179085, 1 -> 73464180710, 6 ->
73467975747, 28 -> 73464178261, 21 -> 73464172377, 9 -> 73464163290, 13 ->
73464165814, 2 -> 73465694416, 17 -> 73465755323, 22 -> 73464173178, 27 ->
73464177443, 7 -> 73464162458, 3 -> 73464159079, 16 -> 73464168210, 11 ->
73464164944, 26 -> 73464176585, 23 -> 73464173976, 30 -> 73464179870, 19 ->
73464170780, 4 -> 73464159901) (kafka.controller.KafkaController)
[2019-11-09 02:00:44,455] DEBUG [Controller id=15] Sending MetadataRequest to
Brokers: Vector(5, 10, 24, 25, 14, 20, 29, 1, 6, 28, 21, 9, 13, 2, 17, 22, 27,
7, 3, 16, 11, 26, 23, 30, 19, 4) for TopicPartitions: ArrayBuffer(En-84)
(kafka.controller.KafkaController)
The controller znode still lists broker 15:
{"version":1,"brokerid":15,"timestamp":"1572385450319"}
cZxid = 0x111ace6fb3
ctime = Tue Oct 29 14:44:10 PDT 2019
mZxid = 0x111ace6fb3
mtime = Tue Oct 29 14:44:10 PDT 2019
pZxid = 0x111ace6fb3
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16c7252b96a8ccf
dataLength = 55
numChildren = 0
was (Author: pmbuko):
I had another broker dropout on one of my clusters. This time, it was the
controller node that dropped out, but this node that is no longer in the
cluster is continuing to act as the controller!
[2019-11-09 02:00:43,638] INFO [Controller id=15] Newly added brokers: ,
deleted brokers: 15, bounced brokers: , all live brokers:
1,2,3,4,5,6,7,9,10,11,13,14,16,17,19,20,21,22,23,24,25,26,27,28,29,30
(kafka.controller.KafkaController)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Shutting
down (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Stopped
(kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,657] INFO [RequestSendThread controllerId=15] Shutdown
completed (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,660] INFO [Controller id=15] Broker failure callback for
15 (kafka.controller.KafkaController)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0023.example2.com:9092 (id: 23 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0017.example2.com:9092 (id: 17 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0011.example2.com:9092 (id: 11 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0026.example2.com:9092 (id: 26 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0005.example2.com:9092 (id: 5 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0025.example2.com:9092 (id: 25 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0022.example2.com:9092 (id: 22 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0029.example2.com:9092 (id: 29 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0004.example2.com:9092 (id: 4 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0013.example2.com:9092 (id: 13 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0007.example2.com:9092 (id: 7 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0020.example2.com:9092 (id: 20 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,809] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0014.example2.com:9092 (id: 14 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0010.example2.com:9092 (id: 10 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0006.example2.com:9092 (id: 6 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0016.example2.com:9092 (id: 16 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0021.example2.com:9092 (id: 21 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0002.example2.com:9092 (id: 2 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0030.example2.com:9092 (id: 30 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0028.example2.com:9092 (id: 28 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0019.example2.com:9092 (id: 19 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0027.example2.com:9092 (id: 27 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0009.example2.com:9092 (id: 9 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0024.example2.com:9092 (id: 24 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0003.example2.com:9092 (id: 3 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:43,810] INFO [RequestSendThread controllerId=15] Controller
15 connected to kafka0001.example2.com:9092 (id: 1 rack: REDACTED) for sending
state change requests (kafka.controller.RequestSendThread)
[2019-11-09 02:00:44,052] DEBUG [Controller id=15] Unregister
BrokerModifications handler for Vector(15) (kafka.controller.KafkaController)
[2019-11-09 02:00:44,053] INFO [Controller id=15] Updated broker epochs cache:
Map(5 -> 73464160830, 10 -> 73464164101, 24 -> 73464174816, 25 -> 73464175663,
14 -> 73464166613, 20 -> 73464171591, 29 -> 73464179085, 1 -> 73464180710, 6 ->
73467975747, 28 -> 73464178261, 21 -> 73464172377, 9 -> 73464163290, 13 ->
73464165814, 2 -> 73465694416, 17 -> 73465755323, 22 -> 73464173178, 27 ->
73464177443, 7 -> 73464162458, 3 -> 73464159079, 16 -> 73464168210, 11 ->
73464164944, 26 -> 73464176585, 23 -> 73464173976, 30 -> 73464179870, 19 ->
73464170780, 4 -> 73464159901) (kafka.controller.KafkaController)
[2019-11-09 02:00:44,455] DEBUG [Controller id=15] Sending MetadataRequest to
Brokers: Vector(5, 10, 24, 25, 14, 20, 29, 1, 6, 28, 21, 9, 13, 2, 17, 22, 27,
7, 3, 16, 11, 26, 23, 30, 19, 4) for TopicPartitions:
ArrayBuffer(EngagedCompletions-84) (kafka.controller.KafkaController)
The controller znode still lists broker 15:
{"version":1,"brokerid":15,"timestamp":"1572385450319"}
cZxid = 0x111ace6fb3
ctime = Tue Oct 29 14:44:10 PDT 2019
mZxid = 0x111ace6fb3
mtime = Tue Oct 29 14:44:10 PDT 2019
pZxid = 0x111ace6fb3
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16c7252b96a8ccf
dataLength = 55
numChildren = 0
> Brokers occasionally (randomly?) dropping out of clusters
> ---------------------------------------------------------
>
> Key: KAFKA-9044
> URL: https://issues.apache.org/jira/browse/KAFKA-9044
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.3.0, 2.3.1
> Environment: Ubuntu 14.04
> Reporter: Peter Bukowinski
> Priority: Major
>
> I have several cluster running kafka 2.3.1 and this issue has affected all of
> them. Because of replication and the size of the clusters (30 brokers), this
> bug is not causing any data loss, but it is nevertheless concerning. When a
> broker drops out, the log gives no indication that there are any zookeeper
> issues (and indeed the zookeepers are healthy when this occurs. Here's
> snippet from a broker log when it occurs:
> {{[2019-10-07 11:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Found deletable segments with base offsets [1975332] due to
> retention time 3600000ms breach (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Scheduling log segment [baseOffset 1975332, size 92076008]
> for deletion. (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Incrementing log start offset to 2000317 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Deleting segment 1975332 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted log
> /data/3/kl/internal_test-52/00000000000001975332.log.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted offset index
> /data/3/kl/internal_test-52/00000000000001975332.index.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,958] INFO Deleted time index
> /data/3/kl/internal_test-52/00000000000001975332.timeindex.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:42:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:52:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:22:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 14:46:07,510] INFO [Partition internal_test-33 broker=14]
> Shrinking ISR from 16,17,14 to 14. Leader: (highWatermark: 2007553,
> endOffset: 2007555). Out of sync replicas: (brokerId: 16, endOffset: 2007553)
> (brokerId: 17, endOffset: 2007553). (kafka.cluster.Partition)}}
> {{[2019-10-07 14:46:07,511] INFO [Partition internal_test-33 broker=14]
> Cached zkVersion [17] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)}}
> The controller log shows the following:
> {{[2019-10-07 14:45:55,427] INFO [Controller id=24] Newly added brokers: ,
> deleted brokers: 14, bounced brokers: , all live brokers:
> 1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,23,24,25
> (kafka.controller.KafkaController)}}
> {{[2019-10-07 14:45:55,477] INFO [RequestSendThread controllerId=24] Shutting
> down (kafka.controller.RequestSendThread)}}
> {{[2019-10-07 14:45:55,477] INFO [RequestSendThread controllerId=24] Shutdown
> completed (kafka.controller.RequestSendThread)}}
> {{[2019-10-07 14:45:55,477] INFO [RequestSendThread controllerId=24] Stopped
> (kafka.controller.RequestSendThread)}}
> {{[2019-10-07 14:45:55,481] INFO [Controller id=24] Broker failure callback
> for 14 (kafka.controller.KafkaController)}}
> The clusters use {{zookeeper.session.timeout.ms=45000}}, and
> {{zookeeper.connection.timeout.ms=90000}}.
> I'm unable to find a cause for this behavior. The only thing I can do to
> resolve the issue is to restart the broker.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
