Digging into on of the case that Broker 1 kicked Broker 6 out of ISR, I found a
weird consequences that Broker 1 firstly decided to remove Broker 6 from ISR
out of no reason, then Broker 6 saw an connection disconnected exception on
ReplicaFetcherThread-0-1.
On Broker 1:
[2017-02-23 00:40:21,814] INFO Partition
[resolver-recording-profiles-prod-els,0] on broker 1: Shrinking ISR for
partition [resolver-recording-profiles-prod-els,0] from 1,6,4 to 1,4
(kafka.cluster.Partition)
[2017-02-23 00:40:21,816] INFO Partition [stolmen-entity-prod-els,2] on broker
1: Shrinking ISR for partition [stolmen-entity-prod-els,2] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,817] INFO Partition [compass-warn-prod-els,0] on broker 1:
Shrinking ISR for partition [compass-warn-prod-els,0] from 1,5,6 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,819] INFO Partition [jtop-migration-prod-els,37] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,37] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,820] INFO Partition [jtop-migration-smoke-els,11] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,11] from 1,6,5
to 1,5 (kafka.cluster.Partition)
[2017-02-23 00:40:21,821] INFO Partition [jtop-migration-prod-els,57] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,57] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,823] INFO Partition [stolmen-bundle-prod-els,3] on broker
1: Shrinking ISR for partition [stolmen-bundle-prod-els,3] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,824] INFO Partition [jtop-migration-prod-els,62] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,62] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,825] INFO Partition
[resolver-notifcation-requests-prod-els,0] on broker 1: Shrinking ISR for
partition [resolver-notifcation-requests-prod-els,0] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,826] INFO Partition [jtop-migration-prod-els,42] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,42] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,828] INFO Partition [jtop-migration-smoke-els,36] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,36] from 1,6,3
to 1,3 (kafka.cluster.Partition)
[2017-02-23 00:40:21,829] INFO Partition [jtop-migration-prod-els,2] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,2] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,830] INFO Partition [jtop-migration-smoke-els,56] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,56] from 1,6,3
to 1,3 (kafka.cluster.Partition)
[2017-02-23 00:40:21,832] INFO Partition [__consumer_offsets,24] on broker 1:
Shrinking ISR for partition [__consumer_offsets,24] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,833] INFO Partition [jtop-migration-smoke-els,51] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,51] from 1,6,5
to 1,5 (kafka.cluster.Partition)
[2017-02-23 00:40:21,834] INFO Partition
[resolver-notification-requests-prod-els,0] on broker 1: Shrinking ISR for
partition [resolver-notification-requests-prod-els,0] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,836] INFO Partition [__consumer_offsets,29] on broker 1:
Shrinking ISR for partition [__consumer_offsets,29] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,837] INFO Partition [jtop-migration-prod-els,17] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,17] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,839] INFO Partition [jtop-migration-prod-els,22] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,22] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,840] INFO Partition [jtop-migration-smoke-els,16] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,16] from 1,6,3
to 1,3 (kafka.cluster.Partition)
[2017-02-23 00:40:21,841] INFO Partition [__consumer_offsets,49] on broker 1:
Shrinking ISR for partition [__consumer_offsets,49] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,843] INFO Partition [__consumer_offsets,44] on broker 1:
Shrinking ISR for partition [__consumer_offsets,44] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,844] INFO Partition [jtop-migration-smoke-els,31] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,31] from 1,6,5
to 1,5 (kafka.cluster.Partition)
[2017-02-23 00:40:21,846] INFO Partition [__consumer_offsets,9] on broker 1:
Shrinking ISR for partition [__consumer_offsets,9] from 1,6,3 to 1,3
(kafka.cluster.Partition)
[2017-02-23 00:40:21,847] INFO Partition [__consumer_offsets,4] on broker 1:
Shrinking ISR for partition [__consumer_offsets,4] from 1,6,5 to 1,5
(kafka.cluster.Partition)
[2017-02-23 00:40:21,848] INFO Partition [account-sharing-prod-els,0] on broker
1: Shrinking ISR for partition [account-sharing-prod-els,0] from 2,1,6 to 2,1
(kafka.cluster.Partition)
On Broker 6:
[2017-02-23 00:40:32,080] WARN [ReplicaFetcherThread-0-1], Error in fetch
kafka.server.ReplicaFetcherThread$FetchRequest@3d2043a0. Possible cause:
java.io.IOException: Connection to 1 was disconnected before the response was
read (kafka.server.ReplicaFetcherThread)
[2017-02-23 00:40:32,080] WARN [ReplicaFetcherThread-0-1], Error in fetch
kafka.server.ReplicaFetcherThread$FetchRequest@3d2043a0. Possible cause:
java.io.IOException: Connection to 1 was disconnected before the response was
read (kafka.server.ReplicaFetcherThread)
[2017-02-23 00:40:33,081] DEBUG [ReplicaFetcherThread-0-1], handling partitions
with error for Set([jtop-migration-smoke-els,31],
[jtop-migration-smoke-els,51], [jtop-migration-prod-els,62],
[__consumer_offsets,49], [__consumer_offsets,44], [stolmen-bundle-prod-els,3],
[account-sharing-prod-els,0], [jtop-migration-smoke-els,56],
[stolmen-entity-prod-els,2], [jtop-migration-prod-els,22],
[jtop-migration-smoke-els,16], [__consumer_offsets,9], [__consumer_offsets,4],
[jtop-migration-prod-els,57], [jtop-migration-smoke-els,36],
[jtop-migration-prod-els,42], [__consumer_offsets,24],
[jtop-migration-prod-els,17], [compass-warn-prod-els,0],
[jtop-migration-prod-els,2], [resolver-notifcation-requests-prod-els,0],
[jtop-migration-smoke-els,11], [resolver-notification-requests-prod-els,0],
[resolver-recording-profiles-prod-els,0], [jtop-migration-prod-els,37],
[__consumer_offsets,29]) (kafka.server.ReplicaFetcherThread)
[2017-02-23 00:40:33,081] DEBUG [ReplicaFetcherThread-0-1], handling partitions
with error for Set([jtop-migration-smoke-els,31],
[jtop-migration-smoke-els,51], [jtop-migration-prod-els,62],
[__consumer_offsets,49], [__consumer_offsets,44], [stolmen-bundle-prod-els,3],
[account-sharing-prod-els,0], [jtop-migration-smoke-els,56],
[stolmen-entity-prod-els,2], [jtop-migration-prod-els,22],
[jtop-migration-smoke-els,16], [__consumer_offsets,9], [mj,0],
[__consumer_offsets,4], [jtop-migration-prod-els,57],
[jtop-migration-smoke-els,36], [jtop-migration-prod-els,42],
[__consumer_offsets,24], [jtop-migration-prod-els,17],
[compass-warn-prod-els,0], [jtop-migration-prod-els,2],
[resolver-notifcation-requests-prod-els,0], [jtop-migration-smoke-els,11],
[resolver-notification-requests-prod-els,0],
[resolver-recording-profiles-prod-els,0], [jtop-migration-prod-els,37],
[__consumer_offsets,29]) (kafka.server.ReplicaFetcherThread)
We’ve confirmed that the clock in both boxes are the same. It would make more
sense if the ReplicaFetcherThread disconnected first, then ISR changed, not the
other way around. Can someone explains this?
Thanks,
Jun
> On Feb 19, 2017, at 2:13 PM, Jun MA <mj.saber1...@gmail.com> wrote:
>
> Hi team,
>
> We are running confluent 0.9.0.1 on a cluster with 6 brokers. These days one
> of our broker(broker 1) frequently shrink the ISR and expand it immediately
> every about 20 minutes and I couldn’t find out why. Based on the log, I can
> kick out any of other brokers, not just a specific one. Here’s one log sample:
>
> [2017-02-19 13:01:01,741] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.ArrayIndexOutOfBoundsException
> [2017-02-19 13:01:01,777] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.ArrayIndexOutOfBoundsException
> [2017-02-19 13:01:01,781] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.ArrayIndexOutOfBoundsException
> [2017-02-19 13:01:01,796] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.ArrayIndexOutOfBoundsException
> [2017-02-19 13:01:17,230] INFO Partition [khloe-updates-prod-els,0] on broker
> 1: Shrinking ISR for partition [khloe-updates-prod-els,0] from 1,5,4 to 1,4
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,231] INFO Partition [jtop-migration-prod-els,37] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,37] from 1,5,6
> to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,232] INFO Partition [hummus-asset-v2-prod-els,0] on
> broker 1: Shrinking ISR for partition [hummus-asset-v2-prod-els,0] from 1,5,3
> to 1,3 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,233] INFO Partition [jtop-migration-prod-els,57] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,57] from 1,5,6
> to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,234] INFO Partition [jtop-migration-smoke-els,6] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,6] from 1,5,4
> to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,234] INFO Partition [jtop-migration-prod-els,12] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,12] from 1,5,4
> to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,235] INFO Partition [__consumer_offsets,39] on broker 1:
> Shrinking ISR for partition [__consumer_offsets,39] from 1,5,4 to 1,4
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,236] INFO Partition [jtop-migration-smoke-els,51] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,51] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,237] INFO Partition [hummus-entity-merged-v2-prod-els,0]
> on broker 1: Shrinking ISR for partition [hummus-entity-merged-v2-prod-els,0]
> from 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,238] INFO Partition
> [resolver-notification-profiles-staging-els,1] on broker 1: Shrinking ISR for
> partition [resolver-notification-profiles-staging-els,1] from 1,5,4 to 1,4
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,239] INFO Partition [jtop-migration-prod-els,52] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,52] from 1,5,4
> to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,240] INFO Partition [compass-home-record-prod-els,0] on
> broker 1: Shrinking ISR for partition [compass-home-record-prod-els,0] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,241] INFO Partition [jtop-migration-smoke-els,46] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,46] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,242] INFO Partition [stolmen-entity-prod-els,2] on
> broker 1: Shrinking ISR for partition [stolmen-entity-prod-els,2] from 1,5,6
> to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,243] INFO Partition [hummus-boundary-v2-prod-els,0] on
> broker 1: Shrinking ISR for partition [hummus-boundary-v2-prod-els,0] from
> 1,5,3 to 1,3 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,244] INFO Partition [jtop-migration-smoke-els,11] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,11] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,244] INFO Partition [flintan-asset-view-smoke-els,4] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,4] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,245] INFO Partition [stolmen-bundle-prod-els,3] on
> broker 1: Shrinking ISR for partition [stolmen-bundle-prod-els,3] from 1,5,6
> to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,246] INFO Partition [flintan-asset-view-prod-els,23] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,23] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,247] INFO Partition
> [resolver-notifcation-requests-prod-els,0] on broker 1: Shrinking ISR for
> partition [resolver-notifcation-requests-prod-els,0] from 1,5,6 to 1,6
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,248] INFO Partition [mj,0] on broker 1: Shrinking ISR
> for partition [mj,0] from 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,249] INFO Partition [flintan-asset-view-prod-els,17] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,17] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,250] INFO Partition [flintan-asset-view-prod-els,47] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,47] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,251] INFO Partition [jtop-migration-smoke-els,26] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,26] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,252] INFO Partition [__consumer_offsets,24] on broker 1:
> Shrinking ISR for partition [__consumer_offsets,24] from 1,5,6 to 1,6
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,253] INFO Partition [eds-incre-prod-els,0] on broker 1:
> Shrinking ISR for partition [eds-incre-prod-els,0] from 1,5,4 to 1,4
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,254] INFO Partition [jtop-migration-prod-els,17] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,17] from 1,5,6
> to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,254] INFO Partition [__consumer_offsets,44] on broker 1:
> Shrinking ISR for partition [__consumer_offsets,44] from 1,5,6 to 1,6
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,255] INFO Partition [jtop-migration-smoke-els,31] on
> broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,31] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,256] INFO Partition [__consumer_offsets,4] on broker 1:
> Shrinking ISR for partition [__consumer_offsets,4] from 1,5,6 to 1,6
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,257] INFO Partition [jtop-migration-prod-els,32] on
> broker 1: Shrinking ISR for partition [jtop-migration-prod-els,32] from 1,5,4
> to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,258] INFO Partition [flintan-asset-view-smoke-els,40] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,40] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,259] INFO Partition [__consumer_offsets,19] on broker 1:
> Shrinking ISR for partition [__consumer_offsets,19] from 1,5,4 to 1,4
> (kafka.cluster.Partition)
> [2017-02-19 13:01:17,260] INFO Partition [flintan-asset-view-smoke-els,10] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,10] from
> 1,5,6 to 1,6 (kafka.cluster.Partition)
> [2017-02-19 13:01:17,261] INFO Partition [flintan-asset-view-smoke-els,34] on
> broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,34] from
> 1,5,4 to 1,4 (kafka.cluster.Partition)
> [2017-02-19 13:01:27,320] INFO [GroupCoordinator 1]: Preparing to restabilize
> group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 with old
> generation 1 (kafka.coordinator.GroupCoordinator)
> [2017-02-19 13:01:27,320] INFO [GroupCoordinator 1]: Group
> flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 generation 1 is
> dead and removed (kafka.coordinator.GroupCoordinator)
> [2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Preparing to restabilize
> group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 with old
> generation 0 (kafka.coordinator.GroupCoordinator)
> [2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Stabilized group
> flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 generation 1
> (kafka.coordinator.GroupCoordinator)
> [2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Assignment received from
> leader for group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642
> for generation 1 (kafka.coordinator.GroupCoordinator)
> [2017-02-19 13:01:33,773] INFO Partition [stolmen-entity-prod-els,2] on
> broker 1: Expanding ISR for partition [stolmen-entity-prod-els,2] from 1,6 to
> 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,774] INFO Partition [jtop-migration-prod-els,57] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,57] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,775] INFO Partition [jtop-migration-smoke-els,31] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,31] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,776] INFO Partition [jtop-migration-smoke-els,26] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,26] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,777] INFO Partition [compass-home-record-prod-els,0] on
> broker 1: Expanding ISR for partition [compass-home-record-prod-els,0] from
> 1,4 to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,778] INFO Partition [flintan-asset-view-smoke-els,4] on
> broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,4] from
> 1,4 to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,779] INFO Partition [flintan-asset-view-smoke-els,10] on
> broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,10] from
> 1,6 to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,780] INFO Partition
> [resolver-notification-profiles-staging-els,1] on broker 1: Expanding ISR for
> partition [resolver-notification-profiles-staging-els,1] from 1,4 to 1,4,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,780] INFO Partition [hummus-asset-v2-prod-els,0] on
> broker 1: Expanding ISR for partition [hummus-asset-v2-prod-els,0] from 1,3
> to 1,3,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,781] INFO Partition [jtop-migration-smoke-els,6] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,6] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,782] INFO Partition
> [resolver-notifcation-requests-prod-els,0] on broker 1: Expanding ISR for
> partition [resolver-notifcation-requests-prod-els,0] from 1,6 to 1,6,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,782] INFO Partition [hummus-boundary-v2-prod-els,0] on
> broker 1: Expanding ISR for partition [hummus-boundary-v2-prod-els,0] from
> 1,3 to 1,3,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,783] INFO Partition [jtop-migration-smoke-els,51] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,51] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,784] INFO Partition [stolmen-bundle-prod-els,3] on
> broker 1: Expanding ISR for partition [stolmen-bundle-prod-els,3] from 1,6 to
> 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,785] INFO Partition [eds-incre-prod-els,0] on broker 1:
> Expanding ISR for partition [eds-incre-prod-els,0] from 1,4 to 1,4,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,785] INFO Partition [jtop-migration-prod-els,52] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,52] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,786] INFO Partition [mj,0] on broker 1: Expanding ISR
> for partition [mj,0] from 1,6 to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,787] INFO Partition [jtop-migration-smoke-els,46] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,46] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,788] INFO Partition [jtop-migration-prod-els,37] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,37] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,789] INFO Partition [jtop-migration-prod-els,12] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,12] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,790] INFO Partition [flintan-asset-view-smoke-els,40] on
> broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,40] from
> 1,6 to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,790] INFO Partition [khloe-updates-prod-els,0] on broker
> 1: Expanding ISR for partition [khloe-updates-prod-els,0] from 1,4 to 1,4,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,791] INFO Partition [jtop-migration-prod-els,17] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,17] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,792] INFO Partition [hummus-entity-merged-v2-prod-els,0]
> on broker 1: Expanding ISR for partition [hummus-entity-merged-v2-prod-els,0]
> from 1,4 to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,793] INFO Partition [flintan-asset-view-smoke-els,34] on
> broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,34] from
> 1,4 to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,794] INFO Partition [jtop-migration-prod-els,32] on
> broker 1: Expanding ISR for partition [jtop-migration-prod-els,32] from 1,4
> to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,795] INFO Partition [jtop-migration-smoke-els,11] on
> broker 1: Expanding ISR for partition [jtop-migration-smoke-els,11] from 1,6
> to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,804] INFO Partition [__consumer_offsets,44] on broker 1:
> Expanding ISR for partition [__consumer_offsets,44] from 1,6 to 1,6,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,805] INFO Partition [__consumer_offsets,4] on broker 1:
> Expanding ISR for partition [__consumer_offsets,4] from 1,6 to 1,6,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,806] INFO Partition [__consumer_offsets,24] on broker 1:
> Expanding ISR for partition [__consumer_offsets,24] from 1,6 to 1,6,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,807] INFO Partition [__consumer_offsets,39] on broker 1:
> Expanding ISR for partition [__consumer_offsets,39] from 1,4 to 1,4,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,808] INFO Partition [__consumer_offsets,19] on broker 1:
> Expanding ISR for partition [__consumer_offsets,19] from 1,4 to 1,4,5
> (kafka.cluster.Partition)
> [2017-02-19 13:01:33,809] INFO Partition [flintan-asset-view-prod-els,17] on
> broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,17] from
> 1,4 to 1,4,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,811] INFO Partition [flintan-asset-view-prod-els,23] on
> broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,23] from
> 1,6 to 1,6,5 (kafka.cluster.Partition)
> [2017-02-19 13:01:33,812] INFO Partition [flintan-asset-view-prod-els,47] on
> broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,47] from
> 1,4 to 1,4,5 (kafka.cluster.Partition)
>
> This time it kicks out broker 5 but it can kick all other brokers, and this
> only happens to broker 1, I don’t see any other broker has such scenario. I
> also take a look at controllers log, here’s the log at corresponding time:
>
> [2017-02-19 13:01:19,731] DEBUG [IsrChangeNotificationListener] Fired!!!
> (kafka.controller.IsrChangeNotificationListener)
> [2017-02-19 13:01:19,765] DEBUG Sending MetadataRequest to
> Brokers:ArrayBuffer(5, 1, 6, 2, 3, 4) for
> TopicAndPartitions:Set([__consumer_offsets,44], [stolmen-entity-prod-els,2],
> [jtop-migration-prod-els,57], [jtop-migration-smoke-els,31],
> [jtop-migration-smoke-els,26], [__consumer_offsets,4],
> [compass-home-record-prod-els,0], [flintan-asset-view-smoke-els,4],
> [flintan-asset-view-smoke-els,10], [__consumer_offsets,24],
> [resolver-notification-profiles-staging-els,1],
> [flintan-asset-view-prod-els,17], [hummus-asset-v2-prod-els,0],
> [jtop-migration-smoke-els,6], [resolver-notifcation-requests-prod-els,0],
> [hummus-boundary-v2-prod-els,0], [__consumer_offsets,39],
> [jtop-migration-smoke-els,51], [stolmen-bundle-prod-els,3],
> [eds-incre-prod-els,0], [flintan-asset-view-prod-els,23],
> [jtop-migration-prod-els,52], [mj,0], [jtop-migration-smoke-els,46],
> [flintan-asset-view-prod-els,47], [jtop-migration-prod-els,37],
> [jtop-migration-prod-els,12], [flintan-asset-view-smoke-els,40],
> [__consumer_offsets,19], [khloe-updates-prod-els,0],
> [jtop-migration-prod-els,17], [hummus-entity-merged-v2-prod-els,0],
> [flintan-asset-view-smoke-els,34], [jtop-migration-prod-els,32],
> [jtop-migration-smoke-els,11])
> (kafka.controller.IsrChangeNotificationListener)
> [2017-02-19 13:01:19,770] DEBUG [IsrChangeNotificationListener] Fired!!!
> (kafka.controller.IsrChangeNotificationListener)
> [2017-02-19 13:01:39,731] DEBUG [IsrChangeNotificationListener] Fired!!!
> (kafka.controller.IsrChangeNotificationListener)
> [2017-02-19 13:01:39,766] DEBUG Sending MetadataRequest to
> Brokers:ArrayBuffer(5, 1, 6, 2, 3, 4) for
> TopicAndPartitions:Set([__consumer_offsets,44], [stolmen-entity-prod-els,2],
> [jtop-migration-prod-els,57], [jtop-migration-smoke-els,31],
> [jtop-migration-smoke-els,26], [__consumer_offsets,4],
> [compass-home-record-prod-els,0], [flintan-asset-view-smoke-els,4],
> [flintan-asset-view-smoke-els,10], [__consumer_offsets,24],
> [resolver-notification-profiles-staging-els,1],
> [flintan-asset-view-prod-els,17], [hummus-asset-v2-prod-els,0],
> [jtop-migration-smoke-els,6], [resolver-notifcation-requests-prod-els,0],
> [hummus-boundary-v2-prod-els,0], [__consumer_offsets,39],
> [jtop-migration-smoke-els,51], [stolmen-bundle-prod-els,3],
> [eds-incre-prod-els,0], [flintan-asset-view-prod-els,23],
> [jtop-migration-prod-els,52], [mj,0], [jtop-migration-smoke-els,46],
> [flintan-asset-view-prod-els,47], [jtop-migration-prod-els,37],
> [jtop-migration-prod-els,12], [flintan-asset-view-smoke-els,40],
> [__consumer_offsets,19], [khloe-updates-prod-els,0],
> [jtop-migration-prod-els,17], [hummus-entity-merged-v2-prod-els,0],
> [flintan-asset-view-smoke-els,34], [jtop-migration-prod-els,32],
> [jtop-migration-smoke-els,11])
> (kafka.controller.IsrChangeNotificationListener)
> [2017-02-19 13:01:39,771] DEBUG [IsrChangeNotificationListener] Fired!!!
> (kafka.controller.IsrChangeNotificationListener)
>
> Sometimes when broker 1 shrink/expand ISR, controller will throw IOException
> saying that all other brokers was disconnected before the response was read,
> but this doesn’t happen on all shrink/expand so I’m not sure if this is also
> related.
>
> [2017-02-19 13:19:19,740] WARN [Controller-4-to-broker-2-send-thread],
> Controller 4 epoch 35 fails to send request
> {controller_id=4,controller_epoch=35,partition_states=[{topic=flintan-asset-view-prod-els,partition=29,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-prod-els,partition=59,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-smoke-els,partition=22,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=resolver-notification-profiles-staging-els,partition=1,controller_epoch=34,leader=-2,leader_epoch=0,isr=[1,4,5],zk_version=0,replicas=[1,4,5]},{topic=flintan-asset-view-smoke-els,partition=52,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=flintan-asset-view-smoke-els,partition=16,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=resolver-notification-profiles-staging-els,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[6,4,3],zk_version=0,replicas=[6,3,4]},{topic=flintan-asset-view-smoke-els,partition=46,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-prod-els,partition=35,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=flintan-asset-view-prod-els,partition=5,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]}],live_brokers=[{id=6,end_points=[{port=9092,host=elsconfluent006.server.hulu.com,security_protocol_type=0}]},{id=5,end_points=[{port=9092,host=elsconfluent005.server.hulu.com,security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=elsconfluent003.server.hulu.com,security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=elsconfluent004.server.hulu.com,security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=elsconfluent002.server.hulu.com,security_protocol_type=0}]},{id=1,end_points=[{port=9092,host=elsconfluent001.server.hulu.com,security_protocol_type=0}]}]}
> to broker Node(2, elsconfluent002.server.hulu.com, 9092). Reconnecting to
> broker. (kafka.controller.RequestSendThread)
> java.io.IOException: Connection to 2 was disconnected before the response was
> read
> at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
> at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
> at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
> at
> kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:129)
> at
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
> at
> kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
> at
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:180)
> at
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
>
> My understand on shrink/expand ISR is that broker n fails to sent fetch
> request to broker 1 in replica.lag.time.max.ms(10000), then broker 1 will
> kick broker n out of ISR. But from the log it doesn’t look like this. So what
> other reason could it be?
> Can someone help me with why this is happening and how can I react to this?
>
> Thanks!
> Jun
