They should be trying to get back into sync on their own. Do you see any errors in broker logs?
Gwen On Tue, Apr 21, 2015 at 10:15 AM, Thomas Kwan <thomas.k...@manage.com> wrote: > We have 5 kafka brokers available, and created a topic with replication > factor of 3. After a few broker issues (e.g. went out of file descriptors), > running kafkacat on the producer node shows the following: > > Command: > > kafkacat-CentOS-6.5-x86_64 -L -b "kafka01-east.manage.com, > kafka02-east.manage.com,kafka03-east.manage.com,kafka04-east.manage.com, > kafka05-east.manage.com" > > Output: > > 5 brokers: > broker 385 at kafka04-east.manage.com:9092 > broker 389 at kafka03-east.manage.com:9092 > broker 381 at kafka01-east.manage.com:9092 > broker 387 at kafka05-east.manage.com:9092 > broker 383 at kafka02-east.manage.com:9092 > ... > topic "raw-events" with 32 partitions: > partition 23, leader 387, replicas: 389,387,381, isrs: 387,389 > partition 8, leader 389, replicas: 381,389,383, isrs: 389,381 > partition 17, leader 389, replicas: 383,389,381, isrs: 389,381 > partition 26, leader 387, replicas: 387,389,381, isrs: 387,389 > partition 11, leader 387, replicas: 389,387,381, isrs: 387,389 > partition 29, leader 389, replicas: 383,389,381, isrs: 389,381 > partition 20, leader 389, replicas: 381,389,383, isrs: 389,381 > partition 2, leader 387, replicas: 387,389,381, isrs: 387 > partition 5, leader 389, replicas: 383,389,381, isrs: 389,381 > partition 14, leader 387, replicas: 387,389,381, isrs: 387,389 > partition 4, leader 387, replicas: 381,387,389, isrs: 387,389 > partition 13, leader 387, replicas: 383,387,389, isrs: 387,389 > partition 22, leader 389, replicas: 387,383,389, isrs: 389,387 > partition 31, leader 387, replicas: 389,383,387, isrs: 387,389 > partition 7, leader 387, replicas: 389,383,387, isrs: 387,389 > partition 16, leader 387, replicas: 381,387,389, isrs: 387 > partition 25, leader 387, replicas: 383,387,389, isrs: 387,389 > partition 10, leader 387, replicas: 387,383,389, isrs: 387,389 > partition 1, leader 387, replicas: 383,387,389, isrs: 387,389 > partition 28, leader 387, replicas: 381,387,389, isrs: 387 > partition 19, leader 387, replicas: 389,383,387, isrs: 387,389 > partition 18, leader 387, replicas: 387,381,383, isrs: 387,381 > partition 9, leader 387, replicas: 383,381,387, isrs: 387,381 > partition 27, leader 389, replicas: 389,381,383, isrs: 389,381 > partition 12, leader 387, replicas: 381,383,387, isrs: 387,381 > partition 21, leader 387, replicas: 383,381,387, isrs: 387,381 > partition 3, leader 389, replicas: 389,381,383, isrs: 389,381 > partition 30, leader 387, replicas: 387,381,383, isrs: 387,381 > partition 15, leader 389, replicas: 389,381,383, isrs: 389,381 > partition 6, leader 387, replicas: 387,381,383, isrs: 387,381 > partition 24, leader 387, replicas: 381,383,387, isrs: 387,381 > partition 0, leader 387, replicas: 381,383,387, isrs: 387,381 > > I notice that some partition (partition #2 for example) only has 1 node > under isrs. From what I read, isrs shows a list of brokers that have data > that is in-sync. > > My question is - now some partitions are out of sync. What do I do to get > them in sync again? > > thanks > thomas
