[
https://issues.apache.org/jira/browse/KAFKA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-3917:
-------------------------------
Affects Version/s: (was: 0.8.2.2)
0.10.2.0
> Some __consumer_offsets replicas grow way too big
> -------------------------------------------------
>
> Key: KAFKA-3917
> URL: https://issues.apache.org/jira/browse/KAFKA-3917
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.2.0
> Environment: Runs with Docker 1.10.1 in a container on
> Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
> Reporter: Maxim Vladimirskiy
> Labels: reliability
>
> We noticed that some replicas of partitions of the __consumer_offsets topic
> grow way too big. Looking inside respective folders it became apparent that
> old segments had not been cleaned up. Please see below example of disk usage
> data for both affected and not affected partitions:
> Not affected partitions:
> Partition: 0 Leader: 2 Replicas: 2,3,4 Isr: 2,4,3
> 2: 49M
> 3: 49M
> 4: 49M
> Affected partitions:
> Partition: 10 Leader: 2 Replicas: 2,0,1 Isr: 1,2,0
> 0: 86M
> 1: 22G <<< too big!
> 2: 86M
> Partition: 38 Leader: 0 Replicas: 0,4,1 Isr: 1,0,4
> 0: 43M
> 1: 26G <<< too big!
> 4: 26G <<< too big!
> As you can see sometimes only one replica is affected, sometimes both
> replicas are affected.
> When I try to restart a broker that has affected replicas it fails to start
> with an exception that looks like this:
> [2016-06-28 23:15:20,441] ERROR There was an error in one of the threads
> during logs loading: java.lang.IllegalArgumentException: requirement failed:
> Corrupt index found, index file
> (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero
> size but the last offset is -676703869 and the base offset is 0
> (kafka.log.LogManager)
> [2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found,
> index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has
> non-zero size but the last offset is -676703869 and the base offset is 0
> at scala.Predef$.require(Predef.scala:233)
> at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.loadSegments(Log.scala:183)
> at kafka.log.Log.<init>(Log.scala:67)
> at
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
> at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> After the content of the affected partition is deleted broker starts
> successfully.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
