[jira] [Updated] (KAFKA-1791) Corrupt index after safe shutdown and restart
[ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vamsi Subhash Achanta updated KAFKA-1791: - Attachment: 0233.log Attached the log file. Corrupt index after safe shutdown and restart - Key: KAFKA-1791 URL: https://issues.apache.org/jira/browse/KAFKA-1791 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Environment: Debian6 with Sun-Java6 Reporter: Vamsi Subhash Achanta Priority: Critical Attachments: 0233.index, 0233.log We have 3 kafka brokers - all VMs. One of the broker was stopped for around 30 minutes to fix a problem with the bare metal. Upon restart, after some time, the broker went out of file-descriptors (FDs) and started throwing errors. Upon restart, it started throwing this corrupted index exceptions. I followed the other JIRAs related to corrupted indices but the solutions mentioned there (deleting the index and restart) didn't work - the index gets created again. The other JIRAs solution of deleting those indexes which got wrongly compacted ( 10MB) didn't work either. What is the error? How can I fix this and bring back the broker? Thanks. INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean shutdown file. Skipping recovery for all logs in data directory '/var/lib/fk-3p-kafka/logs' INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log 'kf.production.b2b.return_order.status-25' FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable - Fatal error during KafkaServerStable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/0233.index) has non-zero size but the last offset is 233 and the base offset is 233 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) at kafka.log.LogManager.loadLogs(LogManager.scala:105) at kafka.log.LogManager.init(LogManager.scala:57) at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275) at kafka.server.KafkaServer.startup(KafkaServer.scala:72) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka Server 2], shutting down INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka Server 2], shut down completed INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer - [Kafka Server 2], shutting down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1791) Corrupt index after safe shutdown and restart
[ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vamsi Subhash Achanta updated KAFKA-1791: - Attachment: 0233.index This was over a period of time when the exception is thrown again and again. The file size when copied increased to 10MB. Was it compressed? Corrupt index after safe shutdown and restart - Key: KAFKA-1791 URL: https://issues.apache.org/jira/browse/KAFKA-1791 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Environment: Debian6 with Sun-Java6 Reporter: Vamsi Subhash Achanta Priority: Critical Attachments: 0233.index We have 3 kafka brokers - all VMs. One of the broker was stopped for around 30 minutes to fix a problem with the bare metal. Upon restart, after some time, the broker went out of file-descriptors (FDs) and started throwing errors. Upon restart, it started throwing this corrupted index exceptions. I followed the other JIRAs related to corrupted indices but the solutions mentioned there (deleting the index and restart) didn't work - the index gets created again. The other JIRAs solution of deleting those indexes which got wrongly compacted ( 10MB) didn't work either. What is the error? How can I fix this and bring back the broker? Thanks. INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean shutdown file. Skipping recovery for all logs in data directory '/var/lib/fk-3p-kafka/logs' INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log 'kf.production.b2b.return_order.status-25' FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable - Fatal error during KafkaServerStable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/0233.index) has non-zero size but the last offset is 233 and the base offset is 233 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118) at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113) at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32) at kafka.log.LogManager.loadLogs(LogManager.scala:105) at kafka.log.LogManager.init(LogManager.scala:57) at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275) at kafka.server.KafkaServer.startup(KafkaServer.scala:72) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka Server 2], shutting down INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka Server 2], shut down completed INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer - [Kafka Server 2], shutting down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
