K Zakee created KAFKA-2038:
------------------------------
Summary: Unable to restart brokers after it went down with no
space left on disk
Key: KAFKA-2038
URL: https://issues.apache.org/jira/browse/KAFKA-2038
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2.1
Reporter: K Zakee
Priority: Blocker
On of the 8 volumes on brokers disks were full and broker wen down with below
error, Cannot restart this broker, as it costantly throws the following error
on startup.
[2015-03-21 03:12:21,433] FATAL [app=broker] [ReplicaFetcherThread-6-3]
[ReplicaFetcherThread-6-3], Disk error while replicating data.
(kafka.server.ReplicaFetcherThread)
kafka.common.KafkaStorageException: I/O exception in append to log ‘Topic-11'
at kafka.log.Log.append(Log.scala:266)
at
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:54)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at
kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:39)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)
at
kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:133)
at kafka.log.FileMessageSet.append(FileMessageSet.scala:210)
at kafka.log.LogSegment.append(LogSegment.scala:85)
at kafka.log.Log.append(Log.scala:309)
... 12 more
=================
[2015-03-21 10:38:25,244] INFO [app=broker] [main] [Kafka Server 5], shut down
completed (kafka.server.KafkaServer)
[2015-03-21 10:38:25,245] FATAL [app=broker] [main] Fatal error during
KafkaServerStartable startup. Prepare to shutdown
(kafka.server.KafkaServerStartable)
java.lang.InternalError: a fault occurred in a recent unsafe memory access
operation in compiled Java code
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188)
at kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.log.LogSegment.recover(LogSegment.scala:175)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:162)
at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:141)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at kafka.log.Log.loadSegments(Log.scala:141)
at kafka.log.Log.<init>(Log.scala:67)
at
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
