Hi,
A Kafka broker in one of our production clusters crashed after a restart.
We have also observed same type of crashes on other smaller Kafka clusters
in our development environment during regular operation.
On the smaller Kafka clusters, there is no direct correlation to increase
in messages in or out per second and seems to be happening some time later
from the last crash, which feels like some problem is gradually building up
to cause the crash at the end.
These are the logs from the Kafka broker that crashed after the restart in
the production cluster.
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007ff036e10000, 65536, 1) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for
committing reserved memory.
# An error report file with more information is saved as:
# /opt/kafka/hs_err_pid1.log[thread 140632874415872 also had an error]
#
# Compiler replay data is saved as:
# /opt/kafka/replay_pid1.log
2018-07-23T23:40:23.798Z ERROR [ReplicaFetcherThread-2-3 ]
[kafka.server.LogDirFailureChannel ]: Error while rolling log
segment for my_topic-56 in dir /var/lib/kafka/logs/2
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:62)
at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:53)
at kafka.log.LogSegment.<init>(LogSegment.scala:75)
at kafka.log.Log.$anonfun$roll$2(Log.scala:1335)
at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
at kafka.log.Log.roll(Log.scala:1297)
at kafka.log.Log.maybeRoll(Log.scala:1284)
at kafka.log.Log.$anonfun$append$2(Log.scala:712)
at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
at kafka.log.Log.append(Log.scala:624)
at kafka.log.Log.appendAsFollower(Log.scala:607)
at
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:102)
at
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:41)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:184)
at scala.Option.foreach(Option.scala:257)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:172)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:169)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5(AbstractFetcherThread.scala:169)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
... 26 more
Production Kafka broker details:
Kafka version: 1.0.1
JVM version: Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed
mode)
Heap parameters: -Xmx4G -Xms4G
Total memory on the instance: 122 GB
CPU usage: ~50% during normal operation and 90% during startup/catchup.
Disk usage: ~25%, we have more than 800GB free on each broker.
Our observations:
After the crash, the command "free -g" indicates that 118 of 122 GB is
being used and when we cleared all of the message data on the broker, the
used memory falls down close from 118 to low values like 2 GB.
What we tried already:
Initially, we suspected vm.max_map_count because we were getting close to
default value 65530 on the production Kafka cluster brokers but this is
also happening on development Kafka clusters where each broker has very few
memory maps, around 600. Currently, our only option is to increase the
cluster capacity and hope that we won't run into this problem.
We would like to know the root cause and remediation for this problem. Any
help would be greatly appreciated.
