Hi Jonathan, Thanks for the inputs. Actually we are not storing data in /tmp/directory :-) . Sure I'll update heap settings as advised.
--Senthil On Wed, 4 Sep, 2019, 6:24 PM Jonathan Santilli, <jonathansanti...@gmail.com> wrote: > Hello Senthil, > > I would recommend to not have the data in /tmp/data. > Also as a recommendation, set to equal values the -Xmx and -Xms parameters. > > > Cheers! > -- > Jonathan > > > > > On Wed, Sep 4, 2019 at 1:27 PM Karolis Pocius > <karolis.poc...@sentiance.com.invalid> wrote: > > > I had the same issue which was solved by increasing max_map_count > > https://stackoverflow.com/a/43675621 > > > > > > On Wed, Sep 4, 2019 at 2:59 PM SenthilKumar K <senthilec...@gmail.com> > > wrote: > > > > > Hello Experts , We have deployed 10 node kafka cluster in production. > > > Recently two of the nodes went down due to network problem and we > brought > > > it up after 24 hours. At the time of bootstrapping the kafka service > on > > > the failed nodes , we have seen the below error & broker failed to come > > up. > > > > > > Kafka Version : kafka_2.11-2.2.0 > > > > > > JVM Options : > > > /a/java64/jdk1.8.0/bin/java -Xmx15G -Xms10G -server -XX:+UseG1GC > > > -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 > > > -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true > > > -Xloggc:/a/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > > -XX:GCLogFileSize=100M > > > -Davoid_insecure_jmxremote > > > > > > > > > [2019-09-03 10:54:10,630] ERROR Error while deleting the clean shutdown > > > file in dir /tmp/data (kafka.server.LogDirFailureChannel) > > > java.io.IOException: Map failed > > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) > > > at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:126) > > > at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:53) > > > at kafka.log.LogSegment$.open(LogSegment.scala:632) > > > at > > > > > > > > > kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:467) > > > at > > > > > > > > > kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:454) > > > at > > > > > > > > > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > > > at > > > > > > > > > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > > > at > > scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) > > > at > > > > > > > > > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > > > at kafka.log.Log.kafka$log$Log$$loadSegmentFiles(Log.scala:454) > > > at > kafka.log.Log$$anonfun$loadSegments$1.apply$mcV$sp(Log.scala:565) > > > at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:559) > > > at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:559) > > > at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2024) > > > at kafka.log.Log.loadSegments(Log.scala:559) > > > at kafka.log.Log.<init>(Log.scala:292) > > > at kafka.log.Log$.apply(Log.scala:2157) > > > at > > > > kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:265) > > > at > > > > > > > > > kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345) > > > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > at java.lang.Thread.run(Thread.java:748) > > > Caused by: java.lang.OutOfMemoryError: Map failed > > > at sun.nio.ch.FileChannelImpl.map0(Native Method) > > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > > > ... 25 more > > > > > > Any hint to solve this problem ? Thanks in advance! > > > > > > --Senthil > > > > > > > > -- > Santilli Jonathan >
