[
https://issues.apache.org/jira/browse/KAFKA-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959521#comment-14959521
]
A.J. edited comment on KAFKA-2560 at 10/15/15 8:03 PM:
-------------------------------------------------------
What "bitness" of JVM are you using?
I ran into this with using a 32-bit JVM on Linux. Switching to a 64-bit JVM
remedied this issue.
It seems it's related to the JVM's ability to address large mapped files. The
32-bit JVM has a limit, which in my experience seems kinda small. Here's a
page online that anecdotally has some suggestions that lead from the cause:
http://javarevisited.blogspot.com/2014/11/javaioioexception-map-failed-javalangoutofmemoryerror.html
Basically, it seems this is a JVM limitation. Not something that Kafka can
correct.
The key part of the stack trace is:
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
was (Author: ajitator):
What "bitness" of JVM are you using?
I ran into this with using a 32-bit JVM on Linux. Switching to a 64-bit JVM
remedied this issue.
It seems it's related to the JVM's ability to address large mapped files. The
32-bit JVM has a limit, which in my experience seems kinda small. Here's a
page online that anecdotally has some suggestions that lead from the cause:
http://javarevisited.blogspot.com/2014/11/javaioioexception-map-failed-javalangoutofmemoryerror.html
Basically, it seems this is a JVM limitation. Not something that Kafka can
correct.
> Fatal error during KafkaServer startup because of Map failed error.
> -------------------------------------------------------------------
>
> Key: KAFKA-2560
> URL: https://issues.apache.org/jira/browse/KAFKA-2560
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.8.2.1
> Environment: Linux
> Reporter: Bo Wang
> Assignee: Jay Kreps
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I have 3 kafka nodes,
> create 30 topics ,every topic has 100 pations, and replica factor is 2.
> Kafka server start failed,
> 2015-09-21 10:28:35,668 | INFO | pool-2-thread-1 | Recovering unflushed
> segment 0 in log testTopic_14-34. |
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,942 | ERROR | main | There was an error in one of the
> threads during logs loading: java.io.IOException: Map failed |
> kafka.utils.Logging$class.error(Logging.scala:97)
> 2015-09-21 10:28:35,943 | INFO | pool-2-thread-5 | Recovering unflushed
> segment 0 in log testTopic_17-23. |
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,944 | INFO | pool-2-thread-5 | Completed load of log
> testTopic_17-23 with log end offset 0 |
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,945 | FATAL | main | [Kafka Server 54], Fatal error
> during KafkaServer startup. Prepare to shutdown |
> kafka.utils.Logging$class.fatal(Logging.scala:116)
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
> at
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at kafka.log.Log.loadSegments(Log.scala:179)
> at kafka.log.Log.<init>(Log.scala:67)
> at
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
> at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
> ... 13 more
> 2015-09-21 10:28:35,946 | INFO | pool-2-thread-5 | Recovering unflushed
> segment 0 in log testTopic_25-77. |
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,949 | INFO | main | [Kafka Server 54], shutting down |
> kafka.utils.Logging$class.info(Logging.scala:68)
> Kafka server host's top infomation below:
> top - 17:16:23 up 53 min, 6 users, load average: 0.42, 0.99, 1.19
> Tasks: 215 total, 1 running, 214 sleeping, 0 stopped, 0 zombie
> Cpu(s): 4.5%us, 2.4%sy, 0.0%ni, 92.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 40169M total, 6118M used, 34050M free, 9M buffers
> Swap: 0M total, 0M used, 0M free, 431M cached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
