[ https://issues.apache.org/jira/browse/CASSANDRA-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joshua McKenzie updated CASSANDRA-8152: --------------------------------------- Resolution: Cannot Reproduce Reproduced In: 2.1.2, 2.1.0 (was: 2.1.0, 2.1.2) Status: Resolved (was: Awaiting Feedback) Closing as cannot reproduce as it's been half a year since last update. If you can get the information requested onto the ticket, we'd be happy to re-open and look into this further. > Cassandra crashes with Native memory allocation failure > ------------------------------------------------------- > > Key: CASSANDRA-8152 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8152 > Project: Cassandra > Issue Type: Bug > Environment: EC2 (i2.xlarge) > Reporter: Babar Tareen > Priority: Minor > Attachments: db06_hs_err_pid26159.log.zip, > db_05_hs_err_pid25411.log.zip > > > On a 6 node Cassandra (datastax-community-2.1) cluster running on EC2 > (i2.xlarge) instances, Jvm hosting the cassandra service randomly crashes > with following error. > {code} > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 12288 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # In 32 bit mode, the process size limit was hit > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Use 64 bit Java on a 64 bit OS > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2747), pid=26159, tid=140305605682944 > # > # JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build > 1.7.0_60-b19) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode > linux-amd64 compressed oops) > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > --------------- T H R E A D --------------- > Current thread (0x0000000008341000): JavaThread "MemtableFlushWriter:2055" > daemon [_thread_new, id=23336, stack(0x00007f9b71c56000,0x00007f9b71c97000)] > Stack: [0x00007f9b71c56000,0x00007f9b71c97000], sp=0x00007f9b71c95820, free > space=254k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [libjvm.so+0x99e7ca] VMError::report_and_die()+0x2ea > V [libjvm.so+0x496fbb] report_vm_out_of_memory(char const*, int, unsigned > long, char const*)+0x9b > V [libjvm.so+0x81d81e] os::Linux::commit_memory_impl(char*, unsigned long, > bool)+0xfe > V [libjvm.so+0x81d8dc] os::pd_commit_memory(char*, unsigned long, bool)+0xc > V [libjvm.so+0x81565a] os::commit_memory(char*, unsigned long, bool)+0x2a > V [libjvm.so+0x81bdcd] os::pd_create_stack_guard_pages(char*, unsigned > long)+0x6d > V [libjvm.so+0x9522de] JavaThread::create_stack_guard_pages()+0x5e > V [libjvm.so+0x958c24] JavaThread::run()+0x34 > V [libjvm.so+0x81f7f8] java_start(Thread*)+0x108 > {code} > Changes in cassandra-env.sh settings > {code} > MAX_HEAP_SIZE="8G" > HEAP_NEWSIZE="800M" > JVM_OPTS="$JVM_OPTS -XX:TargetSurvivorRatio=50" > JVM_OPTS="$JVM_OPTS -XX:+AggressiveOpts" > JVM_OPTS="$JVM_OPTS -XX:+UseLargePages" > {code} > Writes are about 10K-15K/sec and there are very few reads. Cassandra 2.0.9 > with same settings never crashed. JVM crash logs are attached from two > machines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)