What's your ulimit -a output? Did you adjust nproc and nofile ulimits up? Do you have JNA installed? What about memlock ulimit and in sysctl.conf => kernel.shmmax?
What's in cassandra.log? On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj < rahul.bhard...@indiamart.com> wrote: > Hi Robert, > > I saw somewhere you answering the same prob but no solution found. plz > check again. > > > regards: > Rahul Bhardwaj > > > > > On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj < > rahul.bhard...@indiamart.com> wrote: > >> Bu it is giving same error with java 7 and open jdk >> >> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <anis...@gmail.com> >> wrote: >> >>> Well i havent used 2.1.x cassandra neither java 8 but any reason for not >>> using oracle JDK as i thought thats what is recommended. i saw a thread >>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about >>> 2.1.x versions. >>> >>> >>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj < >>> rahul.bhard...@indiamart.com> wrote: >>> >>>> PFA of error log >>>> hs_err_pid9656.log >>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web> >>>> >>>> >>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj < >>>> rahul.bhard...@indiamart.com> wrote: >>>> >>>>> free RAM: >>>>> >>>>> >>>>> free -m >>>>> total used free shared buffers >>>>> cached >>>>> Mem: 64398 23753 40644 0 108 >>>>> 8324 >>>>> -/+ buffers/cache: 15319 49078 >>>>> Swap: 2925 15 2909 >>>>> >>>>> >>>>> ulimit -a >>>>> core file size (blocks, -c) 0 >>>>> data seg size (kbytes, -d) unlimited >>>>> scheduling priority (-e) 0 >>>>> file size (blocks, -f) unlimited >>>>> pending signals (-i) 515041 >>>>> max locked memory (kbytes, -l) 64 >>>>> max memory size (kbytes, -m) unlimited >>>>> open files (-n) 1024 >>>>> pipe size (512 bytes, -p) 8 >>>>> POSIX message queues (bytes, -q) 819200 >>>>> real-time priority (-r) 0 >>>>> stack size (kbytes, -s) 10240 >>>>> cpu time (seconds, -t) unlimited >>>>> max user processes (-u) 515041 >>>>> virtual memory (kbytes, -v) unlimited >>>>> file locks (-x) unlimited >>>>> >>>>> >>>>> Also attaching complete error file >>>>> >>>>> >>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <anis...@gmail.com> >>>>> wrote: >>>>> >>>>>> the memory cassandra is trying to allocate is pretty small. you sure >>>>>> there is no hardware failure on the machine. what is the free ram on the >>>>>> box ? >>>>>> >>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj < >>>>>> rahul.bhard...@indiamart.com> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was >>>>>>> running in healthy state. Suddenly one machine's cassandra daemon stops >>>>>>> working and shut down. >>>>>>> >>>>>>> On restarting it after 2 minutes it again stops and is getting stop >>>>>>> after returning below error in cassandra.log >>>>>>> >>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot >>>>>>> allocate memory' (errno=12) >>>>>>> # >>>>>>> # There is insufficient memory for the Java Runtime Environment to >>>>>>> continue. >>>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes >>>>>>> for committing reserved memory. >>>>>>> # An error report file with more information is saved as: >>>>>>> # /tmp/hs_err_pid23215.log >>>>>>> INFO 09:50:41 Loading settings from >>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml >>>>>>> INFO 09:50:41 Node >>>>>>> configuration:[authenticator=AllowAllAuthenticator; >>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true; >>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; >>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; >>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64; >>>>>>> commit_failure_policy=stop; >>>>>>> commitlog_directory=/var/lib/cassandra/commitlog; >>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic; >>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; >>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; >>>>>>> concurrent_reads=32; >>>>>>> concurrent_writes=32; counter_cache_save_period=7200; >>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; >>>>>>> cross_node_timeout=false; >>>>>>> data_file_directories=[/var/lib/cassandra/data]; >>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; >>>>>>> dynamic_snitch_reset_interval_in_ms=600000; >>>>>>> dynamic_snitch_update_interval_in_ms=100; >>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; >>>>>>> hinted_handoff_enabled=true; >>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false; >>>>>>> index_summary_capacity_in_mb=null; >>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; >>>>>>> internode_compression=all; key_cache_save_period=14400; >>>>>>> key_cache_size_in_mb=null; listen_address=null; >>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; >>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042; >>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; >>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000; >>>>>>> read_request_timeout_in_ms=999999999; >>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler; >>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0; >>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true; >>>>>>> rpc_port=9160; rpc_server_type=sync; >>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches; >>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, >>>>>>> parameters=[{seeds=206.191.151.199}]}]; >>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false; >>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; >>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000; >>>>>>> thrift_framed_transport_size_in_mb=15; >>>>>>> tombstone_failure_threshold=100000; >>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false; >>>>>>> trickle_fsync_interval_in_kb=10240; >>>>>>> truncate_request_timeout_in_ms=60000; >>>>>>> write_request_timeout_in_ms=999999999] >>>>>>> ERROR 09:50:41 Exception encountered during startup >>>>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60] >>>>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60] >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) >>>>>>> ~[na:1.7.0_60] >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) >>>>>>> ~[na:1.7.0_60] >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) >>>>>>> ~[na:1.7.0_60] >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) >>>>>>> ~[na:1.7.0_60] >>>>>>> at >>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721) >>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643) >>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535) >>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) >>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) >>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) >>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>>>> at java.lang.Thread.start0(Native Method) >>>>>>> at java.lang.Thread.start(Thread.java:714) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) >>>>>>> at >>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) >>>>>>> at >>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>>>>>> at >>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721) >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643) >>>>>>> at >>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535) >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) >>>>>>> at >>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) >>>>>>> Exception encountered during startup: unable to create new native >>>>>>> thread >>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot >>>>>>> allocate memory' (errno=12 >>>>>>> >>>>>>> This is our production cluster and rest 2 nodes are still in healthy >>>>>>> state. >>>>>>> We have also changed heap size from 10G to 20 G but problem still >>>>>>> persists. Please help in finding the reason behind this. >>>>>>> >>>>>>> >>>>>>> Regards: >>>>>>> Rahul Bhardwaj >>>>>>> >>>>>>> >>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates >>>>>>> on this and more: <https://plus.google.com/+indiamart> >>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>>>>>> Mobile Channel: >>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>>>>>> <http://m.indiamart.com/> >>>>>>> >>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>>>>>> Check out some smart tips to generate more business - *IndiaMART >>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on >>>> this and more: <https://plus.google.com/+indiamart> >>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>>> Mobile Channel: >>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>>> <http://m.indiamart.com/> >>>> >>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>>> Check out some smart tips to generate more business - *IndiaMART >>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/> >>>> >>> >>> >> > > > Follow IndiaMART.com <http://www.indiamart.com> for latest updates on > this and more: <https://plus.google.com/+indiamart> > <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> > Mobile Channel: > <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> > <https://play.google.com/store/apps/details?id=com.indiamart.m> > <http://m.indiamart.com/> > > <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> > Check out some smart tips to generate more business - *IndiaMART Learning > Centre* <http://corporate.indiamart.com/quick-learn/> >
