Hi, thank you for response. using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.
# cat /proc/13405/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 260000 260000 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 255762 255762 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Is the stack size and / or Max open files is sufficient? The rest fd pretty much infinite. Jason On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink <clohfin...@gmail.com> wrote: > if your using 64 bit, check output of: > > cat /proc/{cassandra pid}/limits > > some older linux kernels wont work with above so if it doesnt exist check > the ulimit -a output for the cassandra user. max processes per user may > be your issue as well. > > --- > Chris Lohfink > > > On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <gra...@vast.com> > wrote: > >> First question are you running 32bit or 64bit… on 32bit you can easily >> run out of virtual address space for thread stacks. >> >> On Nov 10, 2014, at 8:25 AM, Jason Wee <peich...@gmail.com> wrote: >> >> Hello people, below is an extraction from cassandra system log. >> >> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java >> (line 139) Fatal exception in thread Thread[Thread-273,5,main] >> java.lang.OutOfMemoryError: unable to create new native thread >> at java.lang.Thread.start0(Native Method) >> at java.lang.Thread.start(Thread.java:640) >> at >> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) >> at >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104) >> at >> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214) >> >> Investigated into the call until the java native call, >> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698 >> >> if (native_thread->osthread() == NULL) { >> // No one should hold a reference to the 'native_thread'. >> delete native_thread; >> if (JvmtiExport::should_post_resource_exhausted()) { >> JvmtiExport::post_resource_exhausted( >> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | >> JVMTI_RESOURCE_EXHAUSTED_THREADS, >> "unable to create new native thread"); >> } >> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), >> "unable to create new native thread"); >> } >> >> Question. Is that out of memory error due to native os memory or java >> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor >> max user processes 260000. open files capped at 65536 >> >> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR >> and JVMTI_RESOURCE_EXHAUSTED_THREADS means too? >> >> Thank you. >> >> Jason >> >> >> >