if your using 64 bit, check output of:
cat /proc/{cassandra pid}/limits
some older linux kernels wont work with above so if it doesnt exist check
the ulimit -a output for the cassandra user. max processes per user may be
your issue as well.
---
Chris Lohfink
On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <[email protected]> wrote:
> First question are you running 32bit or 64bit… on 32bit you can easily run
> out of virtual address space for thread stacks.
>
> On Nov 10, 2014, at 8:25 AM, Jason Wee <[email protected]> wrote:
>
> Hello people, below is an extraction from cassandra system log.
>
> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
> at
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>
> Investigated into the call until the java native call,
> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>
> if (native_thread->osthread() == NULL) {
> // No one should hold a reference to the 'native_thread'.
> delete native_thread;
> if (JvmtiExport::should_post_resource_exhausted()) {
> JvmtiExport::post_resource_exhausted(
> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
> JVMTI_RESOURCE_EXHAUSTED_THREADS,
> "unable to create new native thread");
> }
> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
> "unable to create new native thread");
> }
>
> Question. Is that out of memory error due to native os memory or java
> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
> max user processes 260000. open files capped at 65536
>
> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
> and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>
> Thank you.
>
> Jason
>
>
>
