|
Description:
|
We are running a Rails 3.2.13 app on Torquebox 2.3.0, and we see a intermittent issue where some of the servers which are not under load show a high CPU(100-800% - 16 CPU server). Once they get into this high CPU state the CPU does not come back down.
I dumped the threads while in this high CPU state, and the active threads all seem to contain a similar traces. In fact the number of these threads is proportional to the percent of CPU consumed. For example some servers are a 100% CPU and there is one of these threads, some are at 600% with 6 threads, 300% with 3
We are using JRuby 1.7.2 with the jruby.compile.mode=JIT. Java version is 1.7.0_10
Here are the two different types of threads I see. Where we have multiple instances of these same types of threads
Thread 3433: (state = IN_JAVA)
- java.lang.String.valueOf(int) @bci=1, line=2959 (Compiled frame; information may be imprecise)
- com.newrelic.agent.instrumentation.ClassloaderJarExtractor.isNewLoader(java.lang.ClassLoader) @bci=19, line=102 (Compiled frame)
- com.newrelic.agent.instrumentation.ClassloaderJarExtractor.grabJarInformation(java.lang.ClassLoader) @bci=21, line=56 (Interpreted frame)
- com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=5, line=150 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
- sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Compiled frame)
- java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
- java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=791 (Interpreted frame)
- org.jruby.util.ClassCache$OneShotClassLoader.defineClass(java.lang.String, byte[]) @bci=9, line=73 (Interpreted frame)
- org.jruby.util.ClassCache.defineClass(org.jruby.util.ClassCache$ClassGenerator) @bci=108, line=128 (Interpreted frame)
- org.jruby.util.ClassCache.cacheClassByKey(java.lang.Object, org.jruby.util.ClassCache$ClassGenerator) @bci=57, line=96 (Interpreted frame)
- org.jruby.compiler.JITCompiler$1.run() @bci=304, line=167 (Interpreted frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Compiled frame)
- java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
- java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
Thread 3434: (state = IN_JAVA)
- java.util.HashMap.getEntry(java.lang.Object) @bci=72, line=446 (Compiled frame; information may be imprecise)
- java.util.HashMap.get(java.lang.Object) @bci=11, line=405 (Compiled frame)
- com.newrelic.agent.instrumentation.ClassloaderJarExtractor.isNewLoader(java.lang.ClassLoader) @bci=28, line=103 (Compiled frame)
- com.newrelic.agent.instrumentation.ClassloaderJarExtractor.grabJarInformation(java.lang.ClassLoader) @bci=21, line=56 (Interpreted frame)
- com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=5, line=150 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
- sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Compiled frame)
- java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
- java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=791 (Interpreted frame)
- org.jruby.util.ClassCache$OneShotClassLoader.defineClass(java.lang.String, byte[]) @bci=9, line=73 (Interpreted frame)
- org.jruby.util.ClassCache.defineClass(org.jruby.util.ClassCache$ClassGenerator) @bci=108, line=128 (Interpreted frame)
- org.jruby.util.ClassCache.cacheClassByKey(java.lang.Object, org.jruby.util.ClassCache$ClassGenerator) @bci=57, line=96 (Interpreted frame)
- org.jruby.compiler.JITCompiler$1.run() @bci=304, line=167 (Interpreted frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Compiled frame)
- java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
- java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
In the past we have seen similar issues running an app with the jruby.compile.mode set to JIT, changing it to FORCE seemed to avoid(or move) the issue, but we are hoping to use JIT.
To work around this we now set jruby.jit.background="" and this seems to eliminate these high CPU occurrences.
|
