Aled Sage created BROOKLYN-181:
----------------------------------
Summary: OutOfMemoryError: unable to create new native thread
(when stopping app)
Key: BROOKLYN-181
URL: https://issues.apache.org/jira/browse/BROOKLYN-181
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Aled Sage
Using 0.9.0-SNAPSHOT...
I deployed an app to OpenStack (Bluebox) - it failed due to insufficient
security groups etc. I terminated the VMs in Bluebox itself, and then I tried
to call the "Stop" effector on the app in Brooklyn. Eventually Brooklyn ran out
of threads.
My app consisted of a riak cluster + web cluster.
However, the log is messy because I had a couple of other apps that I also
started/stopped during this time (I think they are unrelated though).
There are 2045 threads, of which 2012 are of the form "brooklyn-execmanager-*".
998 of these threads are doing:
{noformat}
"brooklyn-execmanager-vvDaziiT-2847" daemon prio=5 tid=0x00007fd4f08aa800
nid=0xc7d0f waiting for monitor entry [0x0000700063d3f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:83)
- waiting to lock <0x00000007c5940bc8> (a
org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
at
org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Another 999 threads look like:
{noformat}
"brooklyn-execmanager-vvDaziiT-2930" daemon prio=5 tid=0x00007fd4ef16f800
nid=0x5e07 in Object.wait() [0x0000700001703000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:275)
- locked <0x00000007c64bafa8> (a java.lang.Object)
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:265)
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The logging shows the number of threads creeping up, with 30 more threads and
30 more active tasks being added every minute, e.g.:
{noformat}
2015-10-12 23:01:14,022 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector
[brooklyn-gc]: brooklyn gc (after) - using 164 MB / 264 MB memory (20.9 kB
soft); 2001 threads; storage: {datagrid={size=7, createCount=7}, refsMapSize=0,
lis
tsMapSize=0}; tasks: 1981 active, 999 unfinished; 1084 remembered, 286554 total
submitted)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
