Hi Ignasi,
It looks like a simple issue of not closing the ComputeServiceContext.
When I added that, all of the background threads seem to exit nicely.
Now I just have to mop up the 60+ places in our code where the contexts
are created and never closed...
Thanks,
--Ryan
On May6 10:57 AM, Ryan Shoemaker wrote:
Hi Ignasi,
I've attached the test code, including the quick&dirty thread set
comparisons. You can strip that out and just watch the threads via
jconsole or some other monitoring app if you prefer.
Thanks,
--Ryan
On May6 3:44 AM, Ignasi Barrera wrote:
Hi Ryan,
Can you share the complete test code, including how you create the
jclouds context and the loop you're using, so I can run it to
reproduce the issue?
Thanks!
I.
El 05/05/2015 21:11, "Ryan Shoemaker"
<[email protected]
<mailto:[email protected]>> escribió:
Hi,
I'm running jclouds 1.8.0 with Guava 17.0 and I'm seeing threads
allocated every time I create an SshClient, but they are never
released and eventually pile up until the JVM can't allocate any
additional threads (java.lang.OutOfMemoryError: unable to create
new native thread).
I hacked java.lang.Thread in my JDK to provide logging
information every time a thread is created, renamed, or stopped
and here's a stack I captured of the thread created during the
creation of an SshClient:
init() pool-124-thread-1
at java.lang.Thread.init(Thread.java:350)
at java.lang.Thread.init(Thread.java:330)
at java.lang.Thread.<init>(Thread.java:650)
at
java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:572)
at
com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:162)
at
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:610)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
at
org.jclouds.concurrent.config.WithSubmissionTrace$ListeningExecutorService.submit(WithSubmissionTrace.java:69)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:130)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:123)
at
com.google.common.base.Predicates$OrPredicate.apply(Predicates.java:393)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:99)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40)
at <my application code below here that creates the SshClient>
This thread is subsequently renamed something along the lines of
"user thread 0" and the stack for the actual thread is:
Thread "user thread 0" thread-id: 1,853 thread-state: WAITING Waiting on
lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33a82f62
at: sun.misc.Unsafe.park(Native Method)
at:
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at:
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
at:
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
at:
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
at:
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:701)
Here's my test code:
SshClient ssh = null;
try {
ComputeService compute = ...;
NodeMetadata meta = compute.getNodeMetadata(...);
final LoginCredentials login
=
LoginCredentials.builder().identity("username").privateKey(...).noPassword().build();
NodeMetadata newMeta =
NodeMetadataBuilder
.fromNodeMetadata(meta)
.credentials(login).build();
Utils utils = compute.getContext().utils();
ssh = utils.sshForNode().apply(newMeta);
ssh.connect();
Thread.sleep(5000);
} catch (Exception ignored) {}
finally {
if (ssh != null) ssh.disconnect();
}
I threw this code in a 10 iteration loop and watched the JVM in
jconsole. After this code completes, there are 10 additional
threads named "user thread n" hanging around that never exit.
Any idea what is preventing these threads from exiting? Any
suggestions on what to try next?
Thanks,
--Ryan
