That is the most common reason threads remain there forever. When the jclouds context is created, two ExecutorServices are created: an I/O executor that runHTTP stuff, and a "user executor" that runs other concurrent tasks. Both executors need to be shut down, and that happens when you close the context.
Apart from that, you might want to configure the "jclouds.user-threads" when creating the context. It is set to 0 by default and that creates an unbounded cached thread pool [2], so threads might be created and remain there for some time. You set a custom value for that property in order to have more control on the size of the thread pool. HTH! I. [1] https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/Constants.java#L26-L31 [2] http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool() On 6 May 2015 at 23:17, Ryan Shoemaker <[email protected]> wrote: > Hi Ignasi, > > It looks like a simple issue of not closing the ComputeServiceContext. When > I added that, all of the background threads seem to exit nicely. Now I just > have to mop up the 60+ places in our code where the contexts are created and > never closed... > > Thanks, > > --Ryan > > > On May6 10:57 AM, Ryan Shoemaker wrote: > > Hi Ignasi, > > I've attached the test code, including the quick&dirty thread set > comparisons. You can strip that out and just watch the threads via jconsole > or some other monitoring app if you prefer. > > Thanks, > > --Ryan > > On May6 3:44 AM, Ignasi Barrera wrote: > > Hi Ryan, > > Can you share the complete test code, including how you create the jclouds > context and the loop you're using, so I can run it to reproduce the issue? > > Thanks! > > I. > > El 05/05/2015 21:11, "Ryan Shoemaker" <[email protected]> > escribió: >> >> Hi, >> >> I'm running jclouds 1.8.0 with Guava 17.0 and I'm seeing threads allocated >> every time I create an SshClient, but they are never released and eventually >> pile up until the JVM can't allocate any additional threads >> (java.lang.OutOfMemoryError: unable to create new native thread). >> >> I hacked java.lang.Thread in my JDK to provide logging information every >> time a thread is created, renamed, or stopped and here's a stack I captured >> of the thread created during the creation of an SshClient: >> >> init() pool-124-thread-1 >> at java.lang.Thread.init(Thread.java:350) >> at java.lang.Thread.init(Thread.java:330) >> at java.lang.Thread.<init>(Thread.java:650) >> at >> java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:572) >> at >> com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:162) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:610) >> at >> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925) >> at >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361) >> at >> com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484) >> at >> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106) >> at >> com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50) >> at >> org.jclouds.concurrent.config.WithSubmissionTrace$ListeningExecutorService.submit(WithSubmissionTrace.java:69) >> at >> org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:130) >> at >> org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:123) >> at >> com.google.common.base.Predicates$OrPredicate.apply(Predicates.java:393) >> at >> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) >> at >> org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:99) >> at >> org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) >> at >> org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) >> at <my application code below here that creates the SshClient> >> >> >> This thread is subsequently renamed something along the lines of "user >> thread 0" and the stack for the actual thread is: >> >> Thread "user thread 0" thread-id: 1,853 thread-state: WAITING Waiting on >> lock: >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33a82f62 >> at: sun.misc.Unsafe.park(Native Method) >> at: >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >> at: >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >> at: >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) >> at: >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) >> at: >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131) >> at: >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at: java.lang.Thread.run(Thread.java:701) >> >> >> Here's my test code: >> >> SshClient ssh = null; >> try { >> ComputeService compute = ...; >> NodeMetadata meta = compute.getNodeMetadata(...); >> final LoginCredentials login >> = >> LoginCredentials.builder().identity("username").privateKey(...).noPassword().build(); >> NodeMetadata newMeta = >> NodeMetadataBuilder >> .fromNodeMetadata(meta) >> .credentials(login).build(); >> Utils utils = compute.getContext().utils(); >> ssh = utils.sshForNode().apply(newMeta); >> ssh.connect(); >> Thread.sleep(5000); >> } catch (Exception ignored) {} >> finally { >> if (ssh != null) ssh.disconnect(); >> } >> >> I threw this code in a 10 iteration loop and watched the JVM in jconsole. >> After this code completes, there are 10 additional threads named "user >> thread n" hanging around that never exit. >> >> Any idea what is preventing these threads from exiting? Any suggestions >> on what to try next? >> >> Thanks, >> >> --Ryan >> >> > >
