Folks, Following the instructions at <https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging>, I got the thread dump and the heap summary information from a Jenkins installation that is hanging. Jenkins did proceed to finish doing the build and the rake task created the .dmg that it was supposed to create after building the code, but Jenkins remained non-responsive via the web interface during and since this event.
The server is running Mac OS X 10.7.4, with Jenkins 1.475, using Sami Tikka's "Jenkins.App" project (see <https://github.com/stisti/jenkins-app>). I can also give you a list of plugins we've got installed, if that's useful. Any advice or assistance you can provide will be greatly appreciated. Thanks! thread dump: ========== $ jstack 38315 2012-07-27 16:41:09 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed mode): "Attach Listener" daemon prio=9 tid=7fc21996d800 nid=0x115a9d000 runnable [00000000] java.lang.Thread.State: RUNNABLE "Timer-0" daemon prio=5 tid=7fc21e52f800 nid=0x117391000 in Object.wait() [117390000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f6526058> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <7f6526058> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "RequestHandlerThread[#20]" daemon prio=5 tid=7fc2198a2000 nid=0x118aa8000 waiting on condition [118aa7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#19]" daemon prio=5 tid=7fc21c52e800 nid=0x1189a5000 waiting on condition [1189a4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#18]" daemon prio=5 tid=7fc2198c6800 nid=0x1188a2000 waiting on condition [1188a1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#17]" daemon prio=5 tid=7fc21daa2000 nid=0x11879f000 waiting on condition [11879e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#16]" daemon prio=5 tid=7fc21e0bd000 nid=0x11852c000 waiting on condition [11852b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#15]" daemon prio=5 tid=7fc21d837000 nid=0x115c02000 waiting on condition [115c01000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "pool-4-thread-1" daemon prio=5 tid=7fc21de25800 nid=0x11869c000 runnable [11869b000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f471c2f0> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at com.youdevise.hudson.slavestatus.SocketHTTPListener.waitForConnection(SlaveListener.java:132) at com.youdevise.hudson.slavestatus.SlaveListener$1.run(SlaveListener.java:63) at com.youdevise.hudson.slavestatus.Daemon.go(Daemon.java:16) at com.youdevise.hudson.slavestatus.SlaveListener.call(SlaveListener.java:83) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) "Executor #1 for master" prio=5 tid=7fc21ea8c800 nid=0x1169fe000 in Object.wait() [1169fd000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f48cba10> (a hudson.model.Queue) at java.lang.Object.wait(Object.java:485) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) - locked <7f48cba10> (a hudson.model.Queue) at hudson.model.Queue.pop(Queue.java:804) - locked <7f48cba10> (a hudson.model.Queue) at hudson.model.Executor.grabJob(Executor.java:284) at hudson.model.Executor.run(Executor.java:205) - locked <7f48cba10> (a hudson.model.Queue) "JmDNS(zeus-local.local.).State.Timer" prio=5 tid=7fc21e32c000 nid=0x11599a000 in Object.wait() [115999000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f47172f0> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <7f47172f0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "JmDNS(zeus-local.local.).Timer" daemon prio=5 tid=7fc21e214800 nid=0x115897000 runnable [115896000] java.lang.Thread.State: RUNNABLE at javax.jmdns.impl.DNSCache.getDNSEntryList(DNSCache.java:364) at javax.jmdns.impl.DNSCache.getDNSEntry(DNSCache.java:399) - locked <7f496f7b8> (a javax.jmdns.impl.DNSCache) at javax.jmdns.impl.JmDNSImpl.getServiceInfoFromCache(JmDNSImpl.java:774) at javax.jmdns.impl.JmDNSImpl.updateRecord(JmDNSImpl.java:1258) at javax.jmdns.impl.JmDNSImpl.cleanCache(JmDNSImpl.java:1775) at javax.jmdns.impl.tasks.RecordReaper.run(RecordReaper.java:58) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) "SocketListener(zeus-local.local.)" daemon prio=5 tid=7fc21e266000 nid=0x115794000 runnable [115793000] java.lang.Thread.State: RUNNABLE at java.util.LinkedList.toArray(LinkedList.java:863) at java.util.Collections$SynchronizedCollection.toArray(Collections.java:1566) - locked <7f7166688> (a java.util.Collections$SynchronizedList) at java.util.ArrayList.addAll(ArrayList.java:472) at javax.jmdns.impl.DNSMessage.getAllAnswers(DNSMessage.java:138) at javax.jmdns.impl.JmDNSImpl.handleResponse(JmDNSImpl.java:1413) at javax.jmdns.impl.SocketListener.run(SocketListener.java:60) "Jenkins UDP 33848 monitoring thread" prio=5 tid=7fc219890800 nid=0x1134d4000 runnable [1134d3000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <7f56daa28> (a java.net.PlainDatagramSocketImpl) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) - locked <7f56daa28> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:725) - locked <7f553eed0> (a java.net.DatagramPacket) - locked <7f54af720> (a java.net.MulticastSocket) at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:79) "TCP slave agent listener port=0" prio=5 tid=7fc21eb24800 nid=0x112bad000 runnable [112bac000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f471c4d0> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) "RequestHandlerThread[#14]" daemon prio=5 tid=7fc2198c6000 nid=0x118429000 waiting on condition [118428000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#13]" daemon prio=5 tid=7fc2198c5000 nid=0x118326000 waiting on condition [118325000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#12]" daemon prio=5 tid=7fc2198c4800 nid=0x118223000 waiting on condition [118222000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#11]" daemon prio=5 tid=7fc2198c3800 nid=0x118120000 waiting on condition [11811f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#10]" daemon prio=5 tid=7fc21de24000 nid=0x11801d000 waiting on condition [11801c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#9]" daemon prio=5 tid=7fc219c29000 nid=0x117f1a000 waiting on condition [117f19000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#8]" daemon prio=5 tid=7fc21eb1e800 nid=0x117e17000 waiting on condition [117e16000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#7]" daemon prio=5 tid=7fc21de3e800 nid=0x117aa0000 waiting on condition [117a9f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#6]" daemon prio=5 tid=7fc219d3a000 nid=0x11799d000 waiting on condition [11799c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#5]" daemon prio=5 tid=7fc21de3a000 nid=0x117796000 waiting on condition [117795000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "Handling GET /login : RequestHandlerThread[#4]" daemon prio=5 tid=7fc219acb000 nid=0x11711a000 runnable [117116000] java.lang.Thread.State: RUNNABLE at java.util.zip.ZipFile.getEntry(Native Method) at java.util.zip.ZipFile.getEntry(ZipFile.java:161) - locked <7f46cf828> (a java.util.jar.JarFile) at java.util.jar.JarFile.getEntry(JarFile.java:208) at java.util.jar.JarFile.getJarEntry(JarFile.java:191) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:757) at sun.misc.URLClassPath.getResource(URLClassPath.java:169) at sun.misc.URLClassPath.getResource(URLClassPath.java:221) at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1150) at java.lang.ClassLoader.getResource(ClassLoader.java:999) at java.lang.ClassLoader.getResource(ClassLoader.java:997) at java.lang.ClassLoader.getResource(ClassLoader.java:997) at java.lang.ClassLoader.getResource(ClassLoader.java:997) at java.lang.ClassLoader.getResource(ClassLoader.java:997) at org.kohsuke.stapler.lang.KlassNavigator$1.getResource(KlassNavigator.java:77) at org.kohsuke.stapler.lang.KlassNavigator$1.getResource(KlassNavigator.java:61) at org.kohsuke.stapler.lang.Klass.getResource(Klass.java:27) at org.kohsuke.stapler.AbstractTearOff.getResource(AbstractTearOff.java:94) at org.kohsuke.stapler.AbstractTearOff.resolveScript(AbstractTearOff.java:69) at org.kohsuke.stapler.AbstractTearOff.loadScript(AbstractTearOff.java:78) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:30) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:27) at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:355) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:184) - locked <7f7164b88> (a com.google.common.collect.MapMakerInternalMap$StrongEntry) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:153) at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69) at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393) at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:61) at org.kohsuke.stapler.AbstractTearOff.loadScript(AbstractTearOff.java:83) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:30) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:27) at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:355) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:184) - locked <7f7164090> (a com.google.common.collect.MapMakerInternalMap$StrongEntry) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:153) at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69) at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393) at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:61) at org.kohsuke.stapler.AbstractTearOff.loadScript(AbstractTearOff.java:83) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:30) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:27) at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:355) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:184) - locked <7f7161330> (a com.google.common.collect.MapMakerInternalMap$StrongEntry) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:153) at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69) at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393) at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:61) at org.kohsuke.stapler.jelly.groovy.GroovyFacet$1.dispatch(GroovyFacet.java:70) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488) at org.kohsuke.stapler.Stapler.service(Stapler.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#3]" daemon prio=5 tid=7fc21eaa8800 nid=0x1168fb000 waiting on condition [1168fa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#2]" daemon prio=5 tid=7fc219ad4800 nid=0x1167f8000 waiting on condition [1167f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "NioSocketAcceptor-1" prio=5 tid=7fc21db83800 nid=0x11611a000 runnable [116119000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <7f5cf7078> (a sun.nio.ch.Util$2) - locked <7f5cf7060> (a java.util.Collections$UnmodifiableSet) - locked <7f5d0da10> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:285) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:401) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#1]" daemon prio=5 tid=7fc21e455000 nid=0x1166f5000 waiting on condition [1166f4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f4683380> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:680) "ViewJob reload thread" daemon prio=5 tid=7fc21d814800 nid=0x1165f2000 in Object.wait() [1165f1000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f58347c0> (a java.util.LinkedHashSet) at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:171) - locked <7f58347c0> (a java.util.LinkedHashSet) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:188) "Jenkins cron thread" prio=5 tid=7fc21c000800 nid=0x113628000 waiting on condition [113627000] java.lang.Thread.State: RUNNABLE at java.text.RBTableBuilder.build(RBTableBuilder.java:104) at java.text.RBCollationTables.<init>(RBCollationTables.java:65) at java.text.RuleBasedCollator.<init>(RuleBasedCollator.java:276) at java.text.Collator.getInstance(Collator.java:251) - locked <7fde1e278> (a java.lang.Class for java.text.Collator) at java.text.Collator.getInstance(Collator.java:207) - locked <7fde1e278> (a java.lang.Class for java.text.Collator) at jenkins.model.Jenkins$9.<init>(Jenkins.java:1489) at jenkins.model.Jenkins.getComputers(Jenkins.java:1488) at hudson.plugins.slave_squatter.Maintenance._run(Maintenance.java:80) at hudson.plugins.slave_squatter.Maintenance.doRun(Maintenance.java:57) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) "DestroyJavaVM" prio=5 tid=7fc219969800 nid=0x108f1c000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "LauncherControlThread[ControlPort=-1]" prio=5 tid=7fc21c29c000 nid=0x1133d1000 waiting on condition [1133d0000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:680) "ConnectorThread:[ajp13-8009]" daemon prio=5 tid=7fc21c29a800 nid=0x1132ce000 runnable [1132cd000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f4769b60> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at winstone.ajp13.Ajp13Listener.run(Ajp13Listener.java:116) at java.lang.Thread.run(Thread.java:680) "ConnectorThread:[http-8080]" daemon prio=5 tid=7fc21c299800 nid=0x1111b5000 runnable [1111b4000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f46d3ac0> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at winstone.HttpListener.run(HttpListener.java:136) at java.lang.Thread.run(Thread.java:680) "WinstoneHostConfigurationMgmt:default" daemon prio=5 tid=7fc21a15c800 nid=0x1130e9000 waiting on condition [1130e8000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:680) "Java2D Disposer" daemon prio=10 tid=7fc21c29f800 nid=0x111790000 in Object.wait() [11178f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f481f150> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f481f150> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at sun.java2d.Disposer.run(Disposer.java:127) at java.lang.Thread.run(Thread.java:680) "AWT-AppKit" daemon prio=5 tid=7fc21b353800 nid=0x7fff7712c960 runnable [00000000] java.lang.Thread.State: RUNNABLE "Poller SunPKCS11-Darwin" daemon prio=1 tid=7fc21a05c800 nid=0x111091000 waiting on condition [111090000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692) at java.lang.Thread.run(Thread.java:680) "Low Memory Detector" daemon prio=5 tid=7fc21c0ec800 nid=0x110698000 runnable [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=9 tid=7fc21c0eb800 nid=0x110595000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=9 tid=7fc21c0eb000 nid=0x110492000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=9 tid=7fc21c0ea000 nid=0x11038f000 runnable [00000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fc21c0e9800 nid=0x11028c000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=7fc21c0d1800 nid=0x10ffc8000 in Object.wait() [10ffc7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f463bd90> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f463bd90> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=7fc21c0d1000 nid=0x10fec5000 in Object.wait() [10fec4000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f463bda8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <7f463bda8> (a java.lang.ref.Reference$Lock) "VM Thread" prio=9 tid=7fc21c0cc000 nid=0x10fdc2000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fc21c002000 nid=0x10c26e000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fc21c002800 nid=0x10c371000 runnable "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fc21c003000 nid=0x10c474000 runnable "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fc21c004000 nid=0x10c577000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=7fc21c07f000 nid=0x10fa3d000 runnable "VM Periodic Task Thread" prio=10 tid=7fc21c0f6000 nid=0x11079b000 waiting on condition "Exception Catcher Thread" prio=10 tid=7fc21c001800 nid=0x109096000 runnable JNI global references: 1738 heap summary: ============ $ jmap -heap 38315 Attaching to process ID 38315, please wait... Debugger attached successfully. Server compiler detected. JVM version is 20.8-b03-424 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 132120576 (126.0MB) NewSize = 21757952 (20.75MB) MaxNewSize = 87228416 (83.1875MB) OldSize = 65404928 (62.375MB) NewRatio = 7 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 85983232 (82.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 19595264 (18.6875MB) used = 17168816 (16.373458862304688MB) free = 2426448 (2.3140411376953125MB) 87.61717116952342% used Eden Space: capacity = 17432576 (16.625MB) used = 17168816 (16.373458862304688MB) free = 263760 (0.2515411376953125MB) 98.48697060032895% used From Space: capacity = 2162688 (2.0625MB) used = 0 (0.0MB) free = 2162688 (2.0625MB) 0.0% used To Space: capacity = 2162688 (2.0625MB) used = 0 (0.0MB) free = 2162688 (2.0625MB) 0.0% used concurrent mark-sweep generation: capacity = 83034112 (79.1875MB) used = 46811120 (44.64256286621094MB) free = 36222992 (34.54493713378906MB) 56.37576999679361% used Perm Generation: capacity = 85983232 (82.0MB) used = 85981344 (81.99819946289062MB) free = 1888 (0.001800537109375MB) 99.99780422303735% used I also have the binary heap.hprof file, but I am not comfortable with posting that information or otherwise making it publicly available. If you can tell me what I should be looking for and what tools I should be using, I'll be glad to try to provide more information as I can. -- Brad Knowles <[email protected]> LinkedIn Profile: <http://tinyurl.com/y8kpxu>
