Wow, can't believe I didn't know about /threadDump before, thanks!
Anyways I mostly see lots (on the order of 100s) of:
"Computer.threadPoolForRemoting [#85811]" Id=380625 Group=main
TIMED_WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@687826ae
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@687826ae
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I did see some (~20) CPS executors that looked suspicious (they're also
hanging forever when i go to the job):
"Running CpsFlowExecution[Owner[workflow_test/619:workflow_test#619]]"
Id=7310 Group=main RUNNABLE
at
org.kohsuke.stapler.framework.io.LargeText$BufferSession.skip(LargeText.java:532)
at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:211)
at
hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:162)
at
org.jenkinsci.plugins.workflow.job.WorkflowRun.copyLogs(WorkflowRun.java:359)
at
org.jenkinsci.plugins.workflow.job.WorkflowRun.access$600(WorkflowRun.java:107)
at
org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:752)
- locked java.util.concurrent.atomic.AtomicBoolean@7eb2df49
at
org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:799)
at
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$4.run(CpsThreadGroup.java:320)
at
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:32)
at
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@d3b1e0f
I noticed them before but I assumed they weren't related to this, issue, I
think they're waiting for slaves nodes that have been removed from our
jenkins (not just disconnected but deleted), I'll clean these up with the
/kill url on workflows and see if that improves things since the threaddump
makes me wonder if these are related.
Thanks again
On Thursday, January 14, 2016 at 4:14:16 PM UTC-5, Christopher Orr wrote:
>
> On 14/01/16 21:30, Kevin wrote:
> > Hi all, I've got some custom cloud providers that don't seem to get
> > triggered, I've traced it back to NodeProvisionerInvoker not being
> > called (even after 30 minutes of waiting), I can call the
> > nodeProvisioners by hand (using the update function) in the same way
> > that the NodeProvisionerInvoker does in the groovy script console and
> > things seem to work. Actually it seems like a number of bits related to
> > PeriodicWork classes aren't working since I assume git polling and the
> > periodic build setting are a PeriodicWork things, but they're also not
> > triggering.
> >
> > I did also look at PeriodicWork.all().collect {it.class.name}
> > and println jenkins.util.Timer.get() and got:
> >
> > hudson.diagnosis.HudsonHomeDiskUsageChecker,
> > hudson.diagnosis.MemoryUsageMonitor,
> > hudson.model.FingerprintCleanupThread,
> > hudson.model.LoadStatistics$LoadStatisticsUpdater,
> > hudson.model.WorkspaceCleanupThread,
> > hudson.slaves.ComputerRetentionWork,
> > hudson.slaves.ConnectionActivityMonitor,
> > hudson.slaves.NodeProvisioner$NodeProvisionerInvoker,
> > hudson.triggers.Trigger$Cron,
> > jenkins.model.DownloadSettings$DailyCheck,
> > org.jenkinsci.plugins.periodicreincarnation.PeriodicReincarnation]
> >
> >
> > and
> >
> >
> jenkins.util.ErrorLoggingScheduledThreadPoolExecutor@6d72bb0e[Running,
> > pool size = 10, active threads = 10, queued tasks = 2521, completed
> > tasks = 1134830]
>
> If the number of active threads remains equal to the pool size, then I
> would guess that some tasks are getting stuck.
>
> Can you see a bunch of timer-related or otherwise suspicious-looking
> tasks running if you go to /threadDump on your Jenkins instance?
>
> Regards,
> Chris
>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/2b6e0c8f-1ad7-4a48-9b3f-c2f78d05a8ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
