I should have mentioned that all 10 of the jenkins.util.Timer's are in the
RUNNABLE state and the Jenkins cron thread is in TIMED_WAITING which seems
to make sense (I think).
The queued and completed tasks from jenkins.util.Timer.get() hasn't changed
appreciably
On Friday, January 15, 2016 at 9:00:23 AM UTC-5, Kevin wrote:
>
> Wow, can't believe I didn't know about /threadDump before, thanks!
>
> Anyways I mostly see lots (on the order of 100s) of:
>
> "Computer.threadPoolForRemoting [#85811]" Id=380625 Group=main
> TIMED_WAITING on
> java.util.concurrent.SynchronousQueue$TransferStack@687826ae
> at sun.misc.Unsafe.park(Native Method)
> - waiting on java.util.concurrent.SynchronousQueue$TransferStack@687826ae
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> I did see some (~20) CPS executors that looked suspicious (they're also
> hanging forever when i go to the job):
>
> "Running CpsFlowExecution[Owner[workflow_test/619:workflow_test#619]]"
> Id=7310 Group=main RUNNABLE
> at
> org.kohsuke.stapler.framework.io.LargeText$BufferSession.skip(LargeText.java:532)
> at
> org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:211)
> at
> hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:162)
> at
> org.jenkinsci.plugins.workflow.job.WorkflowRun.copyLogs(WorkflowRun.java:359)
> at
> org.jenkinsci.plugins.workflow.job.WorkflowRun.access$600(WorkflowRun.java:107)
> at
> org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:752)
> - locked java.util.concurrent.atomic.AtomicBoolean@7eb2df49
> at
> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:799)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$4.run(CpsThreadGroup.java:320)
> at
> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:32)
> at
> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
> at
> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@d3b1e0f
>
>
> I noticed them before but I assumed they weren't related to this, issue, I
> think they're waiting for slaves nodes that have been removed from our
> jenkins (not just disconnected but deleted), I'll clean these up with the
> /kill url on workflows and see if that improves things since the threaddump
> makes me wonder if these are related.
>
>
> Thanks again
>
> On Thursday, January 14, 2016 at 4:14:16 PM UTC-5, Christopher Orr wrote:
>>
>> On 14/01/16 21:30, Kevin wrote:
>> > Hi all, I've got some custom cloud providers that don't seem to get
>> > triggered, I've traced it back to NodeProvisionerInvoker not being
>> > called (even after 30 minutes of waiting), I can call the
>> > nodeProvisioners by hand (using the update function) in the same way
>> > that the NodeProvisionerInvoker does in the groovy script console and
>> > things seem to work. Actually it seems like a number of bits related
>> to
>> > PeriodicWork classes aren't working since I assume git polling and the
>> > periodic build setting are a PeriodicWork things, but they're also not
>> > triggering.
>> >
>> > I did also look at PeriodicWork.all().collect {it.class.name}
>> > and println jenkins.util.Timer.get() and got:
>> >
>> > hudson.diagnosis.HudsonHomeDiskUsageChecker,
>> > hudson.diagnosis.MemoryUsageMonitor,
>> > hudson.model.FingerprintCleanupThread,
>> > hudson.model.LoadStatistics$LoadStatisticsUpdater,
>> > hudson.model.WorkspaceCleanupThread,
>> > hudson.slaves.ComputerRetentionWork,
>> > hudson.slaves.ConnectionActivityMonitor,
>> > hudson.slaves.NodeProvisioner$NodeProvisionerInvoker,
>> > hudson.triggers.Trigger$Cron,
>> > jenkins.model.DownloadSettings$DailyCheck,
>> > org.jenkinsci.plugins.periodicreincarnation.PeriodicReincarnation]
>> >
>> >
>> > and
>> >
>> >
>> jenkins.util.ErrorLoggingScheduledThreadPoolExecutor@6d72bb0e[Running,
>> > pool size = 10, active threads = 10, queued tasks = 2521, completed
>> > tasks = 1134830]
>>
>> If the number of active threads remains equal to the pool size, then I
>> would guess that some tasks are getting stuck.
>>
>> Can you see a bunch of timer-related or otherwise suspicious-looking
>> tasks running if you go to /threadDump on your Jenkins instance?
>>
>> Regards,
>> Chris
>>
>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/754475d6-b63e-4e21-9fa5-a211e870129a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
