I dumped the thread dump into one of the thread dump analysis tool and the tool is reporting on Dead Lock.
Thread CpsStepContext.isReady [#4] is in deadlock with thread Finalizing set up Thanks M.Madhu On Mon, Mar 30, 2020 at 12:45 PM M.Madhusudana Reddy <[email protected]> wrote: > Ok. I am able to reproduce the issue even with HotSpot JAVA of versino 1.8 > update 121 > > java version "1.8.0_121" > Java(TM) SE Runtime Environment (build 1.8.0_121-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) > > ###################### > > Thread CpsStepContext.isReady [#4] is in deadlock with thread Finalizing > set up > > CpsStepContext.isReady [#4]priority:5 - threadId:0x00007f843c075000 - > nativeId:0x4d0f - nativeId (decimal):19727 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at hudson.ExtensionList.ensureLoaded(ExtensionList.java:315) > - waiting to lock <0x00000006c191ac60> (a hudson.ExtensionList$Lock) > at hudson.ExtensionList.iterator(ExtensionList.java:172) > at org.jenkinsci.plugins.workflow.log.LogStorage.of(LogStorage.java:163) > at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener( > WorkflowRun.java:229) > at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300( > WorkflowRun.java:137) > at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener( > WorkflowRun.java:974) > at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess( > CpsFlowExecution.java:789) > at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess( > CpsFlowExecution.java:776) > at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run( > Futures.java:150) > at > com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute( > MoreExecutors.java:253) > at > com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute( > ExecutionList.java:149) > at com.google.common.util.concurrent.ExecutionList.add( > ExecutionList.java:105) > at com.google.common.util.concurrent.AbstractFuture.addListener( > AbstractFuture.java:155) > at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback( > Futures.java:160) > at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback( > Futures.java:90) > at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync( > CpsFlowExecution.java:773) > at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad( > CpsFlowExecution.java:740) > at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution( > WorkflowRun.java:672) > - locked <0x00000006c73dcd98> (a > org.jenkinsci.plugins.workflow.job.WorkflowRun) > at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad( > WorkflowRun.java:533) > - locked <0x00000006c73dd740> (a java.lang.Object) > at hudson.model.RunMap.retrieve(RunMap.java:225) > at hudson.model.RunMap.retrieve(RunMap.java:57) > at jenkins.model.lazy.AbstractLazyLoadRunMap.load( > AbstractLazyLoadRunMap.java:501) > at jenkins.model.lazy.AbstractLazyLoadRunMap.load( > AbstractLazyLoadRunMap.java:483) > at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber( > AbstractLazyLoadRunMap.java:381) > - locked <0x00000006c639f178> (a hudson.model.RunMap) > at hudson.model.RunMap.getById(RunMap.java:205) > at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run( > WorkflowRun.java:921) > at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get( > WorkflowRun.java:932) > at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution( > CpsStepContext.java:217) > at > org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously( > CpsStepContext.java:242) > at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000( > CpsStepContext.java:97) > at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call( > CpsStepContext.java:263) > at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call( > CpsStepContext.java:261) > at jenkins.util.ContextResettingExecutorService$2.call( > ContextResettingExecutorService.java:46) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Locked ownable synchronizers: > - <0x00000006c73dbb00> (a java.util.concurrent.ThreadPoolExecutor$Worker) > > Finalizing set uppriority:5 - threadId:0x00007f845c00d800 - > nativeId:0x4ca8 - nativeId (decimal):19624 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber( > AbstractLazyLoadRunMap.java:371) > - waiting to lock <0x00000006c639f178> (a hudson.model.RunMap) > at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber( > LazyBuildMixIn.java:228) > at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber( > WorkflowJob.java:232) > at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber( > WorkflowJob.java:103) > at hudson.model.Run.fromExternalizableId(Run.java:2446) > at > org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.runForDisplay( > ExecutorStepExecution.java:445) > at > org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getDisplayName( > ExecutorStepExecution.java:461) > at org.jenkinsci.plugins.jqsmonitoring.buildqueue.QueueJob.<init>( > QueueJob.java:50) > at > org.jenkinsci.plugins.jqsmonitoring.buildqueue.BuildQueue.retrieveQueueJobs( > BuildQueue.java:87) > at org.jenkinsci.plugins.jqsmonitoring.buildqueue.BuildQueue.<init>( > BuildQueue.java:36) > at org.jenkinsci.plugins.jqsmonitoring.jqscore.JQSMonitoring.<init>( > JQSMonitoring.java:43) > at > org.jenkinsci.plugins.jqsmonitoring.jqscore.JQSMonitoring$$FastClassByGuice$$a25d5b0d.newInstance( > <generated>) > at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance( > FastConstructor.java:40) > at > com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance( > DefaultConstructionProxyFactory.java:61) > at com.google.inject.internal.ConstructorInjector.provision( > ConstructorInjector.java:105) > at com.google.inject.internal.ConstructorInjector.access$000( > ConstructorInjector.java:32) > at com.google.inject.internal.ConstructorInjector$1.call( > ConstructorInjector.java:89) > at > com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision( > ProvisionListenerStackCallback.java:115) > at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision( > ExtensionFinder.java:567) > at > com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision( > ProvisionListenerStackCallback.java:126) > at com.google.inject.internal.ProvisionListenerStackCallback.provision( > ProvisionListenerStackCallback.java:68) > at com.google.inject.internal.ConstructorInjector.construct( > ConstructorInjector.java:87) > at com.google.inject.internal.ConstructorBindingImpl$Factory.get( > ConstructorBindingImpl.java:267) > at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call( > ProviderToInternalFactoryAdapter.java:46) > at com.google.inject.internal.InjectorImpl.callInContext( > InjectorImpl.java:1103) > at com.google.inject.internal.ProviderToInternalFactoryAdapter.get( > ProviderToInternalFactoryAdapter.java:40) > at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145 > ) > at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get( > ExtensionFinder.java:439) > at com.google.inject.internal.InternalFactoryToProviderAdapter.get( > InternalFactoryToProviderAdapter.java:41) > at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016 > ) > at com.google.inject.internal.InjectorImpl.callInContext( > InjectorImpl.java:1092) > at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) > at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) > at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) > at hudson.ClassicPluginStrategy.findComponents( > ClassicPluginStrategy.java:344) > at hudson.ExtensionList.load(ExtensionList.java:381) > at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317) > - locked <0x00000006c191ac60> (a hudson.ExtensionList$Lock) > at hudson.ExtensionList.iterator(ExtensionList.java:172) > at jenkins.model.Jenkins$16.run(Jenkins.java:3285) > at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run( > TaskGraphBuilder.java:169) > at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) > at jenkins.model.Jenkins$5.runTask(Jenkins.java:1122) > at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) > at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Locked ownable synchronizers: > - <0x00000006c1256288> (a java.util.concurrent.ThreadPoolExecutor$Worker) > - <0x00000006c26f4b30> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > ###################################### > > I am also attaching the thread dumps that are captured. > > Thanks inadvance for your help. > > Thanks > M.Madhu > > > On Mon, Mar 16, 2020 at 3:49 AM Martijn Verburg <[email protected]> > wrote: > >> Please note that AdoptOpenJDK produces both OpenJDK builds (i.e. HotSpot >> JVM) and OpenJ9 + OpenJDK class library builds. >> >> Cheers, >> Martijn >> >> >> On Thu, 12 Mar 2020 at 17:05, M.Madhusudana Reddy <[email protected]> >> wrote: >> >>> Thanks for the update Mark. >>> >>> I believe we tried with AdoptOpenJDK and ran into same issue but I will >>> confirm that and let you know. >>> >>> Thanks >>> M.Madhu >>> >>> >>> On Thu, Mar 12, 2020 at 9:15 AM Mark Waite <[email protected]> >>> wrote: >>> >>>> FYI IBM openJ9 is not supported by the Jenkins project. Initial >>>> testing has found no showstoppers, but IBM OpenJ9 introduces a very >>>> significant difference between what you're running and what is used and >>>> tested by people in the Jenkins project. >>>> >>>> The Jenkins Platform SIG <https://jenkins.io/sigs/platform/> has >>>> started the process of evaluating OpenJ9 with the help of Jim Crowley of >>>> IBM. There are at least 4 open issues >>>> <https://issues.jenkins-ci.org/issues/?jql=text%20~%20%22OpenJ9%22%20and%20status%20%3D%20Open> >>>> related to OpenJ9 and another 15 issues >>>> <https://issues.jenkins-ci.org/issues/?jql=text%20~%20%22J9%22%20%20and%20status%20%3D%20Open> >>>> related to J9. Probably best to confirm that the same problem occurs with >>>> AdoptOpenJDK or with OpenJDK. >>>> >>>> Mark Waite >>>> >>>> On Wednesday, March 11, 2020 at 3:03:40 PM UTC-6, Madhu Muchukota wrote: >>>>> >>>>> Hi Devin, please find the thread dump attached. >>>>> >>>>> jfyi, we are using IBM J9. >>>>> >>>>> Thanks >>>>> M.Madhu >>>>> >>>>> >>>>> On Wed, Mar 11, 2020 at 2:05 PM Devin Nusbaum wrote: >>>>> >>>>>> Do you have the full thread dump from a tool like jstack? If so, >>>>>> please attach it to this thread, or paste it somewhere like >>>>>> gist.github.com or pastebin.com and then paste a link to it here. >>>>>> >>>>>> On Mar 11, 2020, at 14:11, M.Madhusudana Reddy wrote: >>>>>> >>>>>> Hi Team, >>>>>> >>>>>> We upgraded our jenkins instance from 2.107.3 to 2.204.3 and >>>>>> sometimes we notice that jenkins master is taking lot of time to come up >>>>>> and jenkins UI/application is unresponsive. >>>>>> >>>>>> I took a thread dump while we ran into this issue and noticed that >>>>>> application is suffering from deadlock. >>>>>> >>>>>> Thread jenkins.util.Timer [#1] is in deadlock with thread >>>>>> CpsStepContext.isReady >>>>>> [#6] >>>>>> >>>>>> jenkins.util.Timer [#1]priority:0x5 - threadId:0x00007F9C4C16EBE0 - >>>>>> nativeId:0x5AF5 - nativeId (decimal):23285 - state:BLOCKED >>>>>> stackTrace: >>>>>> at jenkins/model/lazy/AbstractLazyLoadRunMap.getByNumber( >>>>>> AbstractLazyLoadRunMap.java:370) >>>>>> at jenkins/model/lazy/LazyBuildMixIn.getBuildByNumber( >>>>>> LazyBuildMixIn.java:228) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowJob.getBuildByNumber( >>>>>> WorkflowJob.java:232) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowJob.getBuildByNumber( >>>>>> WorkflowJob.java:103) >>>>>> at hudson/model/Run.fromExternalizableId(Run.java:2446) >>>>>> at >>>>>> org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution$PlaceholderTask.runForDisplay( >>>>>> ExecutorStepExecution.java:445) >>>>>> at >>>>>> org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution$PlaceholderTask.getDisplayName( >>>>>> ExecutorStepExecution.java:461) >>>>>> at org/jenkinsci/plugins/jqsmonitoring/buildqueue/QueueJob.<init>( >>>>>> QueueJob.java:50) >>>>>> at >>>>>> org/jenkinsci/plugins/jqsmonitoring/buildqueue/BuildQueue.retrieveQueueJobs( >>>>>> BuildQueue.java:87) >>>>>> at org/jenkinsci/plugins/jqsmonitoring/buildqueue/BuildQueue.<init>( >>>>>> BuildQueue.java:36) >>>>>> at org/jenkinsci/plugins/jqsmonitoring/jqscore/JQSMonitoring.<init>( >>>>>> JQSMonitoring.java:43) >>>>>> at >>>>>> org/jenkinsci/plugins/jqsmonitoring/jqscore/JQSMonitoring$$FastClassByGuice$$a25d5b0d.newInstance( >>>>>> <generated>) >>>>>> at >>>>>> com/google/inject/internal/cglib/reflect/$FastConstructor.newInstance( >>>>>> FastConstructor.java:40) >>>>>> at >>>>>> com/google/inject/internal/DefaultConstructionProxyFactory$1.newInstance( >>>>>> DefaultConstructionProxyFactory.java:61) >>>>>> at com/google/inject/internal/ConstructorInjector.provision( >>>>>> ConstructorInjector.java:105) >>>>>> at com/google/inject/internal/ConstructorInjector.access$000( >>>>>> ConstructorInjector.java:32) >>>>>> at com/google/inject/internal/ConstructorInjector$1.call( >>>>>> ConstructorInjector.java:89) >>>>>> at >>>>>> com/google/inject/internal/ProvisionListenerStackCallback$Provision.provision( >>>>>> ProvisionListenerStackCallback.java:115) >>>>>> at >>>>>> hudson/ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:567(Compiled >>>>>> Code)) >>>>>> at >>>>>> com/google/inject/internal/ProvisionListenerStackCallback$Provision.provision( >>>>>> ProvisionListenerStackCallback.java:126) >>>>>> at >>>>>> com/google/inject/internal/ProvisionListenerStackCallback.provision( >>>>>> Provision >>>>>> >>>>>> ####################### >>>>>> >>>>>> >>>>>> CpsStepContext.isReady [#6]priority:0x5 - >>>>>> threadId:0x00007F9C6802DE40 - nativeId:0x5B72 - nativeId (decimal):23410 >>>>>> - >>>>>> state:BLOCKED >>>>>> stackTrace: >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsGroovyShell.doParse( >>>>>> CpsGroovyShell.java:133) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsGroovyShell.reparse( >>>>>> CpsGroovyShell.java:127) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.parseScript( >>>>>> CpsFlowExecution.java:561) >>>>>> at >>>>>> org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.loadProgramAsync( >>>>>> CpsFlowExecution.java:770) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.onLoad( >>>>>> CpsFlowExecution.java:740) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowRun.getExecution( >>>>>> WorkflowRun.java:672) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowRun.onLoad( >>>>>> WorkflowRun.java:533) >>>>>> at hudson/model/RunMap.retrieve(RunMap.java:225) >>>>>> at hudson/model/RunMap.retrieve(RunMap.java:57) >>>>>> at jenkins/model/lazy/AbstractLazyLoadRunMap.load( >>>>>> AbstractLazyLoadRunMap.java:501) >>>>>> at jenkins/model/lazy/AbstractLazyLoadRunMap.load( >>>>>> AbstractLazyLoadRunMap.java:483) >>>>>> at jenkins/model/lazy/AbstractLazyLoadRunMap.getByNumber( >>>>>> AbstractLazyLoadRunMap.java:381) >>>>>> at hudson/model/RunMap.getById(RunMap.java:205) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowRun$Owner.run( >>>>>> WorkflowRun.java:921) >>>>>> at org/jenkinsci/plugins/workflow/job/WorkflowRun$Owner.get( >>>>>> WorkflowRun.java:932) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsStepContext.getExecution( >>>>>> CpsStepContext.java:217) >>>>>> at >>>>>> org/jenkinsci/plugins/workflow/cps/CpsStepContext.getThreadGroupSynchronously( >>>>>> CpsStepContext.java:242) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsStepContext.access$000( >>>>>> CpsStepContext.java:97) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsStepContext$1.call( >>>>>> CpsStepContext.java:263) >>>>>> at org/jenkinsci/plugins/workflow/cps/CpsStepContext$1.call( >>>>>> CpsStepContext.java:261) >>>>>> at jenkins/util/ContextResettingExecutorService$2.call( >>>>>> ContextResettingExecutorService.java:46) >>>>>> at java/util/concurrent/FutureTask.run(FutureTask.java:277) >>>>>> at >>>>>> java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153(Compiled >>>>>> Code)) >>>>>> at java/util/concurrent/ThreadPoolExecutor$Worker.run( >>>>>> ThreadPoolExecutor.java:628) >>>>>> at java/lang/Thread.run(Thread.java:785) >>>>>> >>>>>> ############################# >>>>>> >>>>>> Any one ran into the same issue and any help on this is highly >>>>>> appreciated please. >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/67489814-6ddf-4d73-915b-0d37004f9124%40googlegroups.com >>>> <https://groups.google.com/d/msgid/jenkinsci-users/67489814-6ddf-4d73-915b-0d37004f9124%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/CAN%2BS7NoQLe%3DY%3Dhzzp9w-gHWjujnCNZf1JqgeP%3Dr4yReTo4AMSA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/jenkinsci-users/CAN%2BS7NoQLe%3DY%3Dhzzp9w-gHWjujnCNZf1JqgeP%3Dr4yReTo4AMSA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAP7YuATjeZeZjMVs5HeeijxRJYxmK6BidGDuCPXVosJ3MrcWJA%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CAP7YuATjeZeZjMVs5HeeijxRJYxmK6BidGDuCPXVosJ3MrcWJA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAN%2BS7NoMnSgF%3DghQeXDvk8%3D%2BYX5ZwYSR1UtDTT_rm9kk40YS7A%40mail.gmail.com.
