[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue

[sam.mxra...@gmail.com (JIRA)]

Title: Message Title

Sam Gleske created an issue   Jenkins / JENKINS-56986 Deadlock on EC2 resources and build queue Issue Type: Bug Assignee: FABRIZIO MANFREDI Components: ec2-plugin Created: 2019-04-11 23:21 Environment: Jenkins version 2.150.2 ace-editor 1.1 ansicolor 0.6.2 antisamy-markup-formatter 1.5 apache-httpcomponents-client-4-api 4.5.5-3.0 authentication-tokens 1.3 aws-credentials 1.24 aws-java-sdk 1.11.457 badge 1.7 basic-branch-build-strategies 1.1.1 blueocean 1.10.2 blueocean-autofavorite 1.2.3 blueocean-bitbucket-pipeline 1.10.2 blueocean-commons 1.10.2 blueocean-config 1.10.2 blueocean-core-js 1.10.2 blueocean-dashboard 1.10.2 blueocean-display-url 2.2.0 blueocean-events 1.10.2 blueocean-git-pipeline 1.10.2 blueocean-github-pipeline 1.10.2 blueocean-i18n 1.10.2 blueocean-jira 1.10.2 blueocean-jwt 1.10.2 blueocean-personalization 1.10.2 blueocean-pipeline-api-impl 1.10.2 blueocean-pipeline-editor 1.10.2 blueocean-pipeline-scm-api 1.10.2 blueocean-rest 1.10.2 blueocean-rest-impl 1.10.2 blueocean-web 1.10.2 bouncycastle-api 2.17 branch-api 2.1.2 cloudbees-bitbucket-branch-source 2.4.1 cloudbees-folder 6.7 cobertura 1.13 code-coverage-api 1.0.7 command-launcher 1.3 config-file-provider 3.5 credentials 2.1.18 credentials-binding 1.17 dashboard-view 2.10 display-url-api 2.3.0 docker-commons 1.13 docker-workflow 1.17 durable-task 1.29 ec2 1.42-SNAPSHOT (private-2daf3555-samgleske) email-ext 2.63 embeddable-build-status 1.9 favorite 2.3.2 git 3.9.3 git-client 2.7.6 git-server 1.7 github 1.29.3 github-api 1.95 github-autostatus 3.2 github-branch-source 2.4.2 github-oauth 0.31 groovy 2.1 groovy-postbuild 2.4.3 handlebars 1.1.1 handy-uri-templates-2-api 2.1.6-1.0 htmlpublisher 1.18 jackson2-api 2.9.8 javadoc 1.4 jdk-tool 1.2 jenkins-design-language 1.10.2 jira 3.0.5 jira-steps 1.4.4 job-dsl 1.71 job-restrictions 0.8 jquery-detached 1.2.1 jsch 0.1.55 junit 1.27 lockable-resources 2.4 mailer 1.23 mask-passwords 2.12.0 matrix-auth 2.3 matrix-project 1.13 maven-plugin 3.2 mercurial 2.5 momentjs 1.1.1 node-iterator-api 1.5.0 pipeline-build-step 2.7 pipeline-github 2.1 pipeline-githubnotify-step 1.0.4 pipeline-graph-analysis 1.9 pipeline-input-step 2.9 pipeline-milestone-step 1.3.1 pipeline-model-api 1.3.4.1 pipeline-model-declarative-agent 1.1.1 pipeline-model-definition 1.3.4.1 pipeline-model-extensions 1.3.4.1 pipeline-multibranch-defaults 2.0 pipeline-rest-api 2.10 pipeline-stage-step 2.3 pipeline-stage-tags-metadata 1.3.4.1 pipeline-stage-view 2.10 plain-credentials 1.5 pubsub-light 1.12 release-archive 0.1.4 rich-text-publisher-plugin 1.4 scm-api 2.3.0 scm-filter-branch-pr 0.4 script-security 1.51 slack 2.16 sonar 2.8.1 sse-gateway 1.17 ssh-agent 1.17 ssh-credentials 1.14 ssh-slaves 1.29.4 structs 1.17 token-macro 2.6 trilead-api 1.0.1 variant 1.1 view-job-filters 2.1.1 windows-slaves 1.4 workflow-aggregator 2.6 workflow-api 2.33 workflow-basic-steps 2.14 workflow-cps 2.63 workflow-cps-global-lib 2.13 workflow-durable-task-step 2.29 workflow-job 2.31 workflow-multibranch 2.20 workflow-scm-step 2.7 workflow-step-api 2.19 workflow-support 3.2 Labels: deadlock Priority: Critical Reporter: Sam Gleske This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same. Environment I'm using a slightly modified version of the EC2 plugin specifically https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal-1 ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released but after https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 was merged. So my custom version includes the deadlock fix for JENKINS-53858. Deadlock behavior During the deadlock the web UI was responsive. The deadlock blocked: New items being queued (such as build events submitted through webhooks). Autoscaling provisioning of new EC2 agents was blocked. I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked. There was a "jenkins.util.Timer 8" thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. "jenkins.util.Timer 8" thread was blocked by "jenkins.util.Timer 5" thread "jenkins.util.Timer 5" thread was blocked by "jenkins.util.Timer 5" My hypothesis I believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other. jenkins.util.Timer [#8] Thread Dump jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run()V (ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:511) at java.util.concurrent.FutureTask.runAndReset()Z (FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)Z (ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624) at java.lang.Thread.run()V (Thread.java:748) jenkins.util.Timer [#5] Thread Dump jenkins.util.Timer [#5] at sun.misc.Unsafe.park(ZJ)V (Native Method) at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()Z (AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;I)Z (AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(I)V (AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()V (ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock()V (ReentrantLock.java:285) at hudson.model.Queue._withLock(Ljava/util/concurrent/Callable;)Ljava/lang/Object; (Queue.java:1438) at hudson.model.Queue.withLock(Ljava/util/concurrent/Callable;)Ljava/lang/Object; (Queue.java:1301) at jenkins.model.Nodes.updateNode(Lhudson/model/Node;)Z (Nodes.java:193) at jenkins.model.Jenkins.updateNode(Lhudson/model/Node;)Z (Jenkins.java:2095) at hudson.model.Node.save()V (Node.java:140) at hudson.util.PersistedList.onModified()V (PersistedList.java:173) at hudson.util.PersistedList.replaceBy(Ljava/util/Collection;)V (PersistedList.java:85) at hudson.model.Slave.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILhudson/model/Node$Mode;Ljava/lang/String;Lhudson/slaves/ComputerLauncher;Lhudson/slaves/RetentionStrategy;Ljava/util/List;)V (Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILhudson/model/Node$Mode;Ljava/lang/String;Lhudson/slaves/ComputerLauncher;Lhudson/slaves/RetentionStrategy;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/util/List;Ljava/lang/String;ZZILhudson/plugins/ec2/AMITypeData;)V (EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILjava/lang/String;Lhudson/model/Node$Mode;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;ZZILhudson/plugins/ec2/AMITypeData;)V (EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILjava/lang/String;Lhudson/model/Node$Mode;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;ZZILhudson/plugins/ec2/AMITypeData;)V (EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(Lcom/amazonaws/services/ec2/model/Instance;)Lhudson/plugins/ec2/EC2OndemandSlave; (SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(Ljava/util/List;)Ljava/util/List; (SlaveTemplate.java:660) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(ILjava/util/EnumSet;)Ljava/util/List; (SlaveTemplate.java:632) at hudson.plugins.ec2.SlaveTemplate.provision(ILjava/util/EnumSet;)Ljava/util/List; (SlaveTemplate.java:463) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(Lhudson/plugins/ec2/SlaveTemplate;IZ)Ljava/util/List; (EC2Cloud.java:587) at hudson.plugins.ec2.EC2Cloud.provision(Lhudson/model/Label;I)Ljava/util/Collection; (EC2Cloud.java:602) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(Lhudson/slaves/NodeProvisioner$StrategyState;)Lhudson/slaves/NodeProvisioner$StrategyDecision; (NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update()V (NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(Lhudson/slaves/NodeProvisioner;)V (NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun()V (NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run()V (ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:511) at java.util.concurrent.FutureTask.runAndReset()Z (FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)Z (ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624) at java.lang.Thread.run()V (Thread.java:748) Add Comment   This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.