[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title
Sam Gleske updated an issue
Jenkins / JENKINS-56986
Deadlock on EC2 resources and build queue
Change By:
Sam Gleske
This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same.h2. EnvironmentI'm using a slightly modified version of the EC2 plugin specifically https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal-2 ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released - but after https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 was merged. So my custom version includes the deadlock fix for JENKINS-53858. - (CORRECTION my version did not include the fix from JENKINS-53858) h2. Deadlock behaviorDuring the deadlock the web UI was responsive. The deadlock blocked:* New items being queued (such as build events submitted through webhooks).* Autoscaling provisioning of new EC2 agents was blocked.* I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked.There was a "jenkins.util.Timer \[#8\]" thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. "jenkins.util.Timer \[#8\]" thread was blocked by "jenkins.util.Timer \[#5\]" thread"jenkins.util.Timer \[#5\]" thread was blocked by waiting on "jenkins.util.Timer \[#8\]" and visa versa.h2. My hypothesisI believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other.h2. jenkins.util.Timer \[#8\] Thread Dump{noformat}jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run()V
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title Sam Gleske closed an issue as Duplicate I misspoke. My compiled plugin does not include the deadlock fix. Jenkins / JENKINS-56986 Deadlock on EC2 resources and build queue Change By: Sam Gleske Status: Open Closed Resolution: Duplicate Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title
Sam Gleske updated an issue
Jenkins / JENKINS-56986
Deadlock on EC2 resources and build queue
Change By:
Sam Gleske
This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same. h2. EnvironmentI'm using a slightly modified version of the EC2 plugin specifically [ https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal- 1] 2 ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released but after [ https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 ] was merged. So my custom version includes the deadlock fix for JENKINS-53858. h2. Deadlock behaviorDuring the deadlock the web UI was responsive. The deadlock blocked: * New items being queued (such as build events submitted through webhooks and other kinds of triggers ). * Autoscaling provisioning of new EC2 agents was blocked. * I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked.There was a * "jenkins.util.Timer \ [#8 \ ]" * thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. * "jenkins.util.Timer \ [#8 \ ]" * thread was blocked by * "jenkins.util.Timer \ [#5 \ ]" * thread * "jenkins.util.Timer \ [#5 \ ]" * thread was blocked by waiting on * "jenkins.util.Timer \ [#8 \ ]" * and visa versa. h2. My hypothesisI believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other. h2. jenkins.util.Timer \ [#8 \ ] Thread Dump {noformat}jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title
Sam Gleske updated an issue
Jenkins / JENKINS-56986
Deadlock on EC2 resources and build queue
Change By:
Sam Gleske
This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same. h2. EnvironmentI'm using a slightly modified version of the EC2 plugin specifically [ https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal-1 ] ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released but after [ https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 ] was merged. So my custom version includes the deadlock fix for JENKINS-53858. h2. Deadlock behaviorDuring the deadlock the web UI was responsive. The deadlock blocked: * New items being queued (such as build events submitted through webhooks and other kinds of triggers).* Autoscaling provisioning of new EC2 agents was blocked.* I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked.There was a * "jenkins.util.Timer \ [#8 \ ]" * thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. * "jenkins.util.Timer \ [#8 \ ]" * thread was blocked by * "jenkins.util.Timer \ [#5 \ ]" * thread * "jenkins.util.Timer \ [#5 \ ]" * thread was blocked by waiting on * "jenkins.util.Timer \ [#8 \ ]" * and visa versa. h2. My hypothesisI believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other. h2. jenkins.util.Timer \ [#8 \ ] Thread Dump {noformat}jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title
Sam Gleske updated an issue
Jenkins / JENKINS-56986
Deadlock on EC2 resources and build queue
Change By:
Sam Gleske
This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same.h2. EnvironmentI'm using a slightly modified version of the EC2 plugin specifically https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal-1 ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released but after https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 was merged. So my custom version includes the deadlock fix for JENKINS-53858.h2. Deadlock behaviorDuring the deadlock the web UI was responsive. The deadlock blocked:* New items being queued (such as build events submitted through webhooks and other kinds of triggers ).* Autoscaling provisioning of new EC2 agents was blocked.* I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked.There was a "jenkins.util.Timer \[#8\]" thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. "jenkins.util.Timer \[#8\]" thread was blocked by "jenkins.util.Timer \[#5\]" thread"jenkins.util.Timer \[#5\]" thread was blocked by waiting on "jenkins.util.Timer \[#8\]" and visa versa.h2. My hypothesisI believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other.h2. jenkins.util.Timer \[#8\] Thread Dump{noformat}jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run()V (ImpersonatingScheduledExecutorService.java:58) at
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title
Sam Gleske updated an issue
Jenkins / JENKINS-56986
Deadlock on EC2 resources and build queue
Change By:
Sam Gleske
This is either the same as or is related to JENKINS-53858. Feel free to close this as a duplicate and re-open JENKINS-53858 if it is the same.h2. EnvironmentI'm using a slightly modified version of the EC2 plugin specifically https://github.com/sgleske-ias/ec2-plugin/tree/ias-internal-1 ; it was built from EC2 plugin 1.42-SNAPSHOT before 1.42 was released but after https://github.com/jenkinsci/ec2-plugin/commit/2f3a04a2d3ce0e51a755792b9d03b4fff4ebe9b3 was merged. So my custom version includes the deadlock fix for JENKINS-53858.h2. Deadlock behaviorDuring the deadlock the web UI was responsive. The deadlock blocked:* New items being queued (such as build events submitted through webhooks).* Autoscaling provisioning of new EC2 agents was blocked.* I was not able to delete the 1 EC2 agent that was provisioned but marked as offline because it was deadlocked.There was a "jenkins.util.Timer \ [#8 \ ]" thread in which most actions in my Jenkins instance were blocked. This was for the cloud provisioner. Most items that were blocked work blocked by this thread. "jenkins.util.Timer \ [#8 \ ]" thread was blocked by "jenkins.util.Timer \ [#5 \ ]" thread"jenkins.util.Timer \ [#5 \ ]" thread was blocked by waiting on "jenkins.util.Timer \ [# 5 8\ ]" and visa versa. h2. My hypothesisI believe they were blocked by the combination of the Queue lock and the EC2Cloud lock. Each needed both and was waiting on the other.h2. jenkins.util.Timer \[#8\] Thread Dump{noformat}jenkins.util.Timer [#8] at hudson.plugins.ec2.EC2Cloud.connect()Lcom/amazonaws/services/ec2/AmazonEC2; (EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(Ljava/lang/String;Lhudson/plugins/ec2/EC2Cloud;)Lcom/amazonaws/services/ec2/model/Instance; (CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState()Lhudson/plugins/ec2/InstanceState; (EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/plugins/ec2/EC2Computer;)J (EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(Lhudson/model/Computer;)J (EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run()V (ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Ljava/lang/Runnable;)V (Queue.java:1381) at hudson.model.Queue.withLock(Ljava/lang/Runnable;)V (Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun()V (ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run()V (SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run()V (ImpersonatingScheduledExecutorService.java:58) at
[JIRA] (JENKINS-56986) Deadlock on EC2 resources and build queue
Title: Message Title Sam Gleske created an issue Jenkins / JENKINS-56986 Deadlock on EC2 resources and build queue Issue Type: Bug Assignee: FABRIZIO MANFREDI Components: ec2-plugin Created: 2019-04-11 23:21 Environment: Jenkins version 2.150.2 ace-editor 1.1 ansicolor 0.6.2 antisamy-markup-formatter 1.5 apache-httpcomponents-client-4-api 4.5.5-3.0 authentication-tokens 1.3 aws-credentials 1.24 aws-java-sdk 1.11.457 badge 1.7 basic-branch-build-strategies 1.1.1 blueocean 1.10.2 blueocean-autofavorite 1.2.3 blueocean-bitbucket-pipeline 1.10.2 blueocean-commons 1.10.2 blueocean-config 1.10.2 blueocean-core-js 1.10.2 blueocean-dashboard 1.10.2 blueocean-display-url 2.2.0 blueocean-events 1.10.2 blueocean-git-pipeline 1.10.2 blueocean-github-pipeline 1.10.2 blueocean-i18n 1.10.2 blueocean-jira 1.10.2 blueocean-jwt 1.10.2 blueocean-personalization 1.10.2 blueocean-pipeline-api-impl 1.10.2 blueocean-pipeline-editor 1.10.2 blueocean-pipeline-scm-api 1.10.2 blueocean-rest 1.10.2 blueocean-rest-impl 1.10.2 blueocean-web 1.10.2 bouncycastle-api 2.17 branch-api 2.1.2 cloudbees-bitbucket-branch-source 2.4.1 cloudbees-folder 6.7 cobertura 1.13 code-coverage-api 1.0.7 command-launcher 1.3 config-file-provider 3.5 credentials 2.1.18 credentials-binding 1.17 dashboard-view 2.10 display-url-api 2.3.0 docker-commons 1.13 docker-workflow 1.17 durable-task 1.29 ec2 1.42-SNAPSHOT (private-2daf3555-samgleske) email-ext 2.63 embeddable-build-status 1.9 favorite 2.3.2 git 3.9.3 git-client 2.7.6 git-server 1.7 github 1.29.3 github-api 1.95 github-autostatus 3.2 github-branch-source 2.4.2 github-oauth 0.31 groovy 2.1 groovy-postbuild 2.4.3 handlebars 1.1.1 handy-uri-templates-2-api 2.1.6-1.0 htmlpublisher 1.18 jackson2-api 2.9.8 javadoc 1.4 jdk-tool 1.2 jenkins-design-language 1.10.2 jira 3.0.5 jira-steps 1.4.4 job-dsl 1.71 job-restrictions 0.8 jquery-detached 1.2.1 jsch 0.1.55 junit 1.27 lockable-resources 2.4 mailer 1.23 mask-passwords 2.12.0 matrix-auth 2.3 matrix-project 1.13 maven-plugin 3.2 mercurial 2.5 momentjs 1.1.1 node-iterator-api 1.5.0 pipeline-build-step 2.7 pipeline-github 2.1 pipeline-githubnotify-step 1.0.4 pipeline-graph-analysis 1.9 pipeline-input-step 2.9 pipeline-milestone-step 1.3.1 pipeline-model-api 1.3.4.1 pipeline-model-declarative-agent 1.1.1 pipeline-model-definition 1.3.4.1 pipeline-model-extensions 1.3.4.1 pipeline-multibranch-defaults 2.0 pipeline-rest-api 2.10 pipeline-stage-step 2.3 pipeline-stage-tags-metadata 1.3.4.1 pipeline-stage-view 2.10 plain-credentials 1.5 pubsub-light 1.12
