[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Pierson Yieh edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart We've identified the cause of our issue. The orphan re-attachment logic is tied the EC2Cloud's provision method. But the issue occurs when the actual number of existing AWS nodes has hit an instance cap (i.e. no more nodes can be provisioned). Because we've hit an instance cap, provisioning isn't even attempted and the orphan re-attachment logic isn't triggered. We're submitting Submitted a PR now that addresses this issue here: [https://github . com/jenkinsci/ec2-plugin/pull/448] Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.11297.1586900160314%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Pierson Yieh commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart We've identified the cause of our issue. The orphan re-attachment logic is tied the EC2Cloud's provision method. But the issue occurs when the actual number of existing AWS nodes has hit an instance cap (i.e. no more nodes can be provisioned). Because we've hit an instance cap, provisioning isn't even attempted and the orphan re-attachment logic isn't triggered. We're submitting a PR now that addresses this issue. Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.11183.1586893200240%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Pierson Yieh commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Jakub Bochenski Was the problem that was solved the issue of orphan nodes not getting reconnected or agents dying during launch? Our issue is of orphan nodes not getting re-attached to their respective Jenkins Masters. Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.8361.1586376180257%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I was able to resolve my problem in the same way as described in https://issues.jenkins-ci.org/browse/JENKINS-61370?focusedCommentId=388247&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-388247 The swallowing of useful error output is a big issue that should be improved Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.8061.1586345460372%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Pierson Yieh edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart We've also seen this behavior before though we're not sure how to reproduce the problem. We saw it when we'd hit our max AWS request limit and Jenkins started losing track of nodes and couldn't spin up new ones cause the orphaned nodes were still being counted towards the max instance count, but weren't showing up in the Jenkins UI.I'm able to "simulate" the "losing track of nodes" by running a groovy script on the Jenkins Master to manually remove the node for the Jenkins object. And we're looking into implementing a feature to automatically re-attach these orphaned nodes to Jenkins. Update: seems the SlaveTemplate.checkInstance() finds our orphan nodes and were able to re-attach them to the Jenkins Master. Not sure why in the past they weren't getting re-attached. Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.7895.1586301900338%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Pierson Yieh commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart We've also seen this behavior before though we're not sure how to reproduce the problem. We saw it when we'd hit our max AWS request limit and Jenkins started losing track of nodes and couldn't spin up new ones cause the orphaned nodes were still being counted towards the max instance count, but weren't showing up in the Jenkins UI. I'm able to "simulate" the "losing track of nodes" by running a groovy script on the Jenkins Master to manually remove the node for the Jenkins object. And we're looking into implementing a feature to automatically re-attach these orphaned nodes to Jenkins. Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.7874.1586299500323%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Raihaan Shouhell I have filled a separate issue about the agents dying during launch as it happens independently of this issue. Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.708.1583509020181%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Jakub Bochenski Environment: Jenkins ver. 2.176.1ec2 plugin 1.43, 1.44, 1.45 , 1.49.1 Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6721.1583407080708%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Jakub Bochenski Environment: Jenkins ver. 2.176.1 , 2.204.2 ec2 plugin 1.43, 1.44, 1.45, 1.49.1 Add Comment This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6723.1583407080729%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Do you have any logs from the retention strategy. I see in your logs that the instances were stopped but I'm not certain why. I'm not sure what you mean. I have pasted all of the Jenkins log output here already. Do you want me to enable DEBUG level logging for some components? Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6933.1582715160463%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
{ code quote }Do you have any logs from the retention strategy. I see in your logs that the instances were stopped but I'm not certain why.{ code quote }I'm not sure what you mean. I have pasted all of the Jenkins log output here already. Do you want me to enable DEBUG level logging for some components?
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6939.1582715160542%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Plugin can't terminate all instances on shutdown / restart simply because that can cause shutdown to stall. Do you have any logs from the retention strategy. I see in your logs that the instances were stopped but I'm not certain why. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6657.1582675320301%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I tried this a few more times. So far it's reproducible 100% (which is in a way good) Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6410.1582646100205%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
Actually I've noticed another problematic thing. After I manually terminate the instance the plugin will spawn 4 new instances that will get terminated immediatelly before finally getting the fifth one up. I thought it's was a fluke at first but it seems to be reproducible consistently.Log: https://gist.github.com/jakub-bochenski/c24b1f8e24e7be77aa2522df2c8caaed It seems the plugin just terminates the instance for no reason:{code}Feb 25, 2020 3:18:30 PM INFO hudson.plugins.ec2.EC2Cloud logLaunching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir /opt/jenkinsFeb 25, 2020 3:18:32 PM INFO hudson.plugins.ec2.EC2OndemandSlave terminateTerminated EC2 instance (terminated): i-046afc69c32c1acddFeb 25, 2020 3:18:32 PM INFO hudson.plugins.ec2.EC2OndemandSlave terminateRemoved EC2 instance from jenkins master: i-046afc69c32c1acdd{code}Also notice this, despite instance cap=1{code}Feb 25, 2020 3:18:21 PM INFO hudson.slaves.NodeProvisioner lambda$update$6EC2 (ec2) - ec2 (ami-028d96c69234f9d1a) provisioning successfully completed. We have now 2 computer(s){code}
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6373.1582645140403
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Actually I've noticed another problematic thing. After I manually terminate the instance the plugin will spawn 4 new instances that will get terminated immediatelly before finally getting the fifth one up. I thought it's was a fluke at first but it seems to be reproducible consistently. Log: https://gist.github.com/jakub-bochenski/c24b1f8e24e7be77aa2522df2c8caaed Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6363.1582644900210%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
We have only 1 cloud active, you can find the groovy script to configure it above (I have disabled to ECS cloud to test this).I'm on version 1.49.1 of plugin and Jenkins ver. 2.204.2.I was able to reproduce this with just: - start Jenkins master instance - trigger job that will ask for the EC2-powered agent - wait for job to finish and stop Jenkins - start Jenkins again - trigger job that will ask for the EC2-powered agent, now it won't be able to hook the running instance againThe log output looks like this: {code}Feb 25, 2020 2:55:21 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:21 PM INFO com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory addProxyConfigConfiguring Proxy. Proxy Host: ew1-internal-proxy.services-ci-infra-services.fsapi.com Proxy Port: 8080Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:31 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}{code}After I terminate the instance manually in EC2 Console the log is {code}Feb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Considering launchingFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDeviceAMI had xvdaFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice{DeleteOnTermination: true,SnapshotId: snap-040ff84a8c849bc15,VolumeSize: 8,VolumeType: gp2,Encrypted: false}Feb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.TerminateFeb 25, 2020 3:18:02 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
We have only 1 cloud active, you can find the groovy script to configure it above (I have disabled to ECS cloud to test this).I'm on version 1.49.1 of plugin and Jenkins ver. 2.204.2.I was able to reproduce this with just: - start Jenkins master instance - trigger job that will ask for the EC2-powered agent - wait for job to finish and stop Jenkins - start Jenkins again - trigger job that will ask for the EC2-powered agent, now it won't be able to hook the running instance againThe log output looks like this: {code}Feb 25, 2020 2:55:21 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:21 PM INFO com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory addProxyConfigConfiguring Proxy. Proxy Host: ew1-internal-proxy.services-ci-infra-services.fsapi.com Proxy Port: 8080Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:31 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}{code} After I terminate the instance manually in EC2 Console the log is {code}Feb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Considering launchingFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDeviceAMI had xvdaFeb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice{DeleteOnTermination: true,SnapshotId: snap-040ff84a8c849bc15,VolumeSize: 8,VolumeType: gp2,Encrypted: false}Feb 25, 2020 3:18:01 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.TerminateFeb 25, 2020 3:18:02 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docke
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart BTW I didn't think about it earlier, but shouldn't the plugin actually terminate the EC2 instance on Jenkins shutdown? Otherwise it could stay there indefinatly indefinitely Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6344.1582643820566%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart BTW I didn't think about it earlier, but shouldn't the plugin actually terminate the EC2 instance on shutdown? Otherwise it could stay there indefinatly Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.6342.1582643820476%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
We have only 1 cloud active, you can find the groovy script to configure it above (I have disabled to ECS cloud to test this).I'm on version 1.49.1 of plugin and Jenkins ver. 2.204.2.I was able to reproduce this with just: - start Jenkins master instance - trigger job that will ask for the EC2-powered agent - wait for job to finish and stop Jenkins - start Jenkins again - trigger job that will ask for the EC2-powered agent, now it won't be able to hook the running instance againThe log output looks like this: {code} Feb 25, 2020 2:55:21 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:21 PM INFO com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory addProxyConfigConfiguring Proxy. Proxy Host: ew1-internal-proxy.services-ci-infra-services.fsapi.com Proxy Port: 8080Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}Feb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsFeb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0Feb 25, 2020 2:55:31 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}{code}
Add Comment
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
We have only 1 cloud active, you can find the groovy script to configure it above (I have disabled to ECS cloud to test this). I'm on version 1.49.1 of plugin and Jenkins ver. 2.204.2. I was able to reproduce this with just:
start Jenkins master instance
trigger job that will ask for the EC2-powered agent
wait for job to finish and stop Jenkins
start Jenkins again
trigger job that will ask for the EC2-powered agent, now it won't be able to hook the running instance again
The log output looks like this:
Feb 25, 2020 2:55:21 PM INFO hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 units
Feb 25, 2020 2:55:21 PM INFO com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory addProxyConfig
Configuring Proxy. Proxy Host: ew1-internal-proxy.services-ci-infra-services.fsapi.com Proxy Port: 8080
Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0
Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provision
Can't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}
Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 units
Feb 25, 2020 2:55:22 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0
Feb 25, 2020 2:55:22 PM WARNING hudson.plugins.ec2.EC2Cloud provision
Can't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}
Feb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 units
Feb 25, 2020 2:55:31 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave
SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0
Feb 25, 2020 2:55:31 PM WARNING hudson.plugins.ec2.EC2Cloud provision
Can't raise nodes for SlaveTemplate{ami='ami-028d96c69234f9d1a', labels='docker docker-bakery'}
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Hey Jakub I haven't had any time to look into this, Do you have multiple clouds enabled? It seems weird that your instance can't be found with just the instance id. On top of that the instance is in the RUNNING state as you say Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.5846.1582620060220%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart We are still seeing issues with spawning instances after restart. Do you plan to work on resolving this? FABRIZIO MANFREDIRaihaan Shouhell Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.5353.1582550760205%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
I'm still seeing issues in 1. 46 45 . The instance is in {{running}} state but plugin can't see t be able to connect it. I can't test it on 1.46 because of JENKINS-59564 {code}Sep 30, 2019 9:32:03 AM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Considering launchingSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDeviceAMI had xvdaSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice{DeleteOnTermination: true,SnapshotId: snap-0f2d5ab1c6f918116,VolumeSize: 8,VolumeType: gp2,Encrypted: false,}Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.TerminateSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-02769bd03e603e42f]}, {Name: instance-type,Values: [t3.micro]}, {Name: key-name,Values: [j4a-bochja]}, {Name: subnet-id,Values: [subnet-0a167eb56a247e891]}, {Name: instance.group-id,Values: [sg-0a4f5e5ac5bb602e4]}, {Name: tag:Name,Values: [ew1-j4a-jenkins-slave-ec2]}, {Name: tag:DeploymentName,Values: [ew1-j4a]}, {Name: tag:CostCenter,Values: [31505]}, {Name: tag:DeploymentType,Values: [dev]}, {Name: tag:DeploymentGroup,Values: [ew1-j4a]}, {Name: tag:jenkins_server_url,Values: [https://acme.com/]}, {Name: tag:jenkins_slave_type,Values: [demand_ec2 (ami-02769bd03e603e42f)]}],InstanceIds: [],}Sep 30, 2019 9:32:05 AM INFO hudson.plugins.ec2.CloudHelper getInstanceUnexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead.Sep 30, 2019 9:32:05 AM WARNING hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:566) at hudson.plugins.ec2.EC2AbstractSlave.(EC2AbstractSlave.java:165) at hudson.plugins.ec2.EC2OndemandSlave.(EC2OndemandSlave.java:56) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:1104) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:773) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:745) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:540) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:589) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:615) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:62
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
I'm still seeing issues in 1.46. The instance is in {{running}} state but plugin can't see t be able to connect it.{code}Sep 30, 2019 9:32:03 AM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 unitsSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Considering launchingSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDeviceAMI had xvdaSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice{DeleteOnTermination: true,SnapshotId: snap-0f2d5ab1c6f918116,VolumeSize: 8,VolumeType: gp2,Encrypted: false,}Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.TerminateSep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-02769bd03e603e42f]}, {Name: instance-type,Values: [t3.micro]}, {Name: key-name,Values: [j4a-bochja]}, {Name: subnet-id,Values: [subnet-0a167eb56a247e891]}, {Name: instance.group-id,Values: [sg-0a4f5e5ac5bb602e4]}, {Name: tag:Name,Values: [ew1-j4a-jenkins-slave-ec2]}, {Name: tag:DeploymentName,Values: [ew1-j4a]}, {Name: tag: Contact,Values: [production-cloud-un...@f-secure.com]}, {Name: tag: CostCenter,Values: [31505]}, {Name: tag:DeploymentType,Values: [dev]}, {Name: tag:DeploymentGroup,Values: [ew1-j4a]}, {Name: tag:jenkins_server_url,Values: [https:// mastermaster acme . j4a.infra-dev-services.fsapi. com/]}, {Name: tag:jenkins_slave_type,Values: [demand_ec2 (ami-02769bd03e603e42f)]}],InstanceIds: [],}Sep 30, 2019 9:32:05 AM INFO hudson.plugins.ec2.CloudHelper getInstanceUnexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead.Sep 30, 2019 9:32:05 AM WARNING hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:566) at hudson.plugins.ec2.EC2AbstractSlave.(EC2AbstractSlave.java:165) at hudson.plugins.ec2.EC2OndemandSlave.(EC2OndemandSlave.java:56) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:1104) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:773) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:745) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:540) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:589) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:615) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at huds
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
I'm still seeing issues in 1.46. The instance is in running state but plugin can't see t be able to connect it.
Sep 30, 2019 9:32:03 AM INFO hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 units
Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Considering launching
Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
AMI had xvda
Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
{DeleteOnTermination: true,SnapshotId: snap-0f2d5ab1c6f918116,VolumeSize: 8,VolumeType: gp2,Encrypted: false,}
Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Sep 30, 2019 9:32:04 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-02769bd03e603e42f]}, {Name: instance-type,Values: [t3.micro]}, {Name: key-name,Values: [j4a-bochja]}, {Name: subnet-id,Values: [subnet-0a167eb56a247e891]}, {Name: instance.group-id,Values: [sg-0a4f5e5ac5bb602e4]}, {Name: tag:Name,Values: [ew1-j4a-jenkins-slave-ec2]}, {Name: tag:DeploymentName,Values: [ew1-j4a]}, {Name: tag:Contact,Values: [production-cloud-un...@f-secure.com]}, {Name: tag:CostCenter,Values: [31505]}, {Name: tag:DeploymentType,Values: [dev]}, {Name: tag:DeploymentGroup,Values: [ew1-j4a]}, {Name: tag:jenkins_server_url,Values: [https://mastermaster.j4a.infra-dev-services.fsapi.com/]}, {Name: tag:jenkins_slave_type,Values: [demand_ec2 (ami-02769bd03e603e42f)]}],InstanceIds: [],}
Sep 30, 2019 9:32:05 AM INFO hudson.plugins.ec2.CloudHelper getInstance
Unexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead.
Sep 30, 2019 9:32:05 AM WARNING hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-02769bd03e603e42f', labels='docker docker-bakery'}. Exception during provisioning
com.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0571c7e8c36a6b783', expected 1 result, found []. Instance seems to be dead.
at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54)
at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25)
at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:566)
at hudson.plugins.ec2.EC2AbstractSlave.(EC2AbstractSlave.java:165)
at hudson.plugins.ec2.EC2OndemandSlave.(EC2OndemandSlave.java:56)
at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:1104)
at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:773)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:745)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:540)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:589)
at hudson.plugins.e
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart same problem (starting a new instance when previous instance has been terminated). seen on PR399. C/ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.5785.1569494520190%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Hi, Status update, since last week I have not reproduced the issue when starting a stopped instance. I have reproduced a dozen of times the issue when the previous instance was terminated. I just saw there was a new PR-399 1.46-rc1052.8c6d855421ac associated to this ticket, so I pushed it to our Jenkins, will keep you updated. C/ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.3849.1569306360198%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Hi Raihaan Shouhell, Using the 1.46-rc1050.43f9773eed95 plugin, I reproduced the issue when starting a new EC2 after the previous one was terminated, see attached log start_fresh_1.46-rc1050.43f9773eed95.txt. (what I believe was fixed in PR-397). Issue from a stopped slave has not yet been reproduced. BR, Cedric. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1108.1568892780555%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: cedric lecoz Attachment: start_fresh_1.46-rc1050.43f9773eed95.txt Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1103.1568892780419%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart cedric lecoz yes it is i attached it directly because CI was struggling to build it yesterday. I have removed the attachment. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.146.1568799002138%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Raihaan Shouhell Attachment: ec2.hpi Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.141.1568799000851%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart hi [~raihaan],Is the ec2.hpi plugin you attached here the same which was built by [https://github.com/jenkinsci/ec2-plugin/pull/398] ?It's easier to add to my ci env (automated) when the plugin comes directly from ci.jenkins.io, and easier to track too :)I am asking because it does not . looks like PR-398 includes what I tested from PR-397.tks, C/ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.136.1568799000691%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart hi [~raihaan],Is the ec2.hpi plugin you attached here the same which was built by [https://github.com/jenkinsci/ec2-plugin/pull/398] ?It's easier to add to my ci env (automated) when the plugin comes directly from ci.jenkins.io, and easier to track too :) I am asking because it does not. looks like PR-398 includes what I tested from PR-397. tks,C/ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.131.1568798940260%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart hi Raihaan Shouhell, Is the ec2.hpi plugin you attached here the same which was built by https://github.com/jenkinsci/ec2-plugin/pull/398 ? It's easier to add to my ci env (automated) when the plugin comes directly from ci.jenkins.io, and easier to track too tks, C/ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.126.1568798880321%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart ok tks, will try asap but that may not be before the WE, jenkins is slightly too busy during the week Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4829.1568706900844%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart ec2.hpi For your latest issue the linked HPI should solve it. The issue you seem to see is when starting from a stopped instance due to eventual consistency of AWS APIs it occasionally sees a freshly started instance as stopped as a result for newly started instances I added a retry to deal with this. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4734.1568694780366%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart [^ec2.hpi] [~sirzic] ^For your latest issue the linked HPI should solve it. The issue you seem to see is when starting from a stopped instance due to eventual consistency of AWS APIs it occasionally sees a freshly started instance as stopped as a result for newly started instances I added a retry to deal with this.^ Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4739.1568694780466%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Raihaan Shouhell Attachment: ec2.hpi Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4729.1568694720485%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
cedric lecoz commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
Hi Raihaan Shouhell, Reproduced it twice this morning, attached jenkins.temp_dsl.log one of the log. Plugin manager still shows {1.46-rc1050.a8a95e8dd7f5} for EC2 plugin C.
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4100.1568628780278%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: cedric lecoz Attachment: jenkins.temp_dsl.log Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4095.1568628720279%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart HI Raihaan Shouhell, I updated to that new version this morning, few tests I did were ok. I added a job which will start / destroy / start ..an ec2 using the plugin every 15min, and asked the team to ping me if they see the problem happen. if we don't see the problem, will try to update this ticket by next Thursday. BR, Cedric. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.3790.1568537340268%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart cedric lecoz Jakub Bochenski could someone test https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-397/2/artifact/org/jenkins-ci/plugins/ec2/1.46-rc1050.a8a95e8dd7f5/ec2-1.46-rc1050.a8a95e8dd7f5.hpi and see if this issue still occurs? This retries on missing instances instead of giving up immediately. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.2527.1568357580439%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Hi Raihaan Shouhell, That's what it says, but the EC2 instance was alive, I could ssh and work away, it's just that Jenkins was not aware of it. my instances are on demand. Attached jenkins_201909121030.log a log with a bit more data 2 differences EC2 had the issue (or similar) the first one (aws-audit-ec2) had an EC2 running and terminated an hour before (so it was still showing as terminated in my EC2 console). the second one the EC2 already existed, and was just stopped. I tried to clean the log at best, but I have too many jobs running on other ec2, it's noisy. BR, Cedric. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1957.1568287740593%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title cedric lecoz updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: cedric lecoz Attachment: jenkins_201909121030.log Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1952.1568287740452%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Raihaan Shouhell commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
From this log ``` WARNING: SlaveTemplate {ami='ami-', labels='build-yocto-persistent'} . Node stopped is neither pending, neither running, its {2}. Terminate provisioning ``` It says that the node has been stopped. Btw are you on ondemand slaves or spots
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1740.1568256420393%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
cedric lecoz edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
Hi all, I've done an upgrade of our jenkins last week, ec2 plugin moved from 1.43 to 1.45. This issue had already been seen on 1.43, but rarelly. On new version I get at least an occurence a day (upgrade was core + plugin, everything a couple month old).Going thought logs to try to figure out what was happening before I found this ticket, I saw the following traces, adding it here in case it could help debug the problem. In all the cases, my EC2 instance is started correctly, it's just that jenkins doesn't see it.When it works:{code:java}Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provisionINFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provisionINFO: We have now 27 computers, waiting for 1 moreSep 05, 2019 9:25:53 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl applyINFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1INFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'} Node EC2 (ec2-project) - build-yocto-persistent (i-) moved to RUNNING state in 5 seconds and is ready to be connected by JenkinsSep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logINFO: Launching instance: i-Sep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logSep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logINFO: Connecting to 10.1.0.234 on port 22, with timeout 1.Sep 05, 2019 9:26:03 AM hudson.slaves.NodeProvisioner$2 runINFO: EC2 (ec2-project) - build-yocto-persistent provisioning successfully completed. We have now 27 computer(s)Sep 05, 2019 9:26:03 AM com.tsystems.sbs.LogFileFilterOutputStream {code}When It does not work:{code:java}Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provisionINFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provisionINFO: We have now 27 computers, waiting for 1 moreSep 05, 2019 11:51:13 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl applyINFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud$1 callWARNING: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Node stopped is neither pending, neither running, its {2}. Terminate provisioningSep 05, 2019 11:51:14 AM hudson.plugins.repo.ChangeLog saveChangeLogINFO: No logs found{code}In that case the Node stopped is neither pending, neither running ... trace popped in less that a second instead of the 5 seconds when no problem.An other observation I made, is in my cloudtrail logs, when it works, I can see the following calls to AWS :{code:java} -09:25:53 StartInstances using the i- instance ID. -09:25:53 DescribeInstances, using the i- instance Id as seen in following requestParameters"requestParameters": {"instancesSet": {"items": [{"instanceId": "i-"}]},"filterSet": {}}, -09:25:55 CreateGrant (for decryption) -09:25:58 DescribeInstance using the i- instance Id as seen in above requestParameters. - ...{code}when it does not work:{code:java} -11:51:13 StartInstances using the i- instance ID. -11:51:14 DescribeInstances, using the i- instance Id as seen in above requestParameters -11
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Is there a way to reproduce this consistently from your end? Also what is the number of instances that get run in your setup? It happens quite often after restart but I have no way to reproduce it 100%. Maybe the fact that the instance counting logic sees the EC2 machine (since it says there is no capacity), but the attaching node can't connect it for some reason would be a hint? lso what is the number of instances that get run in your setup? I'm not sure if I understand. The instance cap is 1 so we have at most 1 instance. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.7226.1567692660525%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
cedric lecoz edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
Hi all, I've done an upgrade of our jenkins last week, ec2 plugin moved from 1.43 to 1.45. This issue had already been seen on 1.43, but rarelly. On new version I get at least an occurence a day (upgrade was core + plugin, everything a couple month old) .Going thought logs to try to figure out what was happening before I found this ticket, I saw the following traces, adding it here in case it could help debug the problem. In all the cases, my EC2 instance is started correctly, it's just that jenkins doesn't see it.When it works:{code:java}Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provisionINFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provisionINFO: We have now 27 computers, waiting for 1 moreSep 05, 2019 9:25:53 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl applyINFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1INFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'} Node EC2 (ec2-project) - build-yocto-persistent (i-) moved to RUNNING state in 5 seconds and is ready to be connected by JenkinsSep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logINFO: Launching instance: i-Sep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logSep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud logINFO: Connecting to 10.1.0.234 on port 22, with timeout 1.Sep 05, 2019 9:26:03 AM hudson.slaves.NodeProvisioner$2 runINFO: EC2 (ec2-project) - build-yocto-persistent provisioning successfully completed. We have now 27 computer(s)Sep 05, 2019 9:26:03 AM com.tsystems.sbs.LogFileFilterOutputStream {code}When It does not work:{code:java}Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provisionINFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provisionINFO: We have now 27 computers, waiting for 1 moreSep 05, 2019 11:51:13 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl applyINFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud$1 callWARNING: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Node stopped is neither pending, neither running, its {2}. Terminate provisioningSep 05, 2019 11:51:14 AM hudson.plugins.repo.ChangeLog saveChangeLogINFO: No logs found{code}In that case the Node stopped is neither pending, neither running ... trace popped in less that a second instead of the 5 seconds when no problem.An other observation I made, is in my cloudtrail logs, when it works, I can see the following calls to AWS :{code:java} -09:25:53 StartInstances using the i- instance ID. -09:25:53 DescribeInstances, using the i- instance Id as seen in following requestParameters"requestParameters": {"instancesSet": {"items": [{"instanceId": "i-"}]},"filterSet": {}}, -09:25:55 CreateGrant (for decryption) -09:25:58 DescribeInstance using the i- instance Id as seen in above requestParameters. - ...{code}when it does not work:{code:java} -11:51:13 StartInstances using the i- instance ID. -11:51:14 DescribeInstances, using the i- instance Id as seen in above requestParameters -
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
cedric lecoz commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
Hi all, I've done an upgrade of our jenkins last week, ec2 plugin moved from 1.43 to 1.45. This issue had already been seen on 1.43, but rarelly. On new version I get at least an occurence a day. Going thought logs to try to figure out what was happening before I found this ticket, I saw the following traces, adding it here in case it could help debug the problem. In all the cases, my EC2 instance is started correctly, it's just that jenkins doesn't see it. When it works:
Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1
Sep 05, 2019 9:25:53 AM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 27 computers, waiting for 1 more
Sep 05, 2019 9:25:53 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1
INFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'} Node EC2 (ec2-project) - build-yocto-persistent (i-) moved to RUNNING state in 5 seconds and is ready to be connected by Jenkins
Sep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud log
INFO: Launching instance: i-
Sep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud log
Sep 05, 2019 9:25:58 AM hudson.plugins.ec2.EC2Cloud log
INFO: Connecting to 10.1.0.234 on port 22, with timeout 1.
Sep 05, 2019 9:26:03 AM hudson.slaves.NodeProvisioner$2 run
INFO: EC2 (ec2-project) - build-yocto-persistent provisioning successfully completed. We have now 27 computer(s)
Sep 05, 2019 9:26:03 AM com.tsystems.sbs.LogFileFilterOutputStream
When It does not work:
Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provision
INFO: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Attempting provision finished, excess workload: -1
Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 27 computers, waiting for 1 more
Sep 05, 2019 11:51:13 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning EC2 (ec2-project) - build-yocto-persistent from ec2-ec2-project with 2 executors. Remaining excess workload: -1
Sep 05, 2019 11:51:13 AM hudson.plugins.ec2.EC2Cloud$1 call
WARNING: SlaveTemplate{ami='ami-', labels='build-yocto-persistent'}. Node stopped is neither pending, neither running, its {2}. Terminate provisioning
Sep 05, 2019 11:51:14 AM hudson.plugins.repo.ChangeLog saveChangeLog
INFO: No logs found
In that case the Node stopped is neither pending, neither running ... trace popped in less that a second instead of the 5 seconds when no problem. An other observation I made, is in my cloudtrail logs, when it works, I can see the following calls to AWS :
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I have made a cloud, set the instanceCap to 1 and restarted without running into the orphaning problem. Is there a way to reproduce this consistently from your end? Also what is the number of instances that get run in your setup? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4165.1567417260241%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Raihaan Shouhell I've provided the full configuration via a groovy script above. What else do you need? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.4160.1567417020226%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Raihaan Shouhell commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I'm not sure how to replicate this issue :/ Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.3997.1567395900211%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Jakub Bochenski Environment: Jenkins ver. 2.176.1ec2 plugin 1.43, 1.44 , 1.45 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1009.1565601840446%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart FABRIZIO MANFREDI the same problem is happening with 1.45 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1007.1565601840419%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title FABRIZIO MANFREDI commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Can you test the 1.45 ? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.474.1565464320251%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart > I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days. FABRIZIO MANFREDI it's been a month now and I can't see any new releases after 1.44.1. Any updates? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.7892.1565011860566%40Atlassian.JIRA.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart > One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ? It's not available as an agent in jenkins. It's still in running state when I check the status in AWS console> Did you apply the all the IAM role requested specify in the ec2 plugin page ? I believe I did, since this is a random error it doesn't happen every time . E.g. the instances get terminated after idle timeout Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.2317.1562330340401%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart > One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ? It's not available as an agent in jenkins. It's still in running state when I check the status in AWS console > Did you apply the all the IAM role requested specify in the ec2 plugin page ? I believe I did, since this is a random error it doesn't happen every time Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.2314.1562330220119%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title FABRIZIO MANFREDI commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I believe I found the problem, I trying to put in the 1.44.2 that should be release in a couple of days. One more question what do you mean with orphaned, stop state or no longer in the jenkins interface ? Did you apply the all the IAM role requested specify in the ec2 plugin page ? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.2233.1562317560275%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski edited a comment on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart [~thoulen][~raihaan] I know this is OSS and there are is not SLA. Still could you tell me if I and when I can expect any assistance from you . ? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1830.1562247420130%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart FABRIZIO MANFREDIRaihaan Shouhell I know this is OSS and there are is not SLA. Still could you tell me if I and when I can expect any assistance from you. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.1828.1562247360241%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski updated an issue Jenkins / JENKINS-57795 Orphaned EC2 instances after Jenkins restart Change By: Jakub Bochenski Environment: Jenkins ver. 2.176.1ec2 plugin 1.43, 1.44 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.10656.1561638300177%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart I double checked this. If I'm at cap = 1 with 1 orphaned instance and increase the cap to 2 then the plugin will spawn a new instance. If I'm at cap = 2 with 2 orphaned instances and terminate one of the instances manually then the plugin will reconnect the other instance Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.10552.1561637520111%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Above looks like maybe there is some "off by one" error, when the plugin won't attempt to re-connect instances if it's at instance cap Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.10543.1561635660112%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
I'm now getting this situation with instance cap = 2. I have two matching instances on EC2, both are active. Plugin is looping with above message, with no agents available for the builds. Now when I terminated one of the instances an interesting thing happened. Jenkins was able to pick up the other instance and reconnect it
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. Cannot provision - no capacity for instances: 0
Jun 27, 2019 11:35:07 AM WARNING hudson.plugins.ec2.EC2Cloud provision
Can't raise nodes for SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}
Jun 27, 2019 11:35:16 AM INFO hudson.plugins.ec2.EC2Cloud provision
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. Attempting to provision slave needed by excess workload of 1 units
Jun 27, 2019 11:35:17 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. Considering launching
Jun 27, 2019 11:35:17 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
AMI had xvda
Jun 27, 2019 11:35:17 AM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
{DeleteOnTermination: true,SnapshotId: snap-0b70f104d64ae4a48,VolumeSize: 8,VolumeType: gp2,Encrypted: false,}
Jun 27, 2019 11:35:17 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jun 27, 2019 11:35:17 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. Looking for existing instances with describe-instance: {Filters: SNAP
Jun 27, 2019 11:35:18 AM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker docker-bakery'}. checkInstance: i-0e454aea630ccb88f. true - Instance is not connected to Jenkins
Add Comment
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Can you try with 2 ? If I reproduce the issue with instance cap = 1, then increase the cap to 2 I will get a new agent spawned (but only 1) Now trying to reproduce this with 2 instances getting orphaned. I also tried setting instance cap on slave template to 2 (it was blank before) – doesn't seem to help Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.10524.1561632480494%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
This is happening at least since 1.43 and it just happened on 1.44I have only one EC2 cloud configured, but I also have an ECS cloud (they use separate agent labels).This is our cloud configuration done via groovy script:{code}final cloud = new AmazonEC2Cloud('ec2',false,config.ec2_access_key,config.ec2_region,config.ec2_ssh_key,config.ec2_instance_cap,[new SlaveTemplate( config.ec2_ami_id, '', null, config.ec2_security_groups, '/tmp', InstanceType.fromValue(config.ec2_instance_type), false, config.ec2_label, Node.Mode.NORMAL, "ec2 (${config.ec2_ami_id})", '', '/tmp', '', '1', config.ec2_remote_user, new UnixData(null, null, null, null), '', false, config.ec2_subnet_id, [ Name: ' ew1-j4a-jenkins-slave-ec2 acme ', CostCenter: '31505', DeploymentGroup: 'ew1-j4a', DeploymentName: 'ew1-j4a', DeploymentType: 'dev', Contact : ' production-cloud-unity acme @ f-secure acme .com', // FIXME don't hard-code the region, parametrize ].collect{ new EC2Tag(it.key,it.value) }, '30', false, '', config.ec2_arn_role, true, false, false, '1800', false, '', false, false, false, false)],config.ec2_arn_role,''){code}
Add Comment
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski edited a comment on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
This is happening at least since 1.43 and it just happened on 1.44 I have only one EC2 cloud configured, but I also have an ECS cloud (they use separate agent labels). This is our cloud configuration done via groovy script:{code}final cloud = new AmazonEC2Cloud('ec2',false,config.ec2_access_key,config.ec2_region,config.ec2_ssh_key,config.ec2_instance_cap,[new SlaveTemplate( config.ec2_ami_id, '', null, config.ec2_security_groups, '/tmp', InstanceType.fromValue(config.ec2_instance_type), false, config.ec2_label, Node.Mode.NORMAL, "ec2 (${config.ec2_ami_id})", '', '/tmp', '', '1', config.ec2_remote_user, new UnixData(null, null, null, null), '', false, config.ec2_subnet_id, [ Name: 'ew1-j4a-jenkins-slave-ec2', CostCenter: '31505', DeploymentGroup: 'ew1-j4a', DeploymentName: 'ew1-j4a', DeploymentType: 'dev', Contact : 'production-cloud-un...@f-secure.com', // FIXME don't hard-code the region, parametrize ].collect{ new EC2Tag(it.key,it.value) }, '30', false, '', config.ec2_arn_role, true, false, false, '1800', false, '', false, false, false, false)],config.ec2_arn_role,''){code}
Add Comment
This message was
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski commented on JENKINS-57795
Re: Orphaned EC2 instances after Jenkins restart
This is happening at least since 1.43 and it just happened on 1.44 This is our cloud configuration done via groovy script:
final cloud = new AmazonEC2Cloud(
'ec2',
false,
config.ec2_access_key,
config.ec2_region,
config.ec2_ssh_key,
config.ec2_instance_cap,
[
new SlaveTemplate(
config.ec2_ami_id,
'',
null,
config.ec2_security_groups,
'/tmp',
InstanceType.fromValue(config.ec2_instance_type),
false,
config.ec2_label,
Node.Mode.NORMAL,
"ec2 (${config.ec2_ami_id})",
'',
'/tmp',
'',
'1',
config.ec2_remote_user,
new UnixData(null, null, null, null),
'',
false,
config.ec2_subnet_id,
[
Name: 'ew1-j4a-jenkins-slave-ec2',
CostCenter: '31505',
DeploymentGroup: 'ew1-j4a',
DeploymentName: 'ew1-j4a',
DeploymentType: 'dev',
Contact : 'production-cloud-un...@f-secure.com',
// FIXME don't hard-code the region, parametrize
].collect{ new EC2Tag(it.key,it.value) },
'30',
false,
'',
config.ec2_arn_role,
true,
false,
false,
'1800',
false,
'',
false,
false,
false,
false
)],
config.ec2_arn_role,
''
)
Add Comment
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski updated an issue
Jenkins / JENKINS-57795
Orphaned EC2 instances after Jenkins restart
Change By:
Jakub Bochenski
Sometimes after a Jenkins restart the plugin won't be able to spawn more agents.The plugin will just loop on this:{code}SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Attempting to provision slave needed by excess workload of 1 unitsMay 31, 2019 2:23:53 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Cannot provision - no capacity for instances: 0May 31, 2019 2:23:53 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}{code}If I go to the EC2 console and terminate the instance manually the plugin will spawn a new one and use it.It seems like there is some mismatch in the plugin logic. The part responsible for calculating the number of instances and checking the cap sees the EC2 instance. However the part responsible for picking up running EC2 instances doesn't seem to be able to find it.We use a single subnet, security group and vpc (I've seen some reports about this causing problems). It seems the problems do not occur when I do a {{/safeRestart}} but they do if I use e.g. "restart Jenkins when no jobs are running" form the Update Center. We use instanceCap = 1 setting as we are testing the plugin, this might make this problem more visible than with a higher cap.
Add Comment
This message was sent by
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title FABRIZIO MANFREDI commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart i Can you tell me which version are you using ? There is a bug of the calculation, but should not affect you case. What is the configuration of your pool ? do you have more then one pool with same description, ami and tags ? Can you try with 2 ? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.9521.1561561380122%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart Raihaan Shouhell maybe you would care to respond? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.9470.1561557420193%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title Jakub Bochenski commented on JENKINS-57795 Re: Orphaned EC2 instances after Jenkins restart FABRIZIO MANFREDI it would be nice to at least get some pointers on how to debug this further or work around it Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.199745.1559315049000.8387.1561470180140%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski updated an issue
Jenkins / JENKINS-57795
Orphaned EC2 instances after Jenkins restart
Change By:
Jakub Bochenski
Sometimes after a Jenkins restart the plugin won't be able to spawn more agents.The plugin will just loop on this:{code}SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Attempting to provision slave needed by excess workload of 1 unitsMay 31, 2019 2:23:53 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Cannot provision - no capacity for instances: 0May 31, 2019 2:23:53 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}{code}If I go to the EC2 console and terminate the instance manually the plugin will spawn a new one and use it.It seems like there is some mismatch in the plugin logic. The part responsible for calculating the number of instances and checking the cap sees the EC2 instance. However the part responsible for picking up running EC2 instances doesn't seem to be able to find it.We use a single subnet, security group and vpc (I've seen some reports about this causing problems).It seems the problems do not occur when I do a ` {{ /safeRestart ` }} but they do if I use e.g. "restart Jenkins when no jobs are running" form the Update Center. We use instanceCap = 1 setting as we are testing the plugin, this might make this problem more visible than with a higher cap.
Add Comment
This message was s
[JIRA] (JENKINS-57795) Orphaned EC2 instances after Jenkins restart
Title: Message Title
Jakub Bochenski created an issue
Jenkins / JENKINS-57795
Orphaned EC2 instances after Jenkins restart
Issue Type:
Bug
Assignee:
FABRIZIO MANFREDI
Components:
ec2-plugin
Created:
2019-05-31 15:04
Priority:
Critical
Reporter:
Jakub Bochenski
Sometimes after a Jenkins restart the plugin won't be able to spawn more agents. The plugin will just loop on this:
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Attempting to provision slave needed by excess workload of 1 units
May 31, 2019 2:23:53 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave
SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}. Cannot provision - no capacity for instances: 0
May 31, 2019 2:23:53 PM WARNING hudson.plugins.ec2.EC2Cloud provision
Can't raise nodes for SlaveTemplate{ami='ami-0efbb291c6e8cc847', labels='docker'}
If I go to the EC2 console and terminate the instance manually the plugin will spawn a new one and use it. It seems like there is some mismatch in the plugin logic. The part responsible for calculating the number of instances and checking the cap sees the EC2 instance. However the part responsible for picking up running EC2 instances doesn't seem to be able to find it. We use a single subnet, security group and vpc (I've seen some reports about this causing problems). It seems the problems do not occur when I do a `/safeRestart` but they do if I use e.g. "restart Jenkins when no jobs are running" form the Update Center. We use instanceCap = 1 setting as we are testing the plugin, this might make this problem more visible than with a higher cap.
