| I realise this is not much help but strangely enough, it started working for us again. No Jenkins/plugin upgrades have been done since the issue happened. The immediate set of actions I did before it started working was:
- To get around this issue, we'd provisioned (through the EC2 plugin) an always-running instance (with no idle timeout)
- Today, because we'd run out of space, I deleted it
- In the interim before I'd had a chance to , we had a build get triggered via a commit
- This automatically brought up a new instance as it used to
I'm also trying to get a debug log across but it's a little difficult as I need to lock in a period of time where our master build server is inaccessible. In the meantime, here's the log from when it started working:
Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Terminated EC2 instance (terminated): i-xxxxxx
Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Removed EC2 instance from jenkins master: i-xxxxxx
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@xxx needed by excess workload of 1 units of label 'null'
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-xxxxxx]}, {Name: subnet-id,Values: [subnet-xxxxxx]}, {Name: instance.group-id,Values: [sg-xxxxxx]}, {Name: key-name,Values: [jenkins-slave]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2RetentionStrategy start
INFO: Start requested for Remote Ubuntu 14.04 Docker Machine (i-xxxxxx)
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Added node named: Remote Ubuntu 14.04 Docker Machine (i-xxxxxx), We have now 2 computers
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: -1
Jul 28, 2016 11:44:26 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Remote Ubuntu 14.04 Docker Machine (ami-xxxxxx) from ec2-jenkins-user with 2 executors. Remaining excess workload: -1
Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 55 ms
|
