[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"

[ja...@lab-y.com (JIRA)]

Title: Message Title

James Judd commented on JENKINS-26854   Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I resolved the conflicts on (#131) and opened a new pull request (#147. Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line INFO: Excess workload after pending Spot instances: 23 is strange, because we are not using spot instances. Going to try applying only the first commit from (#131) and seeing if that works, sans the fix that martin made. .... com.amazonaws.auth.InstanceProfileCredentialsProvider handleError SEVERE: Unable to load credentials from Amazon EC2 metadata service com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113) at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92) at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ... 9 more Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110) ... 12 more May 21, 2015 3:15:37 AM hudson.plugins.ec2.EC2Cloud provision INFO: Excess workload after pending Spot instances: 23 May 21, 2015 3:15:37 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@39ed6b39 failed java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) at com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:100) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:456) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:440) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:468) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:233) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:304) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:394) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:683) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:284) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:778) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Add Comment   This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.