Issue Type: Improvement Improvement
Assignee: Francis Upton
Components: ec2-plugin
Created: 05/Feb/15 9:48 AM
Description:

EC2 Jenkins slaves sometimes fails to launch because AWS throttle the API calls. While this would normally not be a problem, this results in a bunch of Jenkins offline slaves which don't go away and can clutter up the UI.

When too many requests hit the EC2 API, Amazon responds with a 503 error like this one:

ERROR: Request limit exceeded. (Service: AmazonEC2; Status Code: 503; Error Code: RequestLimitExceeded; Request ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX)
com.amazonaws.AmazonServiceException: Request limit exceeded. (Service: AmazonEC2; Status Code: 503; Error Code: RequestLimitExceeded; Request ID:xxxxx)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:934)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8567)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4143)
at hudson.plugins.ec2.EC2Computer._describeInstanceOnce(EC2Computer.java:158)
at hudson.plugins.ec2.EC2Computer._describeInstance(EC2Computer.java:141)
at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:117)
at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:60)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:232)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

The EC2 Jenkins plugin could better handle this kind of AWS exceptions. If this error is thrown, maybe it could attempt again after a few minutes.

Enhancing this behavior requires to wrap all API call to handle this error. Any advice on a way to do it properly ?

Project: Jenkins
Priority: Minor Minor
Reporter: Yoann Dubreuil
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Reply via email to