[
https://issues.apache.org/jira/browse/BROOKLYN-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169941#comment-16169941
]
ASF GitHub Bot commented on BROOKLYN-533:
-----------------------------------------
GitHub user aledsage opened a pull request:
https://github.com/apache/brooklyn-server/pull/828
BROOKLYN-533: adds maxConcurrentMachineDeletions
Workaround for BROOKLYN-533: `maxConcurrentMachineDeletions` will ensure
that at most this many machine deletions are done at one time for a given
`JcloudsLocation` instance.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aledsage/brooklyn-server
BROOKLYN-533-maxConcurrentMachineDeletions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/828.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #828
----
commit c8f470c097db6ae8892c39ce2aeb304789bf8ddd
Author: Aled Sage <[email protected]>
Date: 2017-09-18T11:11:32Z
BROOKLYN-533: adds maxConcurrentMachineDeletions
----
> AWS VM deletion failed with "Request limit exceeded"
> ----------------------------------------------------
>
> Key: BROOKLYN-533
> URL: https://issues.apache.org/jira/browse/BROOKLYN-533
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.11.0
> Reporter: Aled Sage
>
> I deployed an app with approx 100 VMs in AWS.
> I then stopped my app, thus terminating all the VMs. However, some requests
> failed with response {{503}}, {{RequestLimitExceeded}}. Those VMs were left
> running.
> [EDIT] jclouds did do an exponential backoff, but because of the heavy load
> it still sometimes failed for the 6 retries.
> The propagated exception is shown below:
> {noformat}
> 2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation
> [r-VlI23lev-80548] Problem releasing machine
> SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
> in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id
> eu-west-1/i-0663997ccc85af459; ignoring and continuing, will throw
> subsequently: org.jclouds.aws.AWSResponseException: request POST
> https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
> AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
> requestToken='null', code='RequestLimitExceeded', message='Request limit
> exceeded.', context='{Response=, Errors=}'}
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
> AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2',
> requestToken='null', code='RequestLimitExceeded', message='Request limit
> exceeded.', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> [259:sts:2.0.2]
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> [66:com.google.guava:18.0.0]
> at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source)
> [47:aws-ec2:2.0.2]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
> [77:ec2:2.0.2]
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
> [47:aws-ec2:2.0.2]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> [77:ec2:2.0.2]
> at
> org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
> [100:jclouds-compute:2.0.2]
> at
> org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
> [100:jclouds-compute:2.0.2]
> at
> org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
> [100:jclouds-compute:2.0.2]
> at
> org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
> [100:jclouds-compute:2.0.2]
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
> [101:jclouds-core:2.0.2.2-20170712_1657]
> at
> org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
> [100:jclouds-compute:2.0.2]
> at
> org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
> [100:jclouds-compute:2.0.2]
> at
> org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
> [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
> [127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
> [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
> [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
> [131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
> [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
> [122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:?]
> at java.lang.Thread.run(Thread.java:748) [?:?]
> {noformat}
> ---
> [EDIT] Extracted the second exception from this bug report to its own jira at
> https://issues.apache.org/jira/browse/BROOKLYN-536
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
