Aled Sage created BROOKLYN-534:
----------------------------------
Summary: ServiceRestarter policy provisioned new VM while entity
was stopping
Key: BROOKLYN-534
URL: https://issues.apache.org/jira/browse/BROOKLYN-534
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Aled Sage
I had an app consisting of approx 100 VMs. I called stop, to terminate all the
VMs.
However, for one of the entities (nginx) it failed to stop the VM (see
https://issues.apache.org/jira/browse/BROOKLYN-533). Unfortunately the
{{ServiceFailureDetector}} was still active, which caused the
{{ServiceRestarter}} to try to restart it! Because we'd partially shutdown
nginx, it decided that a new machine needed to be provisioned!
I presume that a fix would be to change {{ServiceRestarter}} so that it doesn't
try to restart it if the expected state is 'stopping' or 'stopped'. We should
also check other policies (e.g. {{ServiceReplacer}} to ensure they don't kick
in if their associated entity is being stopped.
Snippets from the log are shown below:
{noformat}
2017-09-15T17:32:52,268 INFO 131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks
[r-VlI23lev-80356] Stopping NginxControllerImpl{id=t2ta5sv8ti} in
[SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]]
2017-09-15T17:32:52,599 DEBUG 122 b.SSH [r-VlI23lev-80528] stopping
NginxControllerImpl{id=t2ta5sv8ti}, initiating ssh on machine
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]:
#!/bin/bash -e
2017-09-15T17:34:09,965 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548]
Problem releasing machine
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id
eu-west-1/i-0663997ccc85af459; ignoring and continuing, will throw
subsequently: org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source)
[47:aws-ec2:2.0.2]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
[77:ec2:2.0.2]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
[47:aws-ec2:2.0.2]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
[77:ec2:2.0.2]
at
org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
[100:jclouds-compute:2.0.2]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation
[r-VlI23lev-80548]
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
invalidating all entries in ssh pool cache. Final stats:
CacheStats{hitCount=803, missCount=6, loadSuccessCount=6, loadExceptionCount=0,
totalLoadTime=1248584, evictionCount=4}
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation
[r-VlI23lev-80548]
BasicPool{name=Pool(34.252.178....@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548]
BasicPool{name=Pool(34.252.178....@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
closing, with 1 resources (currentLeased=0; totalLeased=789; totalCreated=1;
totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation
[r-VlI23lev-80548]
org.apache.brooklyn.location.ssh.SshMachineLocation$8@710df8f2 closing pool for
[email protected]:22
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation
[r-VlI23lev-80548]
BasicPool{name=Pool(34.252.178....@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
evicted from SshTool cache. Eviction cause: EXPLICIT
2017-09-15T17:34:09,966 DEBUG 143 o.a.b.u.p.BasicPool [r-VlI23lev-80548]
BasicPool{name=Pool(34.252.178....@ec2-34-252-178-167.eu-west-1.compute.amazonaws.com/34.252.178.167:22:hash1856793534)}
closing, with 1 resources (currentLeased=0; totalLeased=1; totalCreated=1;
totalClosed=0)
2017-09-15T17:34:09,966 DEBUG 122 o.a.b.l.s.SshMachineLocation
[r-VlI23lev-80548]
org.apache.brooklyn.location.ssh.SshMachineLocation$8@6a0ceb61 closing pool for
[email protected]:22
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager
[r-VlI23lev-80548]
org.apache.brooklyn.core.mgmt.internal.LocalLocationManager@68f8b22f stopped
management of location
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager
[r-VlI23lev-80548] Missing transition mode for
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
when unmanaging; assuming primary/destroying
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalUsageManager
[r-VlI23lev-80548] Storing location lifecycle usage event: location
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
in state destroyed; caller context NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,967 INFO 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-80548]
Attempted release of unknown machine
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.m.i.LocalLocationManager
[r-VlI23lev-80548] Error releasing
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
in its parent JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]:
java.lang.IllegalArgumentException: Unknown machine
SshMachineLocation[34.252.178.167:[email protected]/34.252.178.167:22(id=taep3uro9m)]
2017-09-15T17:34:09,967 DEBUG 122 o.a.b.c.s.AttributeMap [r-VlI23lev-80548]
removing attribute internal.termination.task.state on
NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,968 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80548]
Emitting sensor notification entity.sensor.removed value Sensor:
internal.termination.task.state
(org.apache.brooklyn.core.entity.internal.AttributesInternal$ProvisioningTaskState)
on NginxControllerImpl{id=t2ta5sv8ti}
...
2017-09-15T17:34:09,972 DEBUG 129 o.a.b.p.h.ServiceFailureDetector
[r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo}
health-check for NginxControllerImpl{id=t2ta5sv8ti}, component now failing:
location=[]; status=Present[value=on-fire]; lastPublished=NONE;
timeNow=2017-09-15 17:34:09.972; currentFailurePeriod=<none> (stabilization
0ms); currentRecoveryPeriod=<none> (stabilization 0ms)
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector
[r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo}
publishing failed (state=Present[value=on-fire];
currentFailureStartTime=2017-09-15 17:34:09.972; now=2017-09-15 17:34:09.972
2017-09-15T17:34:09,973 DEBUG 122 o.a.b.c.e.AbstractEntity [r-VlI23lev-80386]
Emitting sensor notification ha.entityFailed value
FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up} on NginxControllerImpl{id=t2ta5sv8ti}
2017-09-15T17:34:09,973 DEBUG 129 o.a.b.p.h.ServiceFailureDetector
[r-VlI23lev-80386] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting
on-fire, now that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,973 INFO 129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80803]
ServiceRestarter notified; dispatching job for
NginxControllerImpl{id=t2ta5sv8ti}
(FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up})
2017-09-15T17:34:09,974 WARN 129 o.a.b.p.h.ServiceRestarter [r-VlI23lev-80548]
ServiceRestarter acting on failure detected at
NginxControllerImpl{id=t2ta5sv8ti}
(FailureDescriptor{component=NginxControllerImpl{id=t2ta5sv8ti},
description=service not up})
2017-09-15T17:34:09,974 DEBUG 129 o.a.b.p.h.ServiceFailureDetector
[r-VlI23lev-80356] ServiceFailureDetector{uniqueTag=service.state.actual,
running=true, entity=NginxControllerImpl{id=t2ta5sv8ti}, id=eox0ce3omo} setting
on-fire, now that deferred period has passed (state=Present[value=on-fire])
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.u.c.t.CompoundTask [r-VlI23lev-80356]
Parent task DstJob:pXCtsb5q ignoring child error
(org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}) in presence of our own error
(org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'})
2017-09-15T17:34:09,974 DEBUG 122 o.a.b.c.e.Effectors [r-VlI23lev-80548]
Replacing invocation of restart[] on NginxControllerImpl{id=t2ta5sv8ti} with
restart[restartChildren,restartMachine] which is the impl defined at that entity
2017-09-15T17:34:09,974 WARN 122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356]
Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
2017-09-15T17:34:09,975 DEBUG 131
o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386]
restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing
machine-level restart
2017-09-15T17:34:09,975 DEBUG 131
o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386]
restart of NginxControllerImpl{id=t2ta5sv8ti} requested be applied at machine
level
2017-09-15T17:34:09,975 DEBUG 131
o.a.b.e.s.b.SoftwareProcessDriverLifecycleEffectorTasks [r-VlI23lev-80386]
restart of NginxControllerImpl{id=t2ta5sv8ti} has no hostname - doing
machine-level restart
...
2017-09-15T17:34:09,975 DEBUG 122 o.a.b.c.m.i.EffectorUtils [r-VlI23lev-80356]
Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129)
[143:org.apache.brooklyn.utils-common:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:870)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStopLatching(MachineLifecycleEffectorTasks.java:757)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stop(MachineLifecycleEffectorTasks.java:744)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:293)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopEffectorBody.call(MachineLifecycleEffectorTasks.java:290)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.core.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:82)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.util.concurrent.ExecutionException:
org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:?]
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
~[?:?]
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:382)
~[?:?]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.doStop(MachineLifecycleEffectorTasks.java:858)
~[?:?]
... 11 more
Caused by: org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[?:?]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[?:?]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[?:?]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[?:?]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source)
~[?:?]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getRunningInstanceInRegion(EC2GetNodeMetadataStrategy.java:64)
~[?:?]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:52)
~[?:?]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[?:?]
at
org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:42)
~[?:?]
at
org.jclouds.compute.predicates.AtomicNodeTerminated.refreshOrNull(AtomicNodeTerminated.java:28)
~[?:?]
at
org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:46)
~[?:?]
at
org.jclouds.compute.predicates.internal.TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.apply(TrueIfNullOrDeletedRefreshAndDoubleCheckOnFalse.java:31)
~[?:?]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
~[?:?]
at
org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
~[?:?]
at
org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
~[?:?]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
~[?:?]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
~[?:?]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
~[?:?]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
~[?:?]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
~[?:?]
... 6 more
2017-09-15T17:34:09,978 DEBUG 122 o.a.b.u.c.t.BasicExecutionManager
[r-VlI23lev-80356] Exception running task Task[stop]@pXCtsb5q (rethrowing):
org.apache.brooklyn.core.mgmt.internal.EffectorUtils$EffectorCallPropagatedRuntimeException:
Error invoking stop at NginxControllerImpl{id=t2ta5sv8ti}:
AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/
HTTP/1.1 failed with code 503, error:
AWSError{requestId='48ed534d-a788-43e2-aa97-2fce47716db2', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
2017-09-15T17:34:09,979 INFO 131 o.a.b.e.s.b.l.MachineLifecycleEffectorTasks
[r-VlI23lev-80833] Stopping NginxControllerImpl{id=t2ta5sv8ti} in []
...
2017-09-15T17:34:10,262 ERROR 107 o.j.a.h.AWSServerErrorRetryHandler
[r-VlI23lev-81233] Cannot retry after server error, command has exceeded retry
limit 6: [method=org.jclouds.aws.ec2.features.AWSInstanceApi.public abstract
java.util.Set
org.jclouds.aws.ec2.features.AWSInstanceApi.describeInstancesInRegion(java.lang.String,java.lang.String[])[eu-west-1,
[Ljava.lang.String;@3c00d5d0], request=POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1]
2017-09-15T17:34:10,263 ERROR 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81233]
Problem releasing machine
SshMachineLocation[34.240.150.69:[email protected]/34.240.150.69:22(id=dpxjea986f)]
in JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx], instance id
eu-west-1/i-02e2311ab1f16a0c8; ignoring and continuing, will throw
subsequently: org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
org.jclouds.aws.AWSResponseException: request POST
https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 503, error:
AWSError{requestId='e0e6ebbf-9a19-43ab-9219-25907eb8aa71', requestToken='null',
code='RequestLimitExceeded', message='Request limit exceeded.',
context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
[259:sts:2.0.2]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:140)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:109)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
[66:com.google.guava:18.0.0]
at com.sun.proxy.$Proxy179.describeInstancesInRegion(Unknown Source)
[47:aws-ec2:2.0.2]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2DestroyNodeStrategy.destroyInstanceInRegion(AWSEC2DestroyNodeStrategy.java:59)
[47:aws-ec2:2.0.2]
at
org.jclouds.ec2.compute.strategy.EC2DestroyNodeStrategy.destroyNode(EC2DestroyNodeStrategy.java:72)
[77:ec2:2.0.2]
at
org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:298)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.internal.BaseComputeService$2.apply(BaseComputeService.java:295)
[100:jclouds-compute:2.0.2]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:117)
[101:jclouds-core:2.0.2.2-20170712_1657]
at
org.jclouds.compute.internal.BaseComputeService.doDestroyNode(BaseComputeService.java:309)
[100:jclouds-compute:2.0.2]
at
org.jclouds.compute.internal.BaseComputeService.destroyNode(BaseComputeService.java:250)
[100:jclouds-compute:2.0.2]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.releaseNode(JcloudsLocation.java:2189)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2141)
[127:org.apache.brooklyn.locations-jclouds:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks.stopAnyProvisionedMachines(MachineLifecycleEffectorTasks.java:1033)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:883)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopAnyProvisionedMachinesTask.call(MachineLifecycleEffectorTasks.java:880)
[131:org.apache.brooklyn.software-base:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
...
2017-09-15T17:36:28,127 INFO 127 o.a.b.l.j.JcloudsLocation [r-VlI23lev-81071]
Creating VM aws-ec2:eu-west-1@NginxControllerImpl{id=h0xas9pjk5} in
JcloudsLocation[AWS Dublin:xxxxxxxx@xxxxxxxx]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
