[
https://issues.apache.org/jira/browse/JCLOUDS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaron Rosenbaum updated JCLOUDS-767:
------------------------------------
Description:
My program creates spot instances (see 'addNode' below).
Recently, in some cases I've been getting "RequestResourceCountExceeded" error
(see below).
I am creating multiple nodes on multiple threads (3).
It appears that jClouds is polling node status too frequently, and AWS
perceives this as a DOS attack. AWS polling queries fail on "The amount of data
you requested and frequency of requests together exceed the levels allowed by
the Spot service".
jClouds interprets this as a failure to create, whereas this is just a failure
to poll.
The spot requests, however, are valid - and eventually fulfilled, but because
of this error - they are never properly configured by jClouds, and in fact -
are just left as orphans.
This scenario, in turn, ends up creating more and more nodes - endlessly, all
left as orphans (but costing dear $).
So,
1) This specific error should not be interpreted as failure
2) The polling interval should be increased, or at least configurable
3) Maybe some sort of exponential backoff should be used in case this error
happens
4) Failed spot requests should verify that the spot request is cancelled
(the code below happens on 3 threads, concurrently)
public ComputeMetadata addNode(Template template) throws
RunNodesException {
ComputeMetadata result =
Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(),
1, template));
LOG.info("Node \"{}\" created", result.getName());
10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is
missing 3 nodes
10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02gebgm2}])
10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02gez2h5}])
10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02ghn9t2}])
10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing
node(us-east-1/sir-02gez2h5):
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
~[na:na]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[na:1.7.0_09]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed
to create node:
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
location(us-east-1) image(ami-06ff446e) size(m1.medium)
options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:
0 error[s]
Node failures:
1) AWSResponseException on node us-east-1/sir-02gez2h5:
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
1 error[s]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
at
io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
~[classes/:na]
at
io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing
node(us-east-1/sir-02gebgm2):
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
~[na:na]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[na:1.7.0_09]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed
to create node:
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
location(us-east-1) image(ami-06ff446e) size(m1.medium)
options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:
0 error[s]
Node failures:
1) AWSResponseException on node us-east-1/sir-02gebgm2:
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
1 error[s]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
at
io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
~[classes/:na]
at
io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node
"AWS-East-1-8b5f5f66" created
10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Adding
node "AWS-East-1-8b5f5f66" to load balancer
10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
Creating node using template: {image={id=us-east-1/ami-06ff446e,
providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION,
id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
os={family=unrecognized, arch=paravirtual, version=,
description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS
beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root,
userMetadata={owner=595879546273, rootDeviceType=ebs,
virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium,
providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750,
volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true,
durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false,
durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false,
durable=false}],
supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())},
location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2,
iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron,
userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013,
groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
was:
My program creates spot instances (see 'addNode' below).
Recently, in some cases I've been getting "RequestResourceCountExceeded" error
(see below).
The problem is that JClouds throws an exception, but the spot request IS
created and executed by AWS. The node is never properly configured. Thus, I
have no way to cancel the spot request (there's no node ID).
Correct behaviour should be: Failure should leave no active spot requests.
At the very least, there should be a way to cancel the spot request if it fails.
Also, it does seem like it shouldn't have failed - since the spot request is
successful...
public ComputeMetadata addNode(Template template) throws
RunNodesException {
ComputeMetadata result =
Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(),
1, template));
LOG.info("Node \"{}\" created", result.getName());
if (isLoadBalancerLogicEnabled()) {
LOG.info("Adding node \"{}\" to load balancer",
result.getName());
addNodeToLoadBalancer(result);
}
10:05:43.966 [main] INFO io.multicloud.controller.AWSProvider - Successfully
created compute context
10:06:02.795 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is
missing 3 nodes
10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02gebgm2}])
10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02gez2h5}])
10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible
instances([{region=us-east-1, name=sir-02ghn9t2}])
10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing
node(us-east-1/sir-02gez2h5):
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
~[na:na]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[na:1.7.0_09]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed
to create node:
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
location(us-east-1) image(ami-06ff446e) size(m1.medium)
options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:
0 error[s]
Node failures:
1) AWSResponseException on node us-east-1/sir-02gez2h5:
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
1 error[s]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
at
io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
~[classes/:na]
at
io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing
node(us-east-1/sir-02gebgm2):
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
~[na:na]
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
~[jclouds-core-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[na:1.7.0_09]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed
to create node:
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
location(us-east-1) image(ami-06ff446e) size(m1.medium)
options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:
0 error[s]
Node failures:
1) AWSResponseException on node us-east-1/sir-02gebgm2:
org.jclouds.aws.AWSResponseException: request POST
https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null',
code='RequestResourceCountExceeded', message='the amount of data you requested
and frequency of requests together exceed the levels allowed by the Spot
service', context='{Response=, Errors=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
at
org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
at
org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
at
org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
at
org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
at
org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
at
org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
1 error[s]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
at
org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
at
io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
~[classes/:na]
at
io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_09]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node
"AWS-East-1-8b5f5f66" created
10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Adding
node "AWS-East-1-8b5f5f66" to load balancer
10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
Creating node using template: {image={id=us-east-1/ami-06ff446e,
providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION,
id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
os={family=unrecognized, arch=paravirtual, version=,
description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS
beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root,
userMetadata={owner=595879546273, rootDeviceType=ebs,
virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium,
providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750,
volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true,
durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false,
durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false,
durable=false}],
supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())},
location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2,
iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron,
userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013,
groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Name:
AWS-East-1-8b5f5f66, User metadata: {TYPE=SPOT, Name=AWS-East-1-8b5f5f66}
10:09:39.106 [main] INFO io.multicloud.controller.AWSProvider - Optimizing
availability: Adding 1 nodes
10:09:39.106 [main] INFO i.m.controller.CloudProviderImpl - Adding 1 nodes
> Creating multiple spot instances fails. Status polling too frequent?
> --------------------------------------------------------------------
>
> Key: JCLOUDS-767
> URL: https://issues.apache.org/jira/browse/JCLOUDS-767
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-compute
> Affects Versions: 1.8.1
> Environment: AWS, Spot instances, CoreOS
> Reporter: Yaron Rosenbaum
> Priority: Blocker
> Labels: AWS, DOS, Spot
>
> My program creates spot instances (see 'addNode' below).
> Recently, in some cases I've been getting "RequestResourceCountExceeded"
> error (see below).
> I am creating multiple nodes on multiple threads (3).
> It appears that jClouds is polling node status too frequently, and AWS
> perceives this as a DOS attack. AWS polling queries fail on "The amount of
> data you requested and frequency of requests together exceed the levels
> allowed by the Spot service".
> jClouds interprets this as a failure to create, whereas this is just a
> failure to poll.
> The spot requests, however, are valid - and eventually fulfilled, but because
> of this error - they are never properly configured by jClouds, and in fact -
> are just left as orphans.
> This scenario, in turn, ends up creating more and more nodes - endlessly, all
> left as orphans (but costing dear $).
> So,
> 1) This specific error should not be interpreted as failure
> 2) The polling interval should be increased, or at least configurable
> 3) Maybe some sort of exponential backoff should be used in case this error
> happens
> 4) Failed spot requests should verify that the spot request is cancelled
> (the code below happens on 3 threads, concurrently)
> public ComputeMetadata addNode(Template template) throws
> RunNodesException {
> ComputeMetadata result =
> Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(),
> 1, template));
> LOG.info("Node \"{}\" created", result.getName());
> 10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is
> missing 3 nodes
> 10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
> 10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02gebgm2}])
> 10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02gez2h5}])
> 10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02ghn9t2}])
> 10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing
> node(us-east-1/sir-02gez2h5):
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> ~[sts-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> ~[na:na]
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> ~[aws-ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl -
> Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
> location(us-east-1) image(ami-06ff446e) size(m1.medium)
> options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
> subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab,
> sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gez2h5:
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
> ~[classes/:na]
> at
> io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
> ~[classes/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing
> node(us-east-1/sir-02gebgm2):
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> ~[sts-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> ~[na:na]
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> ~[aws-ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl -
> Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
> location(us-east-1) image(ami-06ff446e) size(m1.medium)
> options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
> subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab,
> sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gebgm2:
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
> ~[classes/:na]
> at
> io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
> ~[classes/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node
> "AWS-East-1-8b5f5f66" created
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
> Adding node "AWS-East-1-8b5f5f66" to load balancer
> 10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
> Creating node using template: {image={id=us-east-1/ami-06ff446e,
> providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION,
> id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
> os={family=unrecognized, arch=paravirtual, version=,
> description=595879546273/CoreOS-beta-444.5.0, is64Bit=true},
> description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available],
> loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs,
> virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium,
> providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750,
> volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true,
> durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false,
> durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false,
> durable=false}],
> supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())},
> location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2,
> iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron,
> userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013,
> groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
