[
https://issues.apache.org/jira/browse/JCLOUDS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219079#comment-14219079
]
Yaron Rosenbaum commented on JCLOUDS-767:
-----------------------------------------
Hi
After further research with Amazon, it looks like jClouds is doing some sort of
POLLING on spot requests, to check if they have been fulfilled and/or the
instance has been created. It looks like, this polling happens too frequently
for Amazon, and they perceive this as some sort of attack, and therefore send
the error I have reported.
I would like to suggest:
1) Increasing the polling interval,
2) Sending a new polling request only when the previous one has returned (or a
long timeout has ben breached)
3) Possibly allowing this to be a configurable parameter
At the moment, the result of this behaviour is that no more than one concurrent
spot request may be handled by jClouds.
Note, that I am requesting a number of spots on different threads, and not
issuing one request for multiple spots.
Here's what Amazon support say:
"This error comes from querying bid status (e.g.,
"ec2-describe-spot-instance-requests") too frequently. In this case, you are
calling this API before a previous call returned (perhaps calling from multiple
clients in a tight loop). While that's supported by many AWS APIs, we've had
problems with brown-outs because of it and need to protect the service. We
believe that most cases where a customer is throttled will be avoided by simply
waiting for a response before calling again (sleeping for 1 second between
response received and next request would avoid throttling for most Spot
customers). The underlying bid data changes slowly, so there little to be
gained by calling more than twice a minute in any case.
"
> Failed spot request but still created, node is null - unable to cancel
> ----------------------------------------------------------------------
>
> Key: JCLOUDS-767
> URL: https://issues.apache.org/jira/browse/JCLOUDS-767
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-compute
> Affects Versions: 1.8.1
> Environment: AWS, Spot instances, CoreOS
> Reporter: Yaron Rosenbaum
> Priority: Critical
>
> My program creates spot instances (see 'addNode' below).
> Recently, in some cases I've been getting "RequestResourceCountExceeded"
> error (see below).
> The problem is that JClouds throws an exception, but the spot request IS
> created and executed by AWS. The node is never properly configured. Thus, I
> have no way to cancel the spot request (there's no node ID).
> Correct behaviour should be: Failure should leave no active spot requests.
> At the very least, there should be a way to cancel the spot request if it
> fails.
> Also, it does seem like it shouldn't have failed - since the spot request is
> successful...
> public ComputeMetadata addNode(Template template) throws
> RunNodesException {
> ComputeMetadata result =
> Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(),
> 1, template));
> LOG.info("Node \"{}\" created", result.getName());
> if (isLoadBalancerLogicEnabled()) {
> LOG.info("Adding node \"{}\" to load balancer",
> result.getName());
> addNodeToLoadBalancer(result);
> }
> 10:05:43.966 [main] INFO io.multicloud.controller.AWSProvider - Successfully
> created compute context
> 10:06:02.795 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:06:02.795 [main] INFO io.multicloud.controller.AWSProvider - quorum is
> missing 3 nodes
> 10:06:03.077 [main] INFO i.m.controller.CloudProviderImpl - Adding 3 nodes
> 10:06:04.113 [pool-3-thread-1] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02gebgm2}])
> 10:06:04.342 [pool-3-thread-3] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02gez2h5}])
> 10:06:04.460 [pool-3-thread-2] WARN jclouds.compute - << not api visible
> instances([{region=us-east-1, name=sir-02ghn9t2}])
> 10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing
> node(us-east-1/sir-02gez2h5):
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> ~[sts-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> ~[na:na]
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> ~[aws-ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl -
> Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
> location(us-east-1) image(ami-06ff446e) size(m1.medium)
> options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
> subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab,
> sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gez2h5:
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
> ~[classes/:na]
> at
> io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
> ~[classes/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing
> node(us-east-1/sir-02gebgm2):
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> ~[sts-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-17.0.jar:na]
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> ~[na:na]
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> ~[aws-ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> ~[jclouds-core-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> [jclouds-compute-1.8.1.jar:1.8.1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [na:1.7.0_09]
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl -
> Failed to create node:
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1)
> location(us-east-1) image(ami-06ff446e) size(m1.medium)
> options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
> subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab,
> sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gebgm2:
> org.jclouds.aws.AWSResponseException: request POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error:
> AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
> requestToken='null', code='RequestResourceCountExceeded', message='the amount
> of data you requested and frequency of requests together exceed the levels
> allowed by the Spot service', context='{Response=, Errors=}'}
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> at
> org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> at
> org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> at
> org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> at
> org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> at
> org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> at
> org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> at
> org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> at
> org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> at
> org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
> ~[jclouds-compute-1.8.1.jar:1.8.1]
> at
> org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
> ~[ec2-1.8.1.jar:1.8.1]
> at
> io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65)
> ~[classes/:na]
> at
> io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
> ~[classes/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_09]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl - Node
> "AWS-East-1-8b5f5f66" created
> 10:09:24.051 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
> Adding node "AWS-East-1-8b5f5f66" to load balancer
> 10:09:25.110 [pool-3-thread-2] INFO i.m.controller.CloudProviderImpl -
> Creating node using template: {image={id=us-east-1/ami-06ff446e,
> providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION,
> id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
> os={family=unrecognized, arch=paravirtual, version=,
> description=595879546273/CoreOS-beta-444.5.0, is64Bit=true},
> description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available],
> loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs,
> virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium,
> providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750,
> volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true,
> durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false,
> durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false,
> durable=false}],
> supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())},
> location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2,
> iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron,
> userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013,
> groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Name:
> AWS-East-1-8b5f5f66, User metadata: {TYPE=SPOT, Name=AWS-East-1-8b5f5f66}
> 10:09:39.106 [main] INFO io.multicloud.controller.AWSProvider - Optimizing
> availability: Adding 1 nodes
> 10:09:39.106 [main] INFO i.m.controller.CloudProviderImpl - Adding 1 nodes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
