[
https://issues.apache.org/jira/browse/WHIRR-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497505#comment-13497505
]
Steve Loughran commented on WHIRR-682:
--------------------------------------
A stack of an attempt to create a two node cluster
{code}
Bootstrapping cluster
Configuring template for bootstrap-hadoop-datanode_hadoop-tasktracker
Configuring template for
bootstrap-hadoop-namenode_hadoop-jobtracker_hadoop-datanode_hadoop-tasktracker
Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker,
hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Unexpected error while starting 1 nodes, minimum 1 nodes for [hadoop-datanode,
hadoop-tasktracker] of cluster hadoop
java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:71)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listNodes(AdaptingComputeServiceStrategies.java:110)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getNextNames(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:185)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:124)
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:148)
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.createNodesInGroup(Unknown Source)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listNodes(AdaptingComputeServiceStrategies.java:110)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getNextNames(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:185)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:124)
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:148)
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.createNodesInGroup(Unknown Source)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30)
... 5 more
Caused by: org.jclouds.http.HttpResponseException: command: GET
https://lon.servers.api.rackspacecloud.com/v2/10025232/servers/detail HTTP/1.1
failed with response: HTTP/1.1 413 Request Entity Too Large; content: [{
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}]
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:48)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
... 5 more
Destroying failed nodes [LON/e5dd56e3-1288-445d-9ba3-18e9d0268428]
*CRITICAL* the cluster failed to launch and the automated node termination
option was not selected, there might be orphaned nodes.
java.io.IOException: java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125)
at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:127)
at
org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
at org.apache.whirr.cli.Main.run(Main.java:69)
at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120)
... 7 more
Caused by: org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:71)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.internal.BaseComputeService.detailsOnAllNodes(BaseComputeService.java:651)
at
org.jclouds.compute.internal.BaseComputeService.nodesMatchingFilterAndNotTerminated(BaseComputeService.java:327)
at
org.jclouds.compute.internal.BaseComputeService.destroyNodesMatching(BaseComputeService.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.destroyNodesMatching(Unknown Source)
at
org.apache.whirr.compute.StartupProcess.cleanupFailedNodes(StartupProcess.java:169)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:97)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.internal.BaseComputeService.detailsOnAllNodes(BaseComputeService.java:651)
at
org.jclouds.compute.internal.BaseComputeService.nodesMatchingFilterAndNotTerminated(BaseComputeService.java:327)
at
org.jclouds.compute.internal.BaseComputeService.destroyNodesMatching(BaseComputeService.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.destroyNodesMatching(Unknown Source)
at
org.apache.whirr.compute.StartupProcess.cleanupFailedNodes(StartupProcess.java:169)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:97)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.http.HttpResponseException: command: GET
https://lon.servers.api.rackspacecloud.com/v2/10025232/servers/detail HTTP/1.1
failed with response: HTTP/1.1 413 Request Entity Too Large; content: [{
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}]
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:48)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
... 5 more
Exception in thread "main" java.lang.RuntimeException: java.io.IOException:
java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:128)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
at org.apache.whirr.cli.Main.run(Main.java:69)
at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: java.io.IOException: java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125)
at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:127)
at
org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
... 4 more
Caused by: java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120)
... 7 more
Caused by: org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:71)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.internal.BaseComputeService.detailsOnAllNodes(BaseComputeService.java:651)
at
org.jclouds.compute.internal.BaseComputeService.nodesMatchingFilterAndNotTerminated(BaseComputeService.java:327)
at
org.jclouds.compute.internal.BaseComputeService.destroyNodesMatching(BaseComputeService.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.destroyNodesMatching(Unknown Source)
at
org.apache.whirr.compute.StartupProcess.cleanupFailedNodes(StartupProcess.java:169)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:97)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.internal.BaseComputeService.detailsOnAllNodes(BaseComputeService.java:651)
at
org.jclouds.compute.internal.BaseComputeService.nodesMatchingFilterAndNotTerminated(BaseComputeService.java:327)
at
org.jclouds.compute.internal.BaseComputeService.destroyNodesMatching(BaseComputeService.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.destroyNodesMatching(Unknown Source)
at
org.apache.whirr.compute.StartupProcess.cleanupFailedNodes(StartupProcess.java:169)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:97)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.http.HttpResponseException: command: GET
https://lon.servers.api.rackspacecloud.com/v2/10025232/servers/detail HTTP/1.1
failed with response: HTTP/1.1 413 Request Entity Too Large; content: [{
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}]
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:48)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
... 5 more
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker,
hadoop-datanode, hadoop-tasktracker]
Unexpected error while starting 1 nodes, minimum 1 nodes for [hadoop-namenode,
hadoop-jobtracker, hadoop-datanode, hadoop-tasktracker] of cluster hadoop
java.util.concurrent.ExecutionException:
org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83)
at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.rest.InsufficientResourcesException: {
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:71)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listNodes(AdaptingComputeServiceStrategies.java:110)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getNextNames(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:185)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:124)
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:148)
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.createNodesInGroup(Unknown Source)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
at
org.jclouds.concurrent.config.DescribingExecutorService.submit(DescribingExecutorService.java:89)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:132)
at
org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
at
org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
at
org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:255)
at
org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:149)
at $Proxy94.listInDetail(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170)
at $Proxy95.listInDetail(Unknown Source)
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.listNodes(NovaComputeServiceAdapter.java:198)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:115)
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listNodes(AdaptingComputeServiceStrategies.java:110)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getNextNames(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:185)
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:124)
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:148)
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at $Proxy73.createNodesInGroup(Unknown Source)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55)
at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30)
... 5 more
Caused by: org.jclouds.http.HttpResponseException: command: GET
https://lon.servers.api.rackspacecloud.com/v2/10025232/servers/detail HTTP/1.1
failed with response: HTTP/1.1 413 Request Entity Too Large; content: [{
"overLimit" : {
"code" : 413,
"message" : "OverLimit Retry...",
"details" : "Error Details...",
"retryAt" : "2012-11-14T21:51:28UTC"
}
}]
at
org.jclouds.openstack.nova.v2_0.handlers.NovaErrorHandler.handleError(NovaErrorHandler.java:48)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)
... 5 more
Destroying failed nodes [LON/4bf3177a-f7d2-49fa-b3bb-51c62bc068ab]
{code}
> OpenStack Nova throttling prevents Whirr from bringing up clusters with
> multiple nodes
> --------------------------------------------------------------------------------------
>
> Key: WHIRR-682
> URL: https://issues.apache.org/jira/browse/WHIRR-682
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.9.0
> Environment: whirr.provider=rackspace-cloudservers-uk
> Reporter: Steve Loughran
>
> You can't create a 3 node hadoop cluster on OpenStack Nova, because it
> rate-limits VM requests (in time and per-day). The endpoints send a 413-coded
> message, but this is not detected as a retryable action
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
