This is the guy who's been an intern for us (I think) and does a ton of Whirr stuff. Always tempting to hire him, even though he's crazy remote. phunt knows him.
---------- Forwarded message ---------- From: Andrei Savu <savu.and...@gmail.com> Date: Wed, May 23, 2012 at 8:01 AM Subject: Re: Running cluster is destroyed when launching cluster with the same name To: "dev@whirr.apache.org" <dev@whirr.apache.org> This behaviour is expected for the current implementation. It's easy to add a naive check using list-cluster but that's only effective if multiple cluster having the same name are *not* started at the same time. We can also implement this by using a file lock but this assumes that for a given cloud provider account we are using a single machine to launch clusters. We had a similar discussion in the past. This is the related issue: https://issues.apache.org/jira/browse/WHIRR-329 Do you want to give it a try to provide a patch? -- Andrei Savu On Wednesday, May 23, 2012, Andrii Vozniuk wrote: > Some clarification. > > Indeed, I wanted to launch a cluster of 5 m1.medium instances > simultaneously with running cluster of 5 m1.small instances. By mistake, > the whirr.cluster-name value was the same in corresponding configuration > files. Here is the log: > > whirr launch-cluster --config recipes/ec2_medium5.properties > Running on provider aws-ec2 using identity ABRACADABRA > Bootstrapping cluster > Configuring template for > bootstrap-hadoop-datanode_hadoop-tasktracker_ganglia-monitor > Configuring template for > > bootstrap-hadoop-namenode_hadoop-jobtracker_ganglia-monitor_ganglia-metad_pig-client > Starting 5 node(s) with roles [hadoop-datanode, hadoop-tasktracker, > ganglia-monitor] > Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker, > ganglia-monitor, ganglia-metad, pig-client] > Unexpected error while starting 5 nodes, minimum 5 nodes for > [hadoop-datanode, hadoop-tasktracker, ganglia-monitor] of cluster small5 > java.util.concurrent.ExecutionException: > org.jclouds.aws.AWSResponseException: request POST > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: > AWSError{requestId='c06fd438-015c-4929-a67e-9d5c70a82dc4', > requestToken='null', code='InstanceLimitExceeded', message='20 instance(s) > are already running.', context='{Response=, Errors=}'} > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > > org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.jclouds.aws.AWSResponseException: request POST > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: > AWSError{requestId='c06fd438-015c-4929-a67e-9d5c70a82dc4', > requestToken='null', code='InstanceLimitExceeded', message='20 instance(s) > are already running.', context='{Response=, Errors=}'} > at > > org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) > at > > org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > at > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > at > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > at > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > at $Proxy77.runInstancesInRegion(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > at $Proxy78.runInstancesInRegion(Unknown Source) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > at > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > at > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > at > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > at > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > at > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > at > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > at $Proxy77.runInstancesInRegion(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > at $Proxy78.runInstancesInRegion(Unknown Source) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > at > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > at > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > at > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > ... 5 more > Starting 5 node(s) with roles [hadoop-datanode, hadoop-tasktracker, > ganglia-monitor] > Unexpected error while starting 5 nodes, minimum 5 nodes for > [hadoop-datanode, hadoop-tasktracker, ganglia-monitor] of cluster small5 > java.util.concurrent.ExecutionException: > org.jclouds.aws.AWSResponseException: request POST > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: > AWSError{requestId='3f857b13-ce9b-4f67-9b15-1d1e64afa928', > requestToken='null', code='InstanceLimitExceeded', message='20 instance(s) > are already running.', context='{Response=, Errors=}'} > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > > org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.jclouds.aws.AWSResponseException: request POST > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: > AWSError{requestId='3f857b13-ce9b-4f67-9b15-1d1e64afa928', > requestToken='null', code='InstanceLimitExceeded', message='20 instance(s) > are already running.', context='{Response=, Errors=}'} > at > > org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) > at > > org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > at > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > at > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > at > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > at $Proxy77.runInstancesInRegion(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > at $Proxy78.runInstancesInRegion(Unknown Source) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > at > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > at > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > at > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > at > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > at > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > at > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > at > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > at > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > at $Proxy77.runInstancesInRegion(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > at $Proxy78.runInstancesInRegion(Unknown Source) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > at > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > at > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > at > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > at > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > ... 5 more > Unable to start the cluster. Terminating all nodes. > java.io.IOException: java.util.concurrent.ExecutionException: > java.io.IOException: Too many instance failed while bootstrapping! 0 > successfully started instances while 0 instances failed > at > > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125) > at > > org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:127) > at > > org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:138) > at > > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:114) > at > > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > at > > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > at org.apache.whirr.cli.Main.run(Main.java:69) > at org.apache.whirr.cli.Main.main(Main.java:102) > Caused by: java.util.concurrent.ExecutionException: java.io.IOException: > Too many instance failed while bootstrapping! 0 successfully started > instances while 0 instances failed > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120) > ... 7 more > Caused by: java.io.IOException: Too many instance failed while > bootstrapping! 0 successfully started instances while 0 instances failed > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:93) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Finished running destroy phase scripts on all cluster instances > Destroying small5 cluster > Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker, > ganglia-monitor, ganglia-metad, pig-client] > ^CException in thread "user thread 20" java.lang.IllegalStateException: > interrupted calling get() on [destroyNode(us-east-1/i-f9d8819f)], so could > not run listeners > at > > org.jclouds.concurrent.Futures$CallGetAndRunExecutionList.run(Futures.java:66) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > On Wed, May 23, 2012 at 4:10 PM, Andrii Vozniuk <and...@vozniuk.com <javascript:;>> > wrote: > > > Hi guys, > > > > I had a running EC2 cluster called "small5" of 5 m1.small instances. It > > was launched with whirr launch-cluster --configure > > recipes/small5.properties. > > > > I wanted to launch a cluster of 10 m1.small instances simultaneously. > > Thus, I created small10.properties based on small5.properties. > > Unfortunately, I forgot to change the whirr.cluster-name. > > > > When I tried to launch the small10 cluster, I've got exceptions and, what > > is more disappointing, the original small5 cluster was destroyed. > > > > Is there any possibility to inform user about launching a cluster with > the > > same name and to require a confirmation before destroying already running > > cluster? > > > > -- > > Best regards > > Andrii Vozniuk > > > > > > > > > -- > Best regards > Andrii Vozniuk > -- -- Andrei Savu / axemblr.com / Tools for Clouds
