My apologies for the spam. How embarassing! -- Philip
On Fri, May 25, 2012 at 4:10 PM, Philip Zeyliger <phi...@cloudera.com>wrote: > This is the guy who's been an intern for us (I think) and does a ton of > Whirr stuff. Always tempting to hire him, even though he's crazy remote. > phunt knows him. > > > ---------- Forwarded message ---------- > From: Andrei Savu <savu.and...@gmail.com> > Date: Wed, May 23, 2012 at 8:01 AM > Subject: Re: Running cluster is destroyed when launching cluster with the > same name > To: "dev@whirr.apache.org" <dev@whirr.apache.org> > > > This behaviour is expected for the current implementation. > > It's easy to add a naive check using list-cluster but that's only effective > if multiple cluster > having the same name are *not* started at the same time. > > We can also implement this by using a file lock but this assumes that for a > given cloud > provider account we are using a single machine to launch clusters. > > We had a similar discussion in the past. This is the related issue: > https://issues.apache.org/jira/browse/WHIRR-329 > > Do you want to give it a try to provide a patch? > > -- Andrei Savu > > On Wednesday, May 23, 2012, Andrii Vozniuk wrote: > > > Some clarification. > > > > Indeed, I wanted to launch a cluster of 5 m1.medium instances > > simultaneously with running cluster of 5 m1.small instances. By mistake, > > the whirr.cluster-name value was the same in corresponding configuration > > files. Here is the log: > > > > whirr launch-cluster --config recipes/ec2_medium5.properties > > Running on provider aws-ec2 using identity ABRACADABRA > > Bootstrapping cluster > > Configuring template for > > bootstrap-hadoop-datanode_hadoop-tasktracker_ganglia-monitor > > Configuring template for > > > > > bootstrap-hadoop-namenode_hadoop-jobtracker_ganglia-monitor_ganglia-metad_pig-client > > Starting 5 node(s) with roles [hadoop-datanode, hadoop-tasktracker, > > ganglia-monitor] > > Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker, > > ganglia-monitor, ganglia-metad, pig-client] > > Unexpected error while starting 5 nodes, minimum 5 nodes for > > [hadoop-datanode, hadoop-tasktracker, ganglia-monitor] of cluster small5 > > java.util.concurrent.ExecutionException: > > org.jclouds.aws.AWSResponseException: request POST > > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, > error: > > AWSError{requestId='c06fd438-015c-4929-a67e-9d5c70a82dc4', > > requestToken='null', code='InstanceLimitExceeded', message='20 > instance(s) > > are already running.', context='{Response=, Errors=}'} > > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > > at > > > > > org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130) > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83) > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > Caused by: org.jclouds.aws.AWSResponseException: request POST > > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, > error: > > AWSError{requestId='c06fd438-015c-4929-a67e-9d5c70a82dc4', > > requestToken='null', code='InstanceLimitExceeded', message='20 > instance(s) > > are already running.', context='{Response=, Errors=}'} > > at > > > > > org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) > > at > > > > > org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > at > > > > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > > at > > > > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > > at > > > > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > > at $Proxy77.runInstancesInRegion(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > > at $Proxy78.runInstancesInRegion(Unknown Source) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > > at > > > > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > > at > > > > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > > at > > > > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > at > > > > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > > at > > > > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > > at > > > > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > > at $Proxy77.runInstancesInRegion(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > > at $Proxy78.runInstancesInRegion(Unknown Source) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > > at > > > > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > > at > > > > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > > at > > > > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > > ... 5 more > > Starting 5 node(s) with roles [hadoop-datanode, hadoop-tasktracker, > > ganglia-monitor] > > Unexpected error while starting 5 nodes, minimum 5 nodes for > > [hadoop-datanode, hadoop-tasktracker, ganglia-monitor] of cluster small5 > > java.util.concurrent.ExecutionException: > > org.jclouds.aws.AWSResponseException: request POST > > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, > error: > > AWSError{requestId='3f857b13-ce9b-4f67-9b15-1d1e64afa928', > > requestToken='null', code='InstanceLimitExceeded', message='20 > instance(s) > > are already running.', context='{Response=, Errors=}'} > > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > > at > > > > > org.apache.whirr.compute.StartupProcess.waitForOutcomes(StartupProcess.java:130) > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:83) > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > Caused by: org.jclouds.aws.AWSResponseException: request POST > > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, > error: > > AWSError{requestId='3f857b13-ce9b-4f67-9b15-1d1e64afa928', > > requestToken='null', code='InstanceLimitExceeded', message='20 > instance(s) > > are already running.', context='{Response=, Errors=}'} > > at > > > > > org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) > > at > > > > > org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > at > > > > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > > at > > > > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > > at > > > > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > > at $Proxy77.runInstancesInRegion(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > > at $Proxy78.runInstancesInRegion(Unknown Source) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > > at > > > > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > > at > > > > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > > at > > > > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > at > > > > > org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184) > > at > > > > > org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127) > > at > > > > > org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54) > > at > > > > > org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198) > > at > > > > > org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134) > > at $Proxy77.runInstancesInRegion(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137) > > at $Proxy78.runInstancesInRegion(Unknown Source) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(EC2CreateNodesInGroupThenAddToSet.java:232) > > at > > > > > org.jclouds.aws.ec2.compute.strategy.AWSEC2CreateNodesInGroupThenAddToSet.createNodesInRegionAndZone(AWSEC2CreateNodesInGroupThenAddToSet.java:116) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.createKeyPairAndSecurityGroupsAsNeededThenRunInstances(EC2CreateNodesInGroupThenAddToSet.java:218) > > at > > > > > org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:140) > > at > > > > > org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200) > > at > > > > > org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:55) > > at org.apache.whirr.compute.NodeStarter.call(NodeStarter.java:30) > > ... 5 more > > Unable to start the cluster. Terminating all nodes. > > java.io.IOException: java.util.concurrent.ExecutionException: > > java.io.IOException: Too many instance failed while bootstrapping! 0 > > successfully started instances while 0 instances failed > > at > > > > > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125) > > at > > > > > org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:127) > > at > > > > > org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:138) > > at > > > > > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:114) > > at > > > > > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > > at > > > > > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > > at org.apache.whirr.cli.Main.run(Main.java:69) > > at org.apache.whirr.cli.Main.main(Main.java:102) > > Caused by: java.util.concurrent.ExecutionException: java.io.IOException: > > Too many instance failed while bootstrapping! 0 successfully started > > instances while 0 instances failed > > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > > at > > > > > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120) > > ... 7 more > > Caused by: java.io.IOException: Too many instance failed while > > bootstrapping! 0 successfully started instances while 0 instances failed > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:93) > > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > Finished running destroy phase scripts on all cluster instances > > Destroying small5 cluster > > Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker, > > ganglia-monitor, ganglia-metad, pig-client] > > ^CException in thread "user thread 20" java.lang.IllegalStateException: > > interrupted calling get() on [destroyNode(us-east-1/i-f9d8819f)], so > could > > not run listeners > > at > > > > > org.jclouds.concurrent.Futures$CallGetAndRunExecutionList.run(Futures.java:66) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > > > On Wed, May 23, 2012 at 4:10 PM, Andrii Vozniuk <and...@vozniuk.com > <javascript:;>> > > wrote: > > > > > Hi guys, > > > > > > I had a running EC2 cluster called "small5" of 5 m1.small instances. It > > > was launched with whirr launch-cluster --configure > > > recipes/small5.properties. > > > > > > I wanted to launch a cluster of 10 m1.small instances simultaneously. > > > Thus, I created small10.properties based on small5.properties. > > > Unfortunately, I forgot to change the whirr.cluster-name. > > > > > > When I tried to launch the small10 cluster, I've got exceptions and, > what > > > is more disappointing, the original small5 cluster was destroyed. > > > > > > Is there any possibility to inform user about launching a cluster with > > the > > > same name and to require a confirmation before destroying already > running > > > cluster? > > > > > > -- > > > Best regards > > > Andrii Vozniuk > > > > > > > > > > > > > > > -- > > Best regards > > Andrii Vozniuk > > > > > -- > -- Andrei Savu / axemblr.com / Tools for Clouds > >
