Hi Everybody, I am trying to get my first hadoop cluster started using the Amazon EC2. I tried quite a few times and searched the web for the solutions, yet I still cannot get it up. I hope somebody can help out here.
Here is what I did based on the Apache Whirr Quick Guide ( http://whirr.apache.org/docs/0.8.1/quick-start-guide.html): 1) I downloaded a Whirr tar ball and installed it. bin/whirr version shows the following: Apache Whirr 0.8.2 & jclouds 1.5.8 2) I created the ./whirr directory and edit the credential file with my Amazon "PROVIDER", "IDENTITY" and "CREDENTIAL" "IDENTITY=AAAAAASSSSSSSSS", with no extra quotes or curly quotes around the actual key_id 3) I used the following command to creat the key pair for whirr and stored it at the folder .ssh ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr 4) I think I am ready to use one of the properties file provided with whirr in the recipes folder. Here is the command I ran: bin/whirr launch-cluster --config recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whi The command ran into the error and did not bring up the hadoop. My questin is: Do we need to change anything the default properties provided in the recipes folder in the "whirr-0.8.2" folder, such as the "hadoop-yarn-ec2.properties " I used? Here are the error messages: --------------------------------------------------- [ec2-user@ip-172-31-20-120 whirr-0.8.2]$ bin/whirr launch-cluster --config recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr Running on provider aws-ec2 using identity AKIAJLFVRARQ3IZE3KGF Unable to start the cluster. Terminating all nodes. com.google.common.util.concurrent.UncheckedExecutionException: com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified. 1 error at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884) at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88) at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80) at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110) at org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137) at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) at org.apache.whirr.cli.Main.run(Main.java:69) at org.apache.whirr.cli.Main.main(Main.java:102) Caused by: com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified. 1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) at com.google.inject.Guice.createInjector(Guice.java:95) at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401) at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325) at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600) at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580) at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119) at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 13 more Unable to load cluster state, assuming it has no running nodes. java.io.FileNotFoundException: /home/ec2-user/.whirr/hadoop-yarn/instances (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:140) at com.google.common.io.Files$1.getInput(Files.java:109) at com.google.common.io.Files$1.getInput(Files.java:106) at com.google.common.io.CharStreams$2.getInput(CharStreams.java:93) at com.google.common.io.CharStreams$2.getInput(CharStreams.java:90) at com.google.common.io.CharStreams.readLines(CharStreams.java:307) at com.google.common.io.Files.readLines(Files.java:543) at org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:55) at org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:59) at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226) at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) at org.apache.whirr.cli.Main.run(Main.java:69) at org.apache.whirr.cli.Main.main(Main.java:102) Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified. 1 error at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884) at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88) at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80) at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110) at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:233) at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226) at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) at org.apache.whirr.cli.Main.run(Main.java:69) at org.apache.whirr.cli.Main.main(Main.java:102) Caused by: com.google.inject.CreationException: Guice creation errors: 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified. 1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) at com.google.inject.Guice.createInjector(Guice.java:95) at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401) at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325) at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600) at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580) at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119) at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) I would really appreciate if somebody can help pinpoint my problem here. Thanks Max
