Do you have the whirr.log from that attempt? A.
On Mon, Aug 5, 2013 at 8:34 PM, a b <[email protected]> wrote: > i built whirr in my own directory - i didn't change it (yet) - i just > check it out and tried to compile - you can see i had some memory issues > that i didn't notice right away: > > 1530 git clone git://git.apache.org/whirr.git > 1531 mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs > 1532 cd whirr/ > 1533 mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs > 1534 mvn install > 1535 cd > 1536 rm whirr.log > 1537 ~/git/whirr/bin/whirr launch-cluster --config > ~/whirr/recipes/hadoop.properties > 1538 export MAVEN_OPTS=-Xmx200m > 1539 cd ~/git/whirr/ > 1540 mvn install > 1541 export MAVEN_OPTS=-Xmx1G > 1542 mvn install > 1543 cd > 1544 ~/git/whirr/bin/whirr launch-cluster --config > ~/whirr/recipes/hadoop.properties > 1545 history > > this is the console at the end of "mvn install" > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Whirr Build Tools .......................... SUCCESS [2.519s] > [INFO] Whirr ............................................. SUCCESS [4.662s] > [INFO] Apache Whirr Core ................................. SUCCESS > [1:43.913s] > [INFO] Apache Whirr Cassandra ............................ SUCCESS > [10.705s] > [INFO] Apache Whirr Ganglia .............................. SUCCESS [3.683s] > [INFO] Apache Whirr Hadoop ............................... SUCCESS > [11.294s] > [INFO] Apache Whirr ZooKeeper ............................ SUCCESS [4.477s] > [INFO] Apache Whirr HBase ................................ SUCCESS [5.196s] > [INFO] Apache Whirr YARN ................................. SUCCESS [4.867s] > [INFO] Apache Whirr CDH .................................. SUCCESS [5.552s] > [INFO] Apache Whirr Mahout ............................... SUCCESS [2.452s] > [INFO] Apache Whirr Pig .................................. SUCCESS [2.393s] > [INFO] Apache Whirr ElasticSearch ........................ SUCCESS [4.145s] > [INFO] Apache Whirr Hama ................................. SUCCESS [2.841s] > [INFO] Apache Whirr Puppet ............................... SUCCESS [3.242s] > [INFO] Apache Whirr Chef ................................. SUCCESS > [12.153s] > [INFO] Apache Whirr Solr ................................. SUCCESS [4.997s] > [INFO] Apache Whirr Kerberos ............................. SUCCESS > [12.207s] > [INFO] Apache Whirr CLI .................................. SUCCESS > [14.940s] > [INFO] Apache Whirr Examples ............................. SUCCESS [4.494s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 3:43.355s > [INFO] Finished at: Mon Aug 05 20:11:58 PDT 2013 > [INFO] Final Memory: 109M/262M > [INFO] > ------------------------------------------------------------------------ > ab@ubuntu12-64:~/git/whirr$ cd > ab@ubuntu12-64:~$ ~/git/whirr/bin/whirr launch-cluster --config > ~/whirr/recipes/hadoop.properties > Running on provider aws-ec2 using identity XXXXXXXXXXXXXXXXXXX > Exception in thread "main" java.lang.RuntimeException: > java.io.IOException: java.util.concurrent.ExecutionException: > java.io.IOException: Too many instance failed while bootstrapping! 0 > successfully started instances while 0 instances failed > at > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:128) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > at org.apache.whirr.cli.Main.run(Main.java:69) > at org.apache.whirr.cli.Main.main(Main.java:102) > Caused by: java.io.IOException: java.util.concurrent.ExecutionException: > java.io.IOException: Too many instance failed while bootstrapping! 0 > successfully started instances while 0 instances failed > at > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125) > at > org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:131) > at > org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137) > at > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113) > ... 4 more > Caused by: java.util.concurrent.ExecutionException: java.io.IOException: > Too many instance failed while bootstrapping! 0 successfully started > instances while 0 instances failed > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120) > ... 7 more > Caused by: java.io.IOException: Too many instance failed while > bootstrapping! 0 successfully started instances while 0 instances failed > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:93) > at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > 2 instances were started - like i asked for - the 1st instance was > terminated almost immediately, the 2nd was left running for a while - > eventually, i hit ^c on the launch and terminated the 2nd instance from the > aws console. > > i need some more coaching. > > ------------------------------ > *From:* Andrew Bayer <[email protected]> > *To:* [email protected]; a b <[email protected]> > *Sent:* Monday, August 5, 2013 5:54 PM > > *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds > > Try not building as root - that can throw things off. > > A. > > On Mon, Aug 5, 2013 at 5:44 PM, a b <[email protected]> wrote: > > as for the java 7 problem - i found this suggestion: > > > http://stackoverflow.com/questions/13225149/how-to-install-jdk-7-on-ec2-cluster-via-whirr > > i tried to download whirr - as suggested here: > > https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute > > o i did: git clone ... > o i modified: core/src/main/resources/functions/... > o i did: mvn eclipse:eclipse ... > o i skipped: eclipse import > o i ran: mvn install > > it fails in the "mvn install" during test > > Tests in error: > > testActionIsExecutedOnAllRelevantNodes(org.apache.whirr.actions.StopServicesActionTest): > cluster-user != root or do not run as root > > testFilterScriptExecutionByRole(org.apache.whirr.actions.StopServicesActionTest): > cluster-user != root or do not run as root > > testFilterScriptExecutionByInstanceId(org.apache.whirr.actions.StopServicesActionTest): > cluster-user != root or do not run as root > > testFilterScriptExecutionByRoleAndInstanceId(org.apache.whirr.actions.StopServicesActionTest): > cluster-user != root or do not run as root > > testNoScriptExecutionsForNoop(org.apache.whirr.actions.StopServicesActionTest): > cluster-user != root or do not run as root > [..] > > testExecuteOnlyBootstrapForNoop(org.apache.whirr.service.DryRunModuleTest): > cluster-user != root or do not run as root > > testExecuteOnlyBootstrapForNoopWithListener(org.apache.whirr.service.DryRunModuleTest): > cluster-user != root or do not run as root > > testNoInitScriptsAfterConfigurationStartedAndNoConfigScriptsAfterDestroy(org.apache.whirr.service.DryRunModuleTest): > cluster-user != root or do not run as root > testOverrides(org.apache.whirr.command.AbstractClusterCommandTest): > cluster-user != root or do not run as root > > Tests run: 99, Failures: 0, Errors: 49, Skipped: 0 > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Whirr Build Tools .......................... SUCCESS [2.612s] > [INFO] Whirr ............................................. SUCCESS [4.488s] > [INFO] Apache Whirr Core ................................. FAILURE > [17.113s] > [INFO] Apache Whirr Cassandra ............................ SKIPPED > [INFO] Apache Whirr Ganglia .............................. SKIPPED > [INFO] Apache Whirr Hadoop ............................... SKIPPED > > is there a better way to try this suggestion? > > ------------------------------ > *From:* Andrew Bayer <[email protected]> > > *To:* a b <[email protected]> > *Cc:* "[email protected]" <[email protected]> > *Sent:* Monday, August 5, 2013 4:48 PM > > *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds > > Looks like it's the Oracle JDK7 download that's failing - > http://download.oracle.com/otn-pub/java/jdk/7/jdk-7-linux-x64.tar.gzisn't > actually there. I don't know if they actually have a consistent link > to get the JDK7 tarball regardless of version. I'd just use OpenJDK for > now, if you can. > > A. > > On Mon, Aug 5, 2013 at 4:33 PM, a b <[email protected]> wrote: > > Get:19 http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 > Packages [1,273 kB]^M > Get:20 http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 > Packages [4,786 kB]^M > Get:21 http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages > [1,274 kB]^M > Get:22 http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 > Packages [4,796 kB]^M > Get:23 http://us-east-1.ec2.archive.ubuntu.com precise/main > TranslationIndex [3,706 B]^M > Get:24 http://us-east-1.ec2.archive.ubuntu.com precise/universe > TranslationIndex [2,922 B]^M > Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main > Sources [412 kB]^M > Get:26 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe > Sources [93.1 kB]^M > Get:27 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 > Packages [672 kB]^M > Get:28 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe > amd64 Packages [210 kB]^M > Get:29 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 > Packages [692 kB]^M > Get:30 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe > i386 Packages [214 kB]^M > Get:31 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main > TranslationIndex [3,564 B]^M > Get:32 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe > TranslationIndex [2,850 B]^M > Get:33 http://us-east-1.ec2.archive.ubuntu.com precise/main > Translation-en [726 kB]^M > Get:34 http://us-east-1.ec2.archive.ubuntu.com precise/universe > Translation-en [3,341 kB]^M > Get:35 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main > Translation-en [298 kB]^M > Get:36 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe > Translation-en [123 kB]^M > Fetched 26.1 MB in 21s (1,241 kB/s)^M > Reading package lists...^M > Could not download > http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz.md5. > Continuing.^M > , error=^M > gzip: stdin: not in gzip format^M > tar: Child returned status 1^M > tar: Error is not recoverable: exiting now^M > mv: cannot stat `jdk1*': No such file or directory^M > update-alternatives: error: alternative path > /usr/lib/jvm/java-7-oracle/bin/java doesn't exist.^M > > ------------------------------ > *From:* Andrew Bayer <[email protected]> > *To:* [email protected]; a b <[email protected]> > *Sent:* Monday, August 5, 2013 4:22 PM > > *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds > > Ok, Whirr does try to download the md5 file, but if it fails to find it, > that's not a blocking error - it'll keep going anyway. What's after that in > the logs? > > A. > > On Mon, Aug 5, 2013 at 4:19 PM, a b <[email protected]> wrote: > > ok - i'm not sure what you are asking. > > *whirr launch-cluster --config ~/whirr/recipes/hadoop.properties* > > where these are the properties i think i changed or added from the > original recipe: > > *whirr.cluster-name=hadoop-ec2* > *whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1 > hadoop-datanode+hadoop-tasktracker* > *whirr.hardware-id=t1.micro* > ***whirr.image-id=us-east-1/ami-25d9a94c* > ***whirr.hadoop.version=1.2.1* > ***whirr.provider=aws-ec2*** > *whirr.identity=${env:AWS_ACCESS_KEY}* > *whirr.credential=${env:AWS_SECRET_KEY}* > *whirr.location-id=us-east-1* > *whirr.java.install-function=install_oracle_jdk7* > > ------------------------------ > *From:* Andrew Bayer <[email protected]> > *To:* [email protected]; a b <[email protected]> > *Sent:* Monday, August 5, 2013 4:11 PM > *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds > > Ok, it looks like they've actually been doing .mds for a while. Where are > you seeing this error? > > A. > > On Mon, Aug 5, 2013 at 4:09 PM, Andrew Bayer <[email protected]>wrote: > > That looks like they broke the hadoop-1.2.1 release - the file should be > .md5. I'd bug the Hadoop project about that. > > A. > > > On Mon, Aug 5, 2013 at 3:42 PM, a b <[email protected]> wrote: > > i get a whirr error: > > Could not download > http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz. > md5 > > when i browse over to: > http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/ > i can see the file is named: hadoop-1.2.1.tar.gz.mds > > how do i tell whirr to use a different suffix? > > > > > > > > > > > > > > >
