Download from a nearby mirror http://www.apache.org/dyn/closer.cgi/whirr/
untar, you should be good to go. I would recommend the following steps 1. Use existing recipe for launching a cluster 2. Customize the existing recipe according to your needs HTH ashish On Thu, Apr 19, 2012 at 3:45 AM, Deb Ghosh <[email protected]> wrote: > Hi Ashish, > > Just wanted to clear a little more on this .. > > So on my local ubuntu machine I will install whirr 0.7.1 . In that case from > which site I should download this for ubuntu 11.10. > Any clue is appreciated. > > Thanks in advance > Debashis > > On Tue, Apr 17, 2012 at 5:41 PM, Ashish <[email protected]> wrote: >> >> I hope your are using whirr 0.7.1 >> whirr installs jdk for you, with 0.7.1 the default is openjdk. >> >> To avoid this JDK issue, I used a custom AMI that has JDK >> pre-installed, and in whirr receipe you can specify the Java Home >> variable to be used. >> >> >> Andrei/Karel - thoughts on this? >> >> cheers >> ashish >> >> On Wed, Apr 18, 2012 at 2:09 AM, Deb Ghosh <[email protected]> wrote: >> > Hi Ashish, >> > >> > First of all many thanks to you time and response , still not able to >> > however I have ssh to the remote and found the following ...seems to be >> > java >> > not installed on the remote ! >> > >> > I have in the remote >> > ubuntu@ip-10-140-10-242:/tmp/runscript$ ls -l >> > total 28 >> > -rwxr--r-- 1 ubuntu ubuntu 764 2012-04-17 20:10 runscript.sh >> > -rw-r--r-- 1 root root 4111 2012-04-17 20:10 stderr.log >> > -rw-r--r-- 1 root root 12683 2012-04-17 20:10 stdout.log >> > ====== >> > >> > 1) in stderr.log at the tail part I have the following >> > >> > ++ echo nn,jt >> > ++ tr , '\n' >> > + for role in '$(echo "$ROLES" | tr "," "\n")' >> > + case $role in >> > + start_namenode >> > + which dpkg >> > + apt-get -y install hadoop-0.20-namenode >> > dpkg-preconfigure: unable to re-open stdin: >> > update-rc.d: warning: hadoop-0.20-namenode start runlevel arguments (2 3 >> > 4 >> > 5) do not match LSB Default-Start values (3 5) >> > update-rc.d: warning: hadoop-0.20-namenode stop runlevel arguments (0 1 >> > 6) >> > do not match LSB Default-Stop values (0 1 2 4 6) >> > + AS_HDFS='su -s /bin/bash - hdfs -c' >> > + '[' '!' -e /mnt/hadoop/hdfs ']' >> > + su -s /bin/bash - hdfs -c 'hadoop-0.20 namenode -format' >> > +======================================================================+ >> > | Error: JAVA_HOME is not set and Java could not be found | >> > +----------------------------------------------------------------------+ >> > | Please download the latest Sun JDK from the Sun Java web site | >> > | > http://java.sun.com/javase/downloads/ < | >> > | | >> > | Hadoop requires Java 1.6 or later. | >> > | NOTE: This script will find Sun Java whether you install using the | >> > | binary or the RPM based installer. | >> > +======================================================================+ >> > >> > >> > 2) and on the tail part of ubuntu@ip-10-140-10-242:/tmp/runscript$ vi >> > stdout.log >> > >> > I have >> > >> > Setting up hadoop-0.20 (0.20.2+923.197-1~lucid-cdh3) ... >> > update-alternatives: using /etc/hadoop-0.20/conf.empty to provide >> > /etc/hadoop-0.20/conf (hadoop-0.20-conf) in auto mode. >> > update-alternatives: using /usr/bin/hadoop-0.20 to provide >> > /usr/bin/hadoop >> > (hadoop-default) in auto mode. >> > >> > Setting up hadoop-0.20-native (0.20.2+923.197-1~lucid-cdh3) ... >> > >> > Processing triggers for libc-bin ... >> > ldconfig deferred processing now taking place >> > update-alternatives: using /etc/hadoop-0.20/conf.dist to provide >> > /etc/hadoop-0.20/conf (hadoop-0.20-conf) in auto mode. >> > Reading package lists... >> > Building dependency tree... >> > Reading state information... >> > The following NEW packages will be installed: >> > hadoop-0.20-namenode >> > 0 upgraded, 1 newly installed, 0 to remove and 122 not upgraded. >> > Need to get 248kB of archives. >> > After this operation, 324kB of additional disk space will be used. >> > Get:1 http://archive.cloudera.com/debian/ lucid-cdh3/contrib >> > hadoop-0.20-namenode 0.20.2+923.197-1~lucid-cdh3 [248kB] >> > Fetched 248kB in 0s (14.8MB/s) >> > Selecting previously deselected package hadoop-0.20-namenode. >> > (Reading database ... 24766 files and directories currently installed.) >> > Unpacking hadoop-0.20-namenode (from >> > .../hadoop-0.20-namenode_0.20.2+923.197-1~lucid-cdh3_all.deb) ... >> > >> > ========================= >> > >> > SO do i need to install java on the remote machine ...bit confused here. >> > >> > Your time is appreciated. >> > >> > Thanks >> > Debashis >> > >> > >> > >> > >> > >> > >> > >> > On Sat, Apr 14, 2012 at 10:20 PM, Ashish <[email protected]> >> > wrote: >> >> >> >> Debashish, >> >> >> >> Can you check in the logs that services got started. You can do this >> >> scanning the whirr.log or by logging into the machines and verifying >> >> it. If something is missing, you can check /tmp/log onto the ec2 >> >> machine, to get an insight into what went wrong. >> >> >> >> I would do the following >> >> 1. Verify everything went fine in whirr.log >> >> 2. Log into the machine and check all services are running (using ps or >> >> jps) >> >> 3. If all is working fine, then you should get the UI. >> >> >> >> If nothing works out, I would use a existing recipe from >> >> whirr_install/recipes and try. This would work as it is battle tested. >> >> >> >> You can also join #whirr on IRC, a lot of folks are hanging out there >> >> and might help you in real time :) >> >> >> >> HTH ! >> >> ashish >> >> >> >> On Fri, Apr 13, 2012 at 2:43 AM, Deb Ghosh <[email protected]> >> >> wrote: >> >> > Hi , >> >> > The problem is I am in Ubuntu 11.10 as my os and after launching >> >> > hadoop >> >> > ec2 >> >> > cluster and running the proxyset up.sh as shown below after ==== , I >> >> > am >> >> > trying to http in the firefox bowser for the nodes like name node >> >> > with >> >> > the >> >> > http://ec2-23-20-228-116.compute-1.amazonaws.com:50070/ – web UI for >> >> > HDFS >> >> > name node(s ) but it does not connect >> >> > >> >> > But when i use hadoop single node cluster in hadoop yahoo in the same >> >> > os >> >> > and >> >> > use the web ui loacal host it goes to to the webui in the firefox >> >> > http://localhost:50070/ – web UI for HDFS name node(s) >> >> > >> >> > ============= >> >> > >> >> > Completed launch of myhadoopcluster >> >> > Web UI available at http://ec2-23-20-228-116.compute-1.amazonaws.com >> >> > Wrote Hadoop site file >> >> > /home/debashig/.whirr/myhadoopcluster/hadoop-site.xml >> >> > Wrote Hadoop proxy script >> >> > /home/debashig/.whirr/myhadoopcluster/hadoop-proxy.sh >> >> > Started cluster of 2 instances >> >> > HadoopCluster{instances=[Instance{roles=[jt, nn], >> >> > >> >> > publicAddress=ec2-23-20-228-116.compute-1.amazonaws.com/23.20.228.116, >> >> > privateAddress=/10.64.74.23}, Instance{roles=[tt, dn], >> >> > publicAddress=/50.17.54.86, privateAddress=/10.204.74.58}], >> >> > >> >> > >> >> > configuration={fs.default.name=hdfs://ec2-23-20-228-116.compute-1.amazonaws.com:8020/, >> >> > mapred.job.tracker=ec2-23-20-228-116.compute-1.amazonaws.com:8021, >> >> > hadoop.job.ugi=root,root, >> >> > >> >> > >> >> > hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory, >> >> > hadoop.socks.server=localhost:6666}} >> >> > debashig@ubuntu:~/amazon/Ec2_basic_setup/cloudera/whirr-0.1.0+23$ sh >> >> > ~/.whirr/myhadoopcluster/hadoop-proxy.sh >> >> > Running proxy to Hadoop cluster at >> >> > ec2-23-20-228-116.compute-1.amazonaws.com. Use Ctrl-c to quit. >> >> > Warning: Permanently added >> >> > 'ec2-23-20-228-116.compute-1.amazonaws.com,23.20.228.116' (RSA) to >> >> > the >> >> > list >> >> > of known hosts. >> >> > ===================================== >> >> > >> >> > Please provide your toughts or solution if any. >> >> > >> >> > Thanks >> >> > Debashis >> >> > >> >> > >> >> > On Tue, Apr 10, 2012 at 8:19 PM, Ashish <[email protected]> >> >> > wrote: >> >> >> >> >> >> The way I do this is, pick up the Name Node and Job Tracker URLs >> >> >> from >> >> >> the whirr logs or console and punch them in on browser without any >> >> >> change, and it works. >> >> >> >> >> >> I hope this is what you are trying to achieve. >> >> >> >> >> >> >> >> >> On Wed, Apr 11, 2012 at 8:35 AM, Deb Ghosh <[email protected]> >> >> >> wrote: >> >> >> > >> >> >> > Hello , >> >> >> > >> >> >> > Would appreciate a help on the following issue :- >> >> >> > >> >> >> > I was installing whirr on Amazon ec2 cluster the launch of whirr >> >> >> > using >> >> >> > my >> >> >> > ubuntu 11.10 was ok >> >> >> > >> >> >> > then we did the following to run the proxy server >> >> >> > >> >> >> > sh ~/.whirr/myhadoopcluster/hadoop-proxy.sh >> >> >> > >> >> >> > Running proxy to Hadoop cluster at >> >> >> > ec2-72-44-45-199.compute-1.amazonaws.com. >> >> >> > Use Ctrl-c to quit. >> >> >> > >> >> >> > >> >> >> > The hadoop-proxy.sh is used to access the web interface of Hadoop >> >> >> > securely. >> >> >> > When we run this it will tunnel through to the cluster and give us >> >> >> > access in >> >> >> > the web browser via a SOCKS proxy. For this we changed the >> >> >> > internet >> >> >> > option >> >> >> > for proxy and after that when we tried to access following jsp >> >> >> > page >> >> >> > >> >> >> > http://<hostname>:50070/dfshealth.jsp this was not working >> >> >> > >> >> >> > >> >> >> > (here the hostname was real DNS address on Amazon ec2). >> >> >> > >> >> >> > It is waiting for sometime and then failing to connect to amazon >> >> >> > server. >> >> >> > >> >> >> > >> >> >> > Please let me know - How I need to resolve this asap. >> >> >> > >> >> >> > >> >> >> > Thanks in advance >> >> >> > Debashis ( mobile # 5103662639) >> >> >> > >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> thanks >> >> >> ashish >> >> >> >> >> >> Blog: http://www.ashishpaliwal.com/blog >> >> >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> thanks >> >> ashish >> >> >> >> Blog: http://www.ashishpaliwal.com/blog >> >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > >> > >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
