The best way would be to define a new service for each one with new install / configure scripts.
See this presentation about creating a new Whirr service: http://www.oscon.com/oscon2011/public/schedule/detail/19214 For Flume the following repo contains some working code: https://github.com/tomwhite/whirr-service-example For Hive we are tracking progress here: https://issues.apache.org/jira/browse/WHIRR-343 Currently there is no open issue for adding Sqoop as a service. I strongly recommend you to submit patches / pull requests as early as possible and I will give as much assistance as needed. See the following wiki page to learn more about how to contribute: https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <[email protected]> wrote: > Thanks much. Will follow your suggestions. A quick question: > > If I have to install Hive, Sqoop and Flume on the my cluster nodes, what > is the suggested way? > --> Log into each cluster node and install them manually? > --> Maybe the I have to hack the install script > (/whirr/services/cdh/src/main/resources/functions) or create a new one for > each of the tools? > > Thanks, > PD. > > > On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <[email protected]>wrote: > >> Here are a few advices: >> >> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS >> version we are testing against before making a release >> * whirr.hadoop.tarball.url is irrelevant if you specify >> whirr.hadoop.install-function. Same is true for whirr.hadoop.version. >> * install_cdh_hadoop will install by default the latest release. >> We've implemented a way of selecting the version in 0.7.0: >> https://issues.apache.org/jira/browse/WHIRR-415 >> * use something bigger than t1.micro - this is just too small to run >> Hadoop >> >> Let me know if you are still having problems. >> >> -- Andrei Savu / andreisavu.ro >> >> >> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <[email protected]>wrote: >> >>> Hi, >>> I am trying Whirr to spin-up EC2 clusters. Here are the specs: >>> >>> >>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2 >>> - EC2 Whirr config : >>> - >>> - whirr.hadoop.install-function=install_cdh_hadoop >>> - whirr.hadoop.configure-function=configure_cdh_hadoop >>> - whirr.hadoop.version=0.20.2 >>> - whirr.hadoop.tarball.url= >>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz >>> - whirr.hardware-id=t1.micro >>> # Ubuntu 11.10 Oneiric EBS boot from alestic >>> - whirr.image-id=us-east-1/ami-a7f539ce >>> - whirr.location-id=us-east-1 >>> >>> >>> *Problem:* The cluster seems to be up and running. But, cannot see Java >>> and Hadoop on the instance. >>> >>> *Question:* Is this a problem or should I clearly mention in my Whirr >>> config/properties file what version of JDK to be installed and its download >>> URL? >>> >>> >>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs) >>> >>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al >>> total 40 >>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 . >>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 .. >>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml >>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml >>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46 >>> jclouds-script-1322977554131 >>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs >>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml >>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh >>> sri@domU-12-31-39-02-21-20:/tmp$ >>> sri@domU-12-31-39-02-21-20:/tmp$ >>> sri@domU-12-31-39-02-21-20:/tmp$ >>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/ >>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l >>> total 12 >>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log >>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log >>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log >>> E: Package 'sun-java6-jdk' has no installation candidate >>> /tmp/setup-sri.sh: line 161: java: command not found >>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>> >>> ---------- >>> >>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> i386 Packages [57.2 kB] >>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> i386 Packages [4396 B] >>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>> TranslationIndex [73 B] >>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> TranslationIndex [72 B >>> ] >>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> TranslationIndex [73 B] >>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>> Translation-en [701 kB] >>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>> Translation-en >>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>> Translation-en [3165 kB] >>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>> Translation-en [99.2 kB] >>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> Translation-en [2419 B >>> ] >>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> Translation-en [35.4 kB] >>> Fetched 16.2 MB in 12s (1290 kB/s) >>> Reading package lists... >>> Reading package lists... >>> Building dependency tree... >>> Reading state information... >>> Package sun-java6-jdk is not available, but is referred to by another >>> package. >>> This may mean that the package is missing, has been obsoleted, or >>> is only available from another source >>> >>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> i386 Packages [57.2 kB] >>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> i386 Packages [4396 B] >>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>> TranslationIndex [73 B] >>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> TranslationIndex [72 B] >>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> TranslationIndex [73 B] >>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>> Translation-en [701 kB] >>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>> Translation-en >>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>> Translation-en [3165 kB] >>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>> Translation-en [99.2 kB] >>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>> Translation-en [2419 B] >>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe >>> Translation-en [35.4 kB] >>> Fetched 16.2 MB in 12s (1290 kB/s) >>> Reading package lists... >>> Reading package lists... >>> Building dependency tree... >>> Reading state information... >>> Package sun-java6-jdk is not available, but is referred to by another >>> package. >>> This may mean that the package is missing, has been obsoleted, or >>> is only available from another source >>> >>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>> >>> *[On a different cluster spun-up at a later time]* >>> >>> sri@domU-12-31-39-10-6D-38:~$ java >>> The program 'java' can be found in the following packages: >>> * gcj-4.4-jre-headless >>> * gcj-4.6-jre-headless >>> * openjdk-6-jre-headless >>> * gcj-4.5-jre-headless >>> * openjdk-7-jre-headless >>> Ask your administrator to install one of them >>> sri@domU-12-31-39-10-6D-38:~$ java -version >>> The program 'java' can be found in the following packages: >>> * gcj-4.4-jre-headless >>> * gcj-4.6-jre-headless >>> * openjdk-6-jre-headless >>> * gcj-4.5-jre-headless >>> * openjdk-7-jre-headless >>> Ask your administrator to install one of them >>> sri@domU-12-31-39-10-6D-38:~$ >>> >>> sri@domU-12-31-39-10-6D-38:~$ hadoop version >>> hadoop: command not found >>> sri@domU-12-31-39-10-6D-38:~$ >>> >>> =========================================== >>> >>> Finally, a quick Googling pointed to a JIRA issue: >>> https://issues.apache.org/jira/browse/WHIRR-435 >>> >>> >>> I am seeing the same stuff. Any updates? If this is a real issues, I do >>> not have any problems in using Ubuntu 11.04 AMI for now. >>> >>> >>> Your help is very much appreciated. >>> >>> >>> Thanks, >>> >>> PD. >>> >> >> >
