I am happy to hear that!! Ping the email list any time you have questions. On Sun, Dec 4, 2011 at 11:14 PM, Periya.Data <[email protected]> wrote:
> Hi Andrei, > Many thanks again. Will see those links and try to create a cluster > with the tools I need. > > Will be happy to contribute in any way I can. Will see the Wiki on how to > contribute and will proceed from there. > > /PD. > > > On Sun, Dec 4, 2011 at 1:10 PM, Andrei Savu <[email protected]> wrote: > >> The best way would be to define a new service for each one with new >> install / configure scripts. >> >> See this presentation about creating a new Whirr service: >> http://www.oscon.com/oscon2011/public/schedule/detail/19214 >> >> For Flume the following repo contains some working code: >> https://github.com/tomwhite/whirr-service-example >> >> For Hive we are tracking progress here: >> https://issues.apache.org/jira/browse/WHIRR-343 >> >> Currently there is no open issue for adding Sqoop as a service. >> >> I strongly recommend you to submit patches / pull requests as early as >> possible and I will give as >> much assistance as needed. See the following wiki page to learn more >> about how to contribute: >> https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute >> >> On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <[email protected]>wrote: >> >>> Thanks much. Will follow your suggestions. A quick question: >>> >>> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what >>> is the suggested way? >>> --> Log into each cluster node and install them manually? >>> --> Maybe the I have to hack the install script >>> (/whirr/services/cdh/src/main/resources/functions) or create a new one for >>> each of the tools? >>> >>> Thanks, >>> PD. >>> >>> >>> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <[email protected]>wrote: >>> >>>> Here are a few advices: >>>> >>>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS >>>> version we are testing against before making a release >>>> * whirr.hadoop.tarball.url is irrelevant if you specify >>>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version. >>>> * install_cdh_hadoop will install by default the latest release. >>>> We've implemented a way of selecting the version in 0.7.0: >>>> https://issues.apache.org/jira/browse/WHIRR-415 >>>> * use something bigger than t1.micro - this is just too small to run >>>> Hadoop >>>> >>>> Let me know if you are still having problems. >>>> >>>> -- Andrei Savu / andreisavu.ro >>>> >>>> >>>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <[email protected]>wrote: >>>> >>>>> Hi, >>>>> I am trying Whirr to spin-up EC2 clusters. Here are the specs: >>>>> >>>>> >>>>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2 >>>>> - EC2 Whirr config : >>>>> - >>>>> - whirr.hadoop.install-function=install_cdh_hadoop >>>>> - whirr.hadoop.configure-function=configure_cdh_hadoop >>>>> - whirr.hadoop.version=0.20.2 >>>>> - whirr.hadoop.tarball.url= >>>>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz >>>>> - whirr.hardware-id=t1.micro >>>>> # Ubuntu 11.10 Oneiric EBS boot from alestic >>>>> - whirr.image-id=us-east-1/ami-a7f539ce >>>>> - whirr.location-id=us-east-1 >>>>> >>>>> >>>>> *Problem:* The cluster seems to be up and running. But, cannot see >>>>> Java and Hadoop on the instance. >>>>> >>>>> *Question:* Is this a problem or should I clearly mention in my Whirr >>>>> config/properties file what version of JDK to be installed and its >>>>> download >>>>> URL? >>>>> >>>>> >>>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the >>>>> logs) >>>>> >>>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al >>>>> total 40 >>>>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 . >>>>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 .. >>>>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml >>>>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml >>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46 >>>>> jclouds-script-1322977554131 >>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs >>>>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml >>>>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh >>>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/ >>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l >>>>> total 12 >>>>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log >>>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log >>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log >>>>> E: Package 'sun-java6-jdk' has no installation candidate >>>>> /tmp/setup-sri.sh: line 161: java: command not found >>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>>> >>>>> ---------- >>>>> >>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> i386 Packages [57.2 kB] >>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> i386 Packages [4396 B] >>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>>> TranslationIndex [73 B] >>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> TranslationIndex [72 B >>>>> ] >>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> TranslationIndex [73 B] >>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>>>> Translation-en [701 kB] >>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>>>> Translation-en >>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>>>> Translation-en [3165 kB] >>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>>> Translation-en [99.2 kB] >>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> Translation-en [2419 B >>>>> ] >>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> Translation-en [35.4 kB] >>>>> Fetched 16.2 MB in 12s (1290 kB/s) >>>>> Reading package lists... >>>>> Reading package lists... >>>>> Building dependency tree... >>>>> Reading state information... >>>>> Package sun-java6-jdk is not available, but is referred to by another >>>>> package. >>>>> This may mean that the package is missing, has been obsoleted, or >>>>> is only available from another source >>>>> >>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> i386 Packages [57.2 kB] >>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> i386 Packages [4396 B] >>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>>> TranslationIndex [73 B] >>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> TranslationIndex [72 B] >>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> TranslationIndex [73 B] >>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>>>> Translation-en [701 kB] >>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>>>> Translation-en >>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>>>> Translation-en [3165 kB] >>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>>> Translation-en [99.2 kB] >>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>>> Translation-en [2419 B] >>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>>> Translation-en [35.4 kB] >>>>> Fetched 16.2 MB in 12s (1290 kB/s) >>>>> Reading package lists... >>>>> Reading package lists... >>>>> Building dependency tree... >>>>> Reading state information... >>>>> Package sun-java6-jdk is not available, but is referred to by another >>>>> package. >>>>> This may mean that the package is missing, has been obsoleted, or >>>>> is only available from another source >>>>> >>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>>> >>>>> *[On a different cluster spun-up at a later time]* >>>>> >>>>> sri@domU-12-31-39-10-6D-38:~$ java >>>>> The program 'java' can be found in the following packages: >>>>> * gcj-4.4-jre-headless >>>>> * gcj-4.6-jre-headless >>>>> * openjdk-6-jre-headless >>>>> * gcj-4.5-jre-headless >>>>> * openjdk-7-jre-headless >>>>> Ask your administrator to install one of them >>>>> sri@domU-12-31-39-10-6D-38:~$ java -version >>>>> The program 'java' can be found in the following packages: >>>>> * gcj-4.4-jre-headless >>>>> * gcj-4.6-jre-headless >>>>> * openjdk-6-jre-headless >>>>> * gcj-4.5-jre-headless >>>>> * openjdk-7-jre-headless >>>>> Ask your administrator to install one of them >>>>> sri@domU-12-31-39-10-6D-38:~$ >>>>> >>>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version >>>>> hadoop: command not found >>>>> sri@domU-12-31-39-10-6D-38:~$ >>>>> >>>>> =========================================== >>>>> >>>>> Finally, a quick Googling pointed to a JIRA issue: >>>>> https://issues.apache.org/jira/browse/WHIRR-435 >>>>> >>>>> >>>>> I am seeing the same stuff. Any updates? If this is a real issues, I >>>>> do not have any problems in using Ubuntu 11.04 AMI for now. >>>>> >>>>> >>>>> Your help is very much appreciated. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> PD. >>>>> >>>> >>>> >>> >> >
