Thanks Andrei. Now the cluster is up. However, I have one more problem left.
I am not able to use hadoop commands. I ran the hddoop-proxy.sh and set HADOOP_CONF_DIR as mentioned on the quick start guide page. Do I need to do anything more? Please see the logs: hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ . ~/.whirr/humaninference_rackspace/hadoop-proxy.sh & [1] 11179 hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ Running proxy to Hadoop cluster at 184-106-176-156.static.cloud-ips.com. Use Ctrl-c to quit. Warning: Permanently added '184-106-176-156.static.cloud-ips.com,184.106.176.156' (RSA) to the list of known hosts. hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ hadoop bash: hadoop: command not found hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ export HADOOP_CONF_DIR=~/.whirr/humaninference_rackspace hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ hadoop fs -ls / bash: hadoop: command not found ________________________________ From: Andrei Savu [[email protected]] Sent: Thursday, June 21, 2012 1:56 PM To: [email protected] Subject: Re: Whirr 0.7.1 : CDH : Rackspace : Problem Try replacing: whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker with: whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1 hadoop-datanode+hadoop-tasktracker The role order is important. -- Andrei Savu / axemblr.com<http://axemblr.com/> / Tools for Clouds On Thu, Jun 21, 2012 at 2:52 PM, Paritosh Ranjan <[email protected]<mailto:[email protected]>> wrote: Sometimes the namenode is up but the jobtracker has never been up. I have tried it 4-5 times. ________________________________ From: Andrei Savu [[email protected]<mailto:[email protected]>] Sent: Thursday, June 21, 2012 1:49 PM To: [email protected]<mailto:[email protected]> Subject: Re: Whirr 0.7.1 : CDH : Rackspace : Problem Have you checked the servers? I see no serious error. The file permissions issue should be harmless. By default Whirr is deploying CDH3U4. Is this what you need? -- Andrei Savu / axemblr.com<http://axemblr.com/> / Tools for Clouds On Thu, Jun 21, 2012 at 2:43 PM, Paritosh Ranjan <[email protected]<mailto:[email protected]>> wrote: This is the complete log, if it can help. ./whirr launch-cluster --config hadoop.properties Bootstrapping cluster Configuring template Configuring template Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker] Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode] Nodes started: [[id=20933521, providerId=20933521, group=humaninferencerackspace, name=humaninferencerackspace-af5, location=[id=e90923cb40144ee405a900686d6cd631, scope=HOST, description=e90923cb40144ee405a900686d6cd631, parent=cloudservers-us, iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu 10.04 LTS], state=RUNNING, loginPort=22, hostname=humaninferencerackspace-af5, privateAddresses=[10.183.34.9], publicAddresses=[50.57.166.152], hardware=[id=4, providerId=4, name=2GB server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, tags=[]]] Nodes started: [[id=20933520, providerId=20933520, group=humaninferencerackspace, name=humaninferencerackspace-d4b, location=[id=c948bbc4457a5a9a1d135260d0f5ec01, scope=HOST, description=c948bbc4457a5a9a1d135260d0f5ec01, parent=cloudservers-us, iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu 10.04 LTS], state=RUNNING, loginPort=22, hostname=humaninferencerackspace-d4b, privateAddresses=[10.183.34.3], publicAddresses=[50.57.166.89], hardware=[id=4, providerId=4, name=2GB server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, tags=[]]] Authorizing firewall ingress to [20933520] on ports [50030] for [108.171.174.174/32<http://108.171.174.174/32>] Authorizing firewall ingress to [20933520] on ports [8021] for [50.57.166.89/32<http://50.57.166.89/32>] Authorizing firewall ingress to [20933520] on ports [50070] for [108.171.174.174/32<http://108.171.174.174/32>] Authorizing firewall ingress to [20933520] on ports [8020, 8021] for [50.57.166.89/32<http://50.57.166.89/32>] Authorizing firewall ingress to [20933520] on ports [50030] for [108.171.174.174/32<http://108.171.174.174/32>] Authorizing firewall ingress to [20933520] on ports [8021] for [50.57.166.89/32<http://50.57.166.89/32>] Starting to run scripts on cluster for phase configureinstances: 20933521 Starting to run scripts on cluster for phase configureinstances: 20933520 Running configure phase script on: 20933521 Running configure phase script on: 20933520 configure phase script run completed on: 20933521 configure phase script run completed on: 20933520 Successfully executed configure script: [output= at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:207) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) hadoop-0.20-namenode. Safe mode is OFF , error=12/06/21 10:58:38 INFO namenode.FSNamesystem: supergroup=supergroup 12/06/21 10:58:38 INFO namenode.FSNamesystem: isPermissionEnabled=true 12/06/21 10:58:38 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=1000 12/06/21 10:58:38 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 12/06/21 10:58:38 INFO common.Storage: Image file of size 110 saved in 0 seconds. 12/06/21 10:58:38 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 12/06/21 10:58:38 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at 50-57-166-89.static.cloud-ips.com/50.57.166.89<http://50-57-166-89.static.cloud-ips.com/50.57.166.89> ************************************************************/ , exitCode=0] Successfully executed configure script: [output=java.io.FileNotFoundException: /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:207) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) hadoop-0.20-tasktracker. , error=dpkg-preconfigure: unable to re-open stdin: update-rc.d: warning: hadoop-0.20-datanode start runlevel arguments (2 3 4 5) do not match LSB Default-Start values (3 5) update-rc.d: warning: hadoop-0.20-datanode stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (0 1 2 4 6) dpkg-preconfigure: unable to re-open stdin: update-rc.d: warning: hadoop-0.20-tasktracker start runlevel arguments (2 3 4 5) do not match LSB Default-Start values (3 5) update-rc.d: warning: hadoop-0.20-tasktracker stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (0 1 2 4 6) , exitCode=0] Finished running configure phase scripts on all cluster instances Completed configuration of humaninference_rackspace role hadoop-jobtracker Jobtracker web UI available at http://50-57-166-89.static.cloud-ips.com:50030 Completed configuration of humaninference_rackspace role hadoop-namenode Namenode web UI available at http://50-57-166-89.static.cloud-ips.com:50070 Wrote Hadoop site file /home/hadoepje/.whirr/humaninference_rackspace/hadoop-site.xml Wrote Hadoop proxy script /home/hadoepje/.whirr/humaninference_rackspace/hadoop-proxy.sh Completed configuration of humaninference_rackspace role hadoop-datanode Completed configuration of humaninference_rackspace role hadoop-tasktracker Wrote instances file /home/hadoepje/.whirr/humaninference_rackspace/instances Started cluster of 2 instances Cluster{instances=[Instance{roles=[hadoop-datanode, hadoop-tasktracker], publicIp=50.57.166.152, privateIp=10.183.34.9, id=20933521, nodeMetadata=[id=20933521, providerId=20933521, group=humaninferencerackspace, name=humaninferencerackspace-af5, location=[id=e90923cb40144ee405a900686d6cd631, scope=HOST, description=e90923cb40144ee405a900686d6cd631, parent=cloudservers-us, iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu 10.04 LTS], state=RUNNING, loginPort=22, hostname=humaninferencerackspace-af5, privateAddresses=[10.183.34.9], publicAddresses=[50.57.166.152], hardware=[id=4, providerId=4, name=2GB server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, tags=[]]}, Instance{roles=[hadoop-jobtracker, hadoop-namenode], publicIp=50.57.166.89, privateIp=10.183.34.3, id=20933520, nodeMetadata=[id=20933520, providerId=20933520, group=humaninferencerackspace, name=humaninferencerackspace-d4b, location=[id=c948bbc4457a5a9a1d135260d0f5ec01, scope=HOST, description=c948bbc4457a5a9a1d135260d0f5ec01, parent=cloudservers-us, iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu 10.04 LTS], state=RUNNING, loginPort=22, hostname=humaninferencerackspace-d4b, privateAddresses=[10.183.34.3], publicAddresses=[50.57.166.89], hardware=[id=4, providerId=4, name=2GB server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, tags=[]]}], configuration={fs.default.name<http://fs.default.name>=hdfs://50-57-166-89.static.cloud-ips.com:8020/<http://50-57-166-89.static.cloud-ips.com:8020/>, mapred.job.tracker=50-57-166-89.static.cloud-ips.com:8021<http://50-57-166-89.static.cloud-ips.com:8021>, hadoop.job.ugi=root,root, hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory, hadoop.socks.server=localhost:6666}} You can log into instances using the following ssh commands: 'ssh -i /home/hadoepje/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected]<mailto:[email protected]>' 'ssh -i /home/hadoepje/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no [email protected]<mailto:[email protected]>' hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) bash: syntax error near unexpected token `Permission' ________________________________ From: Paritosh Ranjan [[email protected]<mailto:[email protected]>] Sent: Thursday, June 21, 2012 1:21 PM To: [email protected]<mailto:[email protected]> Subject: Whirr 0.7.1 : CDH : Rackspace : Problem Hi, I am switching to Whirr 0.7.1 from 0.6-someversion I am facing some issue while installing the cdh distribution on RackSpace ( I am able to installing the Apache distribution ). I have followed the instructions on this page http://whirr.apache.org/docs/0.7.1/quick-start-guide.html. This is my hadoop.properties file whirr.cluster-name=<theclustername> whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker whirr.provider=cloudservers-us whirr.identity=<theidentity :)> whirr.credential=<thepassword :)> whirr.hardware-id=4 whirr.image=49 whirr.private-key-file=/home/hadoepje/.ssh/id_rsa whirr.public-key-file=/home/hadoepje/.ssh/id_rsa.pub whirr.hadoop.install-function=install_cdh_hadoop whirr.hadoop.configure-function=configure_cdh_hadoop However, I am getting this error. Can someone help? Successfully executed configure script: [output= at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:207) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) hadoop-0.20-namenode. Safe mode is OFF , error=12/06/21 10:58:38 INFO namenode.FSNamesystem: supergroup=supergroup 12/06/21 10:58:38 INFO namenode.FSNamesystem: isPermissionEnabled=true 12/06/21 10:58:38 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=1000 12/06/21 10:58:38 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 12/06/21 10:58:38 INFO common.Storage: Image file of size 110 saved in 0 seconds. 12/06/21 10:58:38 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 12/06/21 10:58:38 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at 50-57-166-89.static.cloud-ips.com/50.57.166.89<http://50-57-166-89.static.cloud-ips.com/50.57.166.89> ************************************************************/ , exitCode=0] Successfully executed configure script: [output=java.io.FileNotFoundException: /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:207) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) hadoop-0.20-tasktracker. Thanks and Regards, Paritosh
