Try replacing: whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
with: whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1 hadoop-datanode+hadoop-tasktracker The role order is important. -- Andrei Savu / axemblr.com / Tools for Clouds On Thu, Jun 21, 2012 at 2:52 PM, Paritosh Ranjan <[email protected]> wrote: > Sometimes the namenode is up but the jobtracker has never been up. > I have tried it 4-5 times. > ------------------------------ > *From:* Andrei Savu [[email protected]] > *Sent:* Thursday, June 21, 2012 1:49 PM > *To:* [email protected] > *Subject:* Re: Whirr 0.7.1 : CDH : Rackspace : Problem > > Have you checked the servers? I see no serious error. The file > permissions issue should be harmless. > > By default Whirr is deploying CDH3U4. Is this what you need? > > -- Andrei Savu / axemblr.com / Tools for Clouds > > On Thu, Jun 21, 2012 at 2:43 PM, Paritosh Ranjan <[email protected]>wrote: > >> This is the complete log, if it can help. >> >> ./whirr launch-cluster --config hadoop.properties >> Bootstrapping cluster >> Configuring template >> Configuring template >> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker] >> Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode] >> Nodes started: [[id=20933521, providerId=20933521, >> group=humaninferencerackspace, name=humaninferencerackspace-af5, >> location=[id=e90923cb40144ee405a900686d6cd631, scope=HOST, >> description=e90923cb40144ee405a900686d6cd631, parent=cloudservers-us, >> iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, >> family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu >> 10.04 LTS], state=RUNNING, loginPort=22, >> hostname=humaninferencerackspace-af5, privateAddresses=[10.183.34.9], >> publicAddresses=[50.57.166.152], hardware=[id=4, providerId=4, name=2GB >> server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, >> type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], >> supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, >> tags=[]]] >> Nodes started: [[id=20933520, providerId=20933520, >> group=humaninferencerackspace, name=humaninferencerackspace-d4b, >> location=[id=c948bbc4457a5a9a1d135260d0f5ec01, scope=HOST, >> description=c948bbc4457a5a9a1d135260d0f5ec01, parent=cloudservers-us, >> iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, >> family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu >> 10.04 LTS], state=RUNNING, loginPort=22, >> hostname=humaninferencerackspace-d4b, privateAddresses=[10.183.34.3], >> publicAddresses=[50.57.166.89], hardware=[id=4, providerId=4, name=2GB >> server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, >> type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], >> supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, >> tags=[]]] >> Authorizing firewall ingress to [20933520] on ports [50030] for [ >> 108.171.174.174/32] >> Authorizing firewall ingress to [20933520] on ports [8021] for [ >> 50.57.166.89/32] >> Authorizing firewall ingress to [20933520] on ports [50070] for [ >> 108.171.174.174/32] >> Authorizing firewall ingress to [20933520] on ports [8020, 8021] for [ >> 50.57.166.89/32] >> Authorizing firewall ingress to [20933520] on ports [50030] for [ >> 108.171.174.174/32] >> Authorizing firewall ingress to [20933520] on ports [8021] for [ >> 50.57.166.89/32] >> Starting to run scripts on cluster for phase configureinstances: 20933521 >> Starting to run scripts on cluster for phase configureinstances: 20933520 >> Running configure phase script on: 20933521 >> Running configure phase script on: 20933520 >> configure phase script run completed on: 20933521 >> configure phase script run completed on: 20933520 >> >> Successfully executed configure script: [output= at >> java.io.FileOutputStream.openAppend(Native Method) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:207) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:131) >> at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) >> at >> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) >> at >> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) >> at >> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) >> hadoop-0.20-namenode. >> Safe mode is OFF >> , error=12/06/21 10:58:38 INFO namenode.FSNamesystem: >> supergroup=supergroup >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: isPermissionEnabled=true >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: >> dfs.block.invalidate.limit=1000 >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: isAccessTokenEnabled=false >> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) >> 12/06/21 10:58:38 INFO common.Storage: Image file of size 110 saved in 0 >> seconds. >> 12/06/21 10:58:38 INFO common.Storage: Storage directory >> /data/hadoop/hdfs/name has been successfully formatted. >> 12/06/21 10:58:38 INFO namenode.NameNode: SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down NameNode at >> 50-57-166-89.static.cloud-ips.com/50.57.166.89 >> ************************************************************/ >> , exitCode=0] >> Successfully executed configure script: >> [output=java.io.FileNotFoundException: >> /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) >> at java.io.FileOutputStream.openAppend(Native Method) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:207) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:131) >> at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) >> at >> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) >> at >> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) >> at >> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) >> hadoop-0.20-tasktracker. >> , error=dpkg-preconfigure: unable to re-open stdin: >> update-rc.d: warning: hadoop-0.20-datanode start runlevel arguments (2 3 >> 4 5) do not match LSB Default-Start values (3 5) >> update-rc.d: warning: hadoop-0.20-datanode stop runlevel arguments (0 1 >> 6) do not match LSB Default-Stop values (0 1 2 4 6) >> dpkg-preconfigure: unable to re-open stdin: >> update-rc.d: warning: hadoop-0.20-tasktracker start runlevel arguments (2 >> 3 4 5) do not match LSB Default-Start values (3 5) >> update-rc.d: warning: hadoop-0.20-tasktracker stop runlevel arguments (0 >> 1 6) do not match LSB Default-Stop values (0 1 2 4 6) >> , exitCode=0] >> Finished running configure phase scripts on all cluster instances >> Completed configuration of humaninference_rackspace role hadoop-jobtracker >> Jobtracker web UI available at >> http://50-57-166-89.static.cloud-ips.com:50030 >> Completed configuration of humaninference_rackspace role hadoop-namenode >> Namenode web UI available at >> http://50-57-166-89.static.cloud-ips.com:50070 >> Wrote Hadoop site file >> /home/hadoepje/.whirr/humaninference_rackspace/hadoop-site.xml >> Wrote Hadoop proxy script >> /home/hadoepje/.whirr/humaninference_rackspace/hadoop-proxy.sh >> Completed configuration of humaninference_rackspace role hadoop-datanode >> Completed configuration of humaninference_rackspace role >> hadoop-tasktracker >> Wrote instances file >> /home/hadoepje/.whirr/humaninference_rackspace/instances >> Started cluster of 2 instances >> Cluster{instances=[Instance{roles=[hadoop-datanode, hadoop-tasktracker], >> publicIp=50.57.166.152, privateIp=10.183.34.9, id=20933521, >> nodeMetadata=[id=20933521, providerId=20933521, >> group=humaninferencerackspace, name=humaninferencerackspace-af5, >> location=[id=e90923cb40144ee405a900686d6cd631, scope=HOST, >> description=e90923cb40144ee405a900686d6cd631, parent=cloudservers-us, >> iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, >> family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu >> 10.04 LTS], state=RUNNING, loginPort=22, >> hostname=humaninferencerackspace-af5, privateAddresses=[10.183.34.9], >> publicAddresses=[50.57.166.152], hardware=[id=4, providerId=4, name=2GB >> server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, >> type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], >> supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, >> tags=[]]}, Instance{roles=[hadoop-jobtracker, hadoop-namenode], >> publicIp=50.57.166.89, privateIp=10.183.34.3, id=20933520, >> nodeMetadata=[id=20933520, providerId=20933520, >> group=humaninferencerackspace, name=humaninferencerackspace-d4b, >> location=[id=c948bbc4457a5a9a1d135260d0f5ec01, scope=HOST, >> description=c948bbc4457a5a9a1d135260d0f5ec01, parent=cloudservers-us, >> iso3166Codes=[], metadata={}], uri=null, imageId=112, os=[name=null, >> family=ubuntu, version=10.04, arch=null, is64Bit=true, description=Ubuntu >> 10.04 LTS], state=RUNNING, loginPort=22, >> hostname=humaninferencerackspace-d4b, privateAddresses=[10.183.34.3], >> publicAddresses=[50.57.166.89], hardware=[id=4, providerId=4, name=2GB >> server, processors=[[cores=8.0, speed=1.0]], ram=2048, volumes=[[id=null, >> type=LOCAL, size=80.0, device=null, durable=true, isBootDevice=true]], >> supportsImage=ALWAYS_TRUE, tags=[]], loginUser=root, userMetadata={}, >> tags=[]]}], configuration={fs.default.name=hdfs:// >> 50-57-166-89.static.cloud-ips.com:8020/, mapred.job.tracker= >> 50-57-166-89.static.cloud-ips.com:8021, hadoop.job.ugi=root,root, >> hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory, >> hadoop.socks.server=localhost:6666}} >> You can log into instances using the following ssh commands: >> 'ssh -i /home/hadoepje/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o >> StrictHostKeyChecking=no [email protected]' >> 'ssh -i /home/hadoepje/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o >> StrictHostKeyChecking=no [email protected]' >> hadoepje@hadoepgrote:~/whirr/whirr-0.7.1/bin$ >> /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) >> bash: syntax error near unexpected token `Permission' >> >> ------------------------------ >> *From:* Paritosh Ranjan [[email protected]] >> *Sent:* Thursday, June 21, 2012 1:21 PM >> *To:* [email protected] >> *Subject:* Whirr 0.7.1 : CDH : Rackspace : Problem >> >> Hi, >> >> I am switching to Whirr 0.7.1 from 0.6-someversion >> I am facing some issue while installing the cdh distribution on RackSpace >> ( I am able to installing the Apache distribution ). >> >> I have followed the instructions on this page >> http://whirr.apache.org/docs/0.7.1/quick-start-guide.html. >> >> This is my hadoop.properties file >> >> whirr.cluster-name=<theclustername> >> whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 >> hadoop-datanode+hadoop-tasktracker >> whirr.provider=cloudservers-us >> whirr.identity=<theidentity :)> >> whirr.credential=<thepassword :)> >> whirr.hardware-id=4 >> whirr.image=49 >> whirr.private-key-file=/home/hadoepje/.ssh/id_rsa >> whirr.public-key-file=/home/hadoepje/.ssh/id_rsa.pub >> whirr.hadoop.install-function=install_cdh_hadoop >> whirr.hadoop.configure-function=configure_cdh_hadoop >> >> >> However, I am getting this error. Can someone help? >> >> Successfully executed configure script: [output= at >> java.io.FileOutputStream.openAppend(Native Method) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:207) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:131) >> at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) >> at >> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) >> at >> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) >> at >> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) >> hadoop-0.20-namenode. >> Safe mode is OFF >> , error=12/06/21 10:58:38 INFO namenode.FSNamesystem: >> supergroup=supergroup >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: isPermissionEnabled=true >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: >> dfs.block.invalidate.limit=1000 >> 12/06/21 10:58:38 INFO namenode.FSNamesystem: isAccessTokenEnabled=false >> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) >> 12/06/21 10:58:38 INFO common.Storage: Image file of size 110 saved in 0 >> seconds. >> 12/06/21 10:58:38 INFO common.Storage: Storage directory >> /data/hadoop/hdfs/name has been successfully formatted. >> 12/06/21 10:58:38 INFO namenode.NameNode: SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down NameNode at >> 50-57-166-89.static.cloud-ips.com/50.57.166.89 >> ************************************************************/ >> , exitCode=0] >> Successfully executed configure script: >> [output=java.io.FileNotFoundException: >> /var/log/hadoop/logs/SecurityAuth.audit (Permission denied) >> at java.io.FileOutputStream.openAppend(Native Method) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:207) >> at java.io.FileOutputStream.<init>(FileOutputStream.java:131) >> at org.apache.log4j.FileAppender.setFile(FileAppender.java:290) >> at >> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:164) >> at >> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:216) >> at >> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133) >> hadoop-0.20-tasktracker. >> >> Thanks and Regards, >> Paritosh >> > >
