Hi, Andy and Ted Thanks a lot for pointing out my silly error. The problem did go away after I change the hostname from 'master' to the result of “hostname –f”.
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot. But it did bring problems. Thanks! On Fri, Nov 9, 2012 at 12:43 PM, Ted Reynolds <[email protected]> wrote: > yinghua, > > One other thing to note here is that on EC2 computer are given two > hostnames, one internal and one external, you need to use the external. > These host names are listed in the ec2 configuration pane on the AWS > management console. > > > Ted Reynolds > Technical Support Engineer > Hortonworks > Work Phone: 408-645-7079 > > <http://hortonworks.com/download/> > > > > > On Fri, Nov 9, 2012 at 11:31 AM, Kartashov, Andy > <[email protected]>wrote: > >> Try running “hostname –f” on each node, take a note of the fully >> qualified host address and replace your “master” with the your respective >> finding. >> >> >> >> Here are my configuration files >> >> >> >> core-site.xml >> >> >> >> <configuration> >> >> <property> >> >> <name>fs.default.name</name> >> >> <value>hdfs://master:9000</value> >> >> </property> >> >> <property> >> >> >> <name>hadoop.tmp.dir</name> >> >> >> <value>/usr/local/hadoop/tmp</value> >> >> >> </property> >> >> >> </configuration> >> >> >> >> >> >> mapred-site.xml >> >> >> >> <configuration> >> >> <property> >> >> <name>mapreduce.framework.name</name> >> >> <value>yarn</value> >> >> </property> >> >> >> >> </configuration> >> >> >> >> hdfs-site.xml >> >> >> >> <configuration> >> >> <property> >> >> <name>dfs.replication</name> >> >> <value>2</value> >> >> </property> >> >> <property> >> >> >> <name>dfs.permissions</name> >> >> >> <value>false</value> >> >> >> </property> >> >> >> <property> >> >> >> <name>dfs.namenode.name.dir</name> >> >> >> <value>file:/home/hduser/yarn_data/hdfs/namenode</value> >> >> >> </property> >> >> >> <property> >> >> >> <name>dfs.datanode.data.dir</name> >> >> >> <value>file:/home/hduser/yarn_data/hdfs/datanode</value> >> >> >> </property> >> >> >> </configuration> >> >> >> >> yarn-site.xml >> >> >> >> <?xml version="1.0"?> >> >> <configuration> >> >> <property> >> >> <name>yarn.nodemanager.aux-services</name> >> >> >> >> <value>mapreduce.shuffle</value> >> >> </property> >> >> <property> >> >> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> >> >> <value>org.apache.hadoop.mapred.ShuffleHandler</value> >> >> </property> >> >> <property> >> >> <name>yarn.nodemanager.log-aggregation-enable</name> >> >> <value>true</value> >> >> </property> >> >> <property> >> >> <name>yarn.resourcemanager.resource-tracker.address</name> >> >> <value>master:8050</value> >> >> </property> >> >> <property> >> >> <name>yarn.resourcemanager.scheduler.address</name> >> >> <value>master:8030</value> >> >> </property> >> >> <property> >> >> <name>yarn.resourcemanager.scheduler.class</name> >> >> >> >> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> >> >> >> >> </property> >> >> >> <property> >> >> >> <name>yarn.resourcemanager.address</name> >> >> >> <value>master:60400</value> >> >> >> </property> >> >> >> >> >> >> </configuration> >> >> >> >> On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu <[email protected]> wrote: >> >> Hi, Andy >> >> >> >> Thanks for suggestions! >> >> >> >> I am running it on a four node cluster on EC2. All the services started >> fine, Namenode, Datanode, ResourceManager, NodeManager and >> JobHistoryServer. Each node can ssh to all the nodes without problem. >> >> >> >> But problem appears when trying to run any job. >> >> >> >> >> >> >> >> *From:* Kartashov, Andy >> *Sent:* Friday, November 09, 2012 12:37 PM >> *To:* [email protected] >> *Subject:* Erro running pi programm >> >> >> >> Yinghua, >> >> >> >> What mode are you running your hadoop in: Local/Pseud/Fully...? >> >> >> >> Your hostname is not recognised >> >> >> >> Your configuration setting seems to be wrong. >> >> >> >> >> >> >> >> >> >> >> >> Hi, all >> >> >> >> Could some help looking at this problem? I am setting up a four node >> cluster on EC2 and seems that the cluster is set up fine until I start >> testing. >> >> >> >> I have tried password-less ssh from each node to all the nodes and there >> is no problem connecting. Any advice will be greatly appreciated! >> >> >> >> [hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar >> share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi - >> Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory >> -libjars >> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000 >> >> Number of Maps = 16 >> >> Samples per Map = 10000 >> >> Wrote input for Map #0 >> >> Wrote input for Map #1 >> >> Wrote input for Map #2 >> >> Wrote input for Map #3 >> >> Wrote input for Map #4 >> >> Wrote input for Map #5 >> >> Wrote input for Map #6 >> >> Wrote input for Map #7 >> >> Wrote input for Map #8 >> >> Wrote input for Map #9 >> >> Wrote input for Map #10 >> >> Wrote input for Map #11 >> >> Wrote input for Map #12 >> >> Wrote input for Map #13 >> >> Wrote input for Map #14 >> >> Wrote input for Map #15 >> >> Starting Job >> >> 12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to >> process : 16 >> >> 12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16 >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is >> deprecated. Instead, use mapreduce.job.classpath.files >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. >> Instead, use mapreduce.job.jar >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is >> deprecated. Instead, use mapreduce.job.cache.files >> >> 12/11/09 12:02:59 WARN conf.Configuration: >> mapred.map.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.map.speculative >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is >> deprecated. Instead, use mapreduce.job.reduces >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is >> deprecated. Instead, use mapreduce.job.output.value.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: >> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.reduce.speculative >> >> 12/11/09 12:02:59 WARN conf.Configuration: >> mapred.used.genericoptionsparser is deprecated. Instead, use >> mapreduce.client.genericoptionsparser.used >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is >> deprecated. Instead, use mapreduce.job.map.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.name is >> deprecated. Instead, use mapreduce.job.name >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is >> deprecated. Instead, use mapreduce.job.reduce.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is >> deprecated. Instead, use mapreduce.job.inputformat.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is >> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is >> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.class >> is deprecated. Instead, use mapreduce.job.outputformat.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is >> deprecated. Instead, use mapreduce.job.maps >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestamps >> is deprecated. Instead, use mapreduce.job.cache.files.timestamps >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.key.class is >> deprecated. Instead, use mapreduce.job.output.key.class >> >> 12/11/09 12:02:59 WARN conf.Configuration: mapred.working.dir is >> deprecated. Instead, use mapreduce.job.working.dir >> >> 12/11/09 12:03:00 INFO mapred.ResourceMgrDelegate: Submitted application >> application_1352478937343_0002 to ResourceManager at master/ >> 10.12.181.233:60400 >> >> 12/11/09 12:03:00 INFO mapreduce.Job: The url to track the job: >> http://master:8088/proxy/application_1352478937343_0002/ >> >> 12/11/09 12:03:00 INFO mapreduce.Job: Running job: job_1352478937343_0002 >> >> 12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 running >> in uber mode : false >> >> 12/11/09 12:03:01 INFO mapreduce.Job: map 0% reduce 0% >> >> 12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 failed >> with state FAILED due to: Application application_1352478937343_0002 failed >> 1 times due to Error launching appattempt_1352478937343_0002_000001. Got >> exception: java.lang.reflect.UndeclaredThrowableException >> >> at >> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111) >> >> at >> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:115) >> >> at >> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:258) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >> at java.lang.Thread.run(Thread.java:722) >> >> Caused by: com.google.protobuf.ServiceException: >> java.net.UnknownHostException: Yinghua java.net.UnknownHostException; For >> more details see: http://wiki.apache.org/hadoop/UnknownHost >> >> at >> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144) >> >> at $Proxy24.startContainer(Unknown Source) >> >> at >> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104) >> >> ... 5 more >> >> Caused by: java.net.UnknownHostException: Yinghua For more details see: >> http://wiki.apache.org/hadoop/UnknownHost >> >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713) >> >> at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:236) >> >> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1068) >> >> at >> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141) >> >> ... 7 more >> >> Caused by: java.net.UnknownHostException >> >> ... 11 more >> >> . Failing the application. >> >> 12/11/09 12:03:01 INFO mapreduce.Job: Counters: 0 >> >> Job Finished in 2.672 seconds >> >> java.io.FileNotFoundException: File does not exist: >> hdfs://master:9000/user/hduser/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out >> >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:738) >> >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1685) >> >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1709) >> >> at >> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) >> >> at >> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351) >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) >> >> at >> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:601) >> >> at >> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) >> >> at >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) >> >> at >> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:601) >> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:208) >> >> >> >> >> >> -- >> Regards, >> >> Yinghua >> >> >> NOTICE: This e-mail message and any attachments are confidential, >> subject to copyright and may be privileged. Any unauthorized use, copying >> or disclosure is prohibited. If you are not the intended recipient, please >> delete and contact the sender immediately. Please consider the environment >> before printing this e-mail. AVIS : le présent courriel et toute pièce >> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur >> et peuvent être couverts par le secret professionnel. Toute utilisation, >> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le >> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement >> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent >> courriel >> > > -- Regards, Yinghua
