Try running "hostname -f" on each node, take a note of the fully qualified host
address and replace your "master" with the your respective finding.
Here are my configuration files
core-site.xml
<configuration>
<property>
<name>fs.default.name<http://fs.default.name></name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name<http://mapreduce.framework.name></name>
<value>yarn</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/yarn_data/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/yarn_data/hdfs/datanode</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8050</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:60400</value>
</property>
</configuration>
On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu
<[email protected]<mailto:[email protected]>> wrote:
Hi, Andy
Thanks for suggestions!
I am running it on a four node cluster on EC2. All the services started fine,
Namenode, Datanode, ResourceManager, NodeManager and JobHistoryServer. Each
node can ssh to all the nodes without problem.
But problem appears when trying to run any job.
From: Kartashov, Andy
Sent: Friday, November 09, 2012 12:37 PM
To: [email protected]
Subject: Erro running pi programm
Yinghua,
What mode are you running your hadoop in: Local/Pseud/Fully...?
Your hostname is not recognised
Your configuration setting seems to be wrong.
Hi, all
Could some help looking at this problem? I am setting up a four node cluster on
EC2 and seems that the cluster is set up fine until I start testing.
I have tried password-less ssh from each node to all the nodes and there is no
problem connecting. Any advice will be greatly appreciated!
[hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi
-Dmapreduce.clientfactory.class.name<http://Dmapreduce.clientfactory.class.name>=org.apache.hadoop.mapred.YarnClientFactory
-libjars share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar
16 10000
Number of Maps = 16
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Starting Job
12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to process : 16
12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is
deprecated. Instead, use mapreduce.job.classpath.files
12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is deprecated.
Instead, use mapreduce.job.cache.files
12/11/09 12:02:59 WARN conf.Configuration:
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated.
Instead, use mapreduce.job.reduces
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
12/11/09 12:02:59 WARN conf.Configuration:
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is deprecated.
Instead, use mapreduce.job.map.class
12/11/09 12:02:59 WARN conf.Configuration:
mapred.job.name<http://mapred.job.name> is deprecated. Instead, use
mapreduce.job.name<http://mapreduce.job.name>
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is
deprecated. Instead, use mapreduce.job.inputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.class is
deprecated. Instead, use mapreduce.job.outputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestamps is
deprecated. Instead, use mapreduce.job.cache.files.timestamps
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.working.dir is deprecated.
Instead, use mapreduce.job.working.dir
12/11/09 12:03:00 INFO mapred.ResourceMgrDelegate: Submitted application
application_1352478937343_0002 to ResourceManager at
master/10.12.181.233:60400<http://10.12.181.233:60400>
12/11/09 12:03:00 INFO mapreduce.Job: The url to track the job:
http://master:8088/proxy/application_1352478937343_0002/
12/11/09 12:03:00 INFO mapreduce.Job: Running job: job_1352478937343_0002
12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 running in
uber mode : false
12/11/09 12:03:01 INFO mapreduce.Job: map 0% reduce 0%
12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 failed with
state FAILED due to: Application application_1352478937343_0002 failed 1 times
due to Error launching appattempt_1352478937343_0002_000001. Got exception:
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:115)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:258)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException:
Yinghua java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
at $Proxy24.startContainer(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
... 5 more
Caused by: java.net.UnknownHostException: Yinghua For more details see:
http://wiki.apache.org/hadoop/UnknownHost
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:236)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
at org.apache.hadoop.ipc.Client.call(Client.java:1068)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
... 7 more
Caused by: java.net.UnknownHostException
... 11 more
. Failing the application.
12/11/09 12:03:01 INFO mapreduce.Job: Counters: 0
Job Finished in 2.672 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://master:9000/user/hduser/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:738)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1685)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1709)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at
org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at
org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to
copyright and may be privileged. Any unauthorized use, copying or disclosure is
prohibited. If you are not the intended recipient, please delete and contact
the sender immediately. Please consider the environment before printing this
e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont
confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le
secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est
interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel,
supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ?
l'environnement avant d'imprimer le pr?sent courriel
