Hi, This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb" as 64MB which is too low.
1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned. 2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory. Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job. Thanks & Regards Rohith Sharma K S This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! From: mail list [mailto:louis.hust...@gmail.com] Sent: 23 October 2014 07:55 To: user@hadoop.apache.org Subject: mapred job pending at "Starting scan to move intermediate done files" hi, all, I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode. When I run a mapped job, the job output the following logs: louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/ 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001 As you see, the job halt. Then I check the jps output: louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps 22433 SecondaryNameNode 22716 NodeManager 22240 DataNode 22577 ResourceManager 23083 JobHistoryServer 23148 Jps 22080 NameNode It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log: 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files Then i check the web ui: It seems the job is pending!!!! The attachment contains some configuration files at etc/hadoop/ . Any idea will be appreciated !
