Hi folks. poking around with the vagrant deployer today, I saw that i couldnt run mapreduce jobs. All the services are running....
8551 DataNode 12182 ResourceManager 13245 JobHistoryServer 8119 NameNode 7293 SecondaryNameNode 7737 WebAppProxyServer 17489 Jps 12788 NodeManager My site.csv file is: cat > /bigtop-puppet/config/site.csv << EOF hadoop_head_node,$1 hadoop_storage_dirs,/data/1,/data/2 bigtop_yumrepo_uri, http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64 jdk_package_name,java-1.7.0-openjdk-devel.x86_64 components,mapred-app,hadoop,yarn, EOF But the "shuffle port" value is returning -1.... Any thoughts on why this might be occuring? Ive tried restarting nodemanager and resource manger but no luck . 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to process : 1 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1399518957219_0005 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit is no longer used. 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application application_1399518957219_0005 to ResourceManager at bigtop1.vagrant/ 10.10.10.12:8032 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job: http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/ 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running in uber mode : false 14/05/08 03:42:07 INFO mapreduce.Job: map 0% reduce 0% 14/05/08 03:42:07 INFO mapreduce.Job: Task Id : attempt_1399518957219_0005_m_000000_0, Status : FAILED Container launch failed for container_1399518957219_0005_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1399518957219_0005_m_000000_0 at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- Jay Vyas http://jayunit100.blogspot.com
