No probably :) In the end I fixed it by exporting the right env variables iirc
> On May 13, 2014, at 7:47 PM, Mark Grover <[email protected]> wrote: > > Sorry about the delay, Jay. Not entirely sure what's going on there but the > shuffle properties were renamed in Hadoop 2.2.0. > > If the VM is using that or later version of hadoop, it might be worthwhile to > ensure property names are correct. > > Please see https://issues.apache.org/jira/browse/BIGTOP-1118 for details. > > Mark > > >> On Wed, May 7, 2014 at 8:48 PM, Jay Vyas <[email protected]> wrote: >> Hi folks. poking around with the vagrant deployer today, I saw that i >> couldnt run mapreduce jobs. All the services are running.... >> >> >> 8551 DataNode >> 12182 ResourceManager >> 13245 JobHistoryServer >> 8119 NameNode >> 7293 SecondaryNameNode >> 7737 WebAppProxyServer >> 17489 Jps >> 12788 NodeManager >> >> My site.csv file is: >> >> cat > /bigtop-puppet/config/site.csv << EOF >> hadoop_head_node,$1 >> hadoop_storage_dirs,/data/1,/data/2 >> bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64 >> jdk_package_name,java-1.7.0-openjdk-devel.x86_64 >> components,mapred-app,hadoop,yarn, >> EOF >> >> But the "shuffle port" value is returning -1.... Any thoughts on why this >> might be occuring? Ive tried restarting nodemanager and resource manger >> but no luck . >> >> 14/05/08 03:41:59 INFO service.AbstractService: >> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >> 14/05/08 03:41:59 INFO service.AbstractService: >> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >> 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to process : >> 1 >> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1 >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated. >> Instead, use mapreduce.job.jar >> 14/05/08 03:41:59 WARN conf.Configuration: >> mapred.map.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.map.speculative >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is >> deprecated. Instead, use mapreduce.job.reduces >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is >> deprecated. Instead, use mapreduce.job.output.value.class >> 14/05/08 03:41:59 WARN conf.Configuration: >> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.reduce.speculative >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is >> deprecated. Instead, use mapreduce.job.map.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated. >> Instead, use mapreduce.job.name >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is >> deprecated. Instead, use mapreduce.job.reduce.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is >> deprecated. Instead, use mapreduce.job.inputformat.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated. >> Instead, use mapreduce.input.fileinputformat.inputdir >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is deprecated. >> Instead, use mapreduce.output.fileoutputformat.outputdir >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class is >> deprecated. Instead, use mapreduce.job.outputformat.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated. >> Instead, use mapreduce.job.maps >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is >> deprecated. Instead, use mapreduce.job.output.key.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is deprecated. >> Instead, use mapreduce.job.working.dir >> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: >> job_1399518957219_0005 >> 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit is >> no longer used. >> 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application >> application_1399518957219_0005 to ResourceManager at >> bigtop1.vagrant/10.10.10.12:8032 >> 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job: >> http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/ >> 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005 >> 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running in >> uber mode : false >> 14/05/08 03:42:07 INFO mapreduce.Job: map 0% reduce 0% >> 14/05/08 03:42:07 INFO mapreduce.Job: Task Id : >> attempt_1399518957219_0005_m_000000_0, Status : FAILED >> Container launch failed for container_1399518957219_0005_01_000002 : >> java.lang.IllegalStateException: Invalid shuffle port number -1 returned for >> attempt_1399518957219_0005_m_000000_0 >> at >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170) >> at >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> >> >> -- >> Jay Vyas >> http://jayunit100.blogspot.com >
