May I recommend using Cascading instead of using MR directly? Cascading supports Hadoop 1.x and Hadoop 2.x based distros and you don't have to wrestle with these things all the time: http://www.cascading.org/ It's OSS, ASL v2 licensed and all the good stuff.
- André On Sun, May 11, 2014 at 1:52 AM, Tony Dean <[email protected]> wrote: > Hi, > > > > I am trying to write a Java application that works with either MR1 and > MR2. At the present I have MR2 (YARN) implementation deployed and > running. I am using mapred API. I believe that I read mapred and > mapreduce APIs are compatible so either should work. The only thing that > is different is the configuration properties that need to be specified > depending on whether the back-end is MR1 or MR2. BTW: I’m using CDH 4.6 > (Hadoop 2.0). > > > > My problem is that I can’t seem to submit a job to the cluster. It always > runs locally. I setup JobConf with appropriate properties and submit the > jobs using JobClient. The properties that I set on JobConf are as follows: > > > > mapreduce.jobtracker.address=host:port (I know this is for MR1, but I’m > trying everything) > > mapreduce.framework.name=yarn > > yarn.resourcemanager.address=host:port > > yarn.resourcemanager.host=host:port > > > > The last 2 are the same but I read 2 different ways to set it in different > conflicting documentations. > > > > Anyway, can someone explain how to get this seemingly simple deployment to > work? What am I missing? > > > > Thanks!!! > -- André Kelpe [email protected] http://concurrentinc.com
