Thanks gabriel and josh : Im back on track with your suggestions!
On Sat, Feb 15, 2014 at 12:48 PM, Josh Wills <[email protected]> wrote: > > Hey Jay, > > Replies inlined. > > On Sat, Feb 15, 2014 at 9:00 AM, Gabriel Reid <[email protected]>wrote: > >> Hi Jay, >> >> On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <[email protected]> wrote: >> > >> > When running MRPipeline here: >> > >> https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java >> > >> > I got two errors: >> > >> > 1) Running the code as-is, I got this very strange stacktrace: >> > >> > Exception in thread "Thread-3" java.lang.VerifyError: class >> > org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method >> > getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; >> > > You're getting this error b/c you're using an older version of Crunch > (0.8.2, maybe?) with Hadoop 2.2. You need to use an 0.9.0 version of > Crunch, which is compiled against the Hadoop 2.2 APIs. The getting started > guide has a section on "Which Version of Crunch do I need?" that maps > Hadoop and HBase version IDs to the corresponding Crunch release: > > http://crunch.apache.org/getting-started.html > > >> It looks like there's a version issue with protobufs in your project >> (it seems yarn is dependent on version 2.5.x, and it's version 2.4 >> that is getting pulled in. There are a number of ways to get around >> this, but easiest one is to add an explicit dependency to the >> protobuf-java at the top of your pom, i.e. >> >> <dependency> >> <groupId>com.google.protobuf</groupId> >> <artifactId>protobuf-java</artifactId> >> <version>2.5.0</version> >> </dependency> >> >> >> >> > 2) I changed line 45 to a "MemPipeline", and a new error took its >> place: >> > >> > java.lang.IncompatibleClassChangeError: Found interface >> > org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was >> expected >> > at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127) >> > at >> > >> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471) >> > at ... >> > >> > I assume there is something wrong with my environment or job setup, but >> ive >> > logically used most of this code before, and am just refactoring, so am >> > puzzled why all these funny errors have cropped up. >> > >> >> Your project is currently dependent on hadoop-2 (i.e. yarn), and is >> using a dependency on Crunch that is specific to hadoop-1. There are >> crunch builds linked to hadoop-2, and you'll need to use one of these >> in order to run Crunch together with hadoop-2. Changing your Crunch >> dependency version to 0.8.2-hadoop2 (or any other version with the >> -hadoop2 suffix) should resolve this. >> >> Hope this helps! >> >> - Gabriel >> > > > > -- > Director of Data Science > Cloudera <http://www.cloudera.com> > Twitter: @josh_wills <http://twitter.com/josh_wills> > -- Jay Vyas http://jayunit100.blogspot.com
