Hey Jay, Replies inlined.
On Sat, Feb 15, 2014 at 9:00 AM, Gabriel Reid <[email protected]>wrote: > Hi Jay, > > On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <[email protected]> wrote: > > > > When running MRPipeline here: > > > https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java > > > > I got two errors: > > > > 1) Running the code as-is, I got this very strange stacktrace: > > > > Exception in thread "Thread-3" java.lang.VerifyError: class > > org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method > > getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; > You're getting this error b/c you're using an older version of Crunch (0.8.2, maybe?) with Hadoop 2.2. You need to use an 0.9.0 version of Crunch, which is compiled against the Hadoop 2.2 APIs. The getting started guide has a section on "Which Version of Crunch do I need?" that maps Hadoop and HBase version IDs to the corresponding Crunch release: http://crunch.apache.org/getting-started.html > It looks like there's a version issue with protobufs in your project > (it seems yarn is dependent on version 2.5.x, and it's version 2.4 > that is getting pulled in. There are a number of ways to get around > this, but easiest one is to add an explicit dependency to the > protobuf-java at the top of your pom, i.e. > > <dependency> > <groupId>com.google.protobuf</groupId> > <artifactId>protobuf-java</artifactId> > <version>2.5.0</version> > </dependency> > > > > > 2) I changed line 45 to a "MemPipeline", and a new error took its > place: > > > > java.lang.IncompatibleClassChangeError: Found interface > > org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was > expected > > at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127) > > at > > > org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471) > > at ... > > > > I assume there is something wrong with my environment or job setup, but > ive > > logically used most of this code before, and am just refactoring, so am > > puzzled why all these funny errors have cropped up. > > > > Your project is currently dependent on hadoop-2 (i.e. yarn), and is > using a dependency on Crunch that is specific to hadoop-1. There are > crunch builds linked to hadoop-2, and you'll need to use one of these > in order to run Crunch together with hadoop-2. Changing your Crunch > dependency version to 0.8.2-hadoop2 (or any other version with the > -hadoop2 suffix) should resolve this. > > Hope this helps! > > - Gabriel > -- Director of Data Science Cloudera <http://www.cloudera.com> Twitter: @josh_wills <http://twitter.com/josh_wills>
