Hi Jay,
On Sat, Feb 15, 2014 at 3:29 AM, Jay Vyas <[email protected]> wrote:
>
> When running MRPipeline here:
> https://github.com/jayunit100/bigpetstore/blob/63958ddde9eb18ea75f078668a0023288547bbcf/src/integration/java/org/bigtop/bigpetstore/integration/BigPetStoreCrunchIT.java
>
> I got two errors:
>
> 1) Running the code as-is, I got this very strange stacktrace:
>
> Exception in thread "Thread-3" java.lang.VerifyError: class
> org.apache.hadoop.yarn.proto.YarnProtos$URLProto overrides final method
> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
It looks like there's a version issue with protobufs in your project
(it seems yarn is dependent on version 2.5.x, and it's version 2.4
that is getting pulled in. There are a number of ways to get around
this, but easiest one is to add an explicit dependency to the
protobuf-java at the top of your pom, i.e.
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
</dependency>
> 2) I changed line 45 to a "MemPipeline", and a new error took its place:
>
> java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
> at org.apache.crunch.DoFn.getConfiguration(DoFn.java:127)
> at
> org.apache.crunch.fn.Aggregators$AggregatorCombineFn.initialize(Aggregators.java:471)
> at ...
>
> I assume there is something wrong with my environment or job setup, but ive
> logically used most of this code before, and am just refactoring, so am
> puzzled why all these funny errors have cropped up.
>
Your project is currently dependent on hadoop-2 (i.e. yarn), and is
using a dependency on Crunch that is specific to hadoop-1. There are
crunch builds linked to hadoop-2, and you'll need to use one of these
in order to run Crunch together with hadoop-2. Changing your Crunch
dependency version to 0.8.2-hadoop2 (or any other version with the
-hadoop2 suffix) should resolve this.
Hope this helps!
- Gabriel
