I have faced this in the past and I have to put a profile -Phadoop2.3...

mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn -DskipTests install

On Wed, Oct 8, 2014 at 1:40 PM, Chuang Liu <[email protected]> wrote:

> Hi:
>
> I tried to build Spark (1.1.0) with hadoop 2.4.0, and ran a simple
> wordcount example using spark_shell on mesos. When I ran my application, I
> got following error that looks related to the mismatch of protobuf versions
> between hadoop cluster (protobuf 2.5) and spark (protobuf 4.1). I ran "mvn
> dependency:tree -Dincludes=*protobuf*", and found that zkka pulled in this
> protobuf 4.1 Have anyone seen this problem before ? Thanks.
>
> Error when running spark on hadoop 2.4.0
> *java.lang.VerifyError: class
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$AppendRequestProto
> overrides final method
> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;*
>
> > mvn dependency:tree -Dincludes=*protobuf*
> ...
>
>
>
>
> *[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
> spark-core_2.10 ---[INFO] org.apache.spark:spark-core_2.10:jar:1.1.0[INFO]
> \-
> org.spark-project.akka:akka-remote_2.10:jar:2.2.3-shaded-protobuf:compile[INFO]
> \- org.spark-project.protobuf:protobuf-java:jar:2.4.1-shaded:compile*
>

Reply via email to