Pete MacKinnon created SPARK-2338:
-------------------------------------
Summary: Jenkins Spark-Master-Maven-with-YARN builds failing due
to test misconfiguration
Key: SPARK-2338
URL: https://issues.apache.org/jira/browse/SPARK-2338
Project: Spark
Issue Type: Bug
Components: Build, Project Infra, YARN
Affects Versions: 1.0.0
Environment: https://amplab.cs.berkeley.edu/jenkins
Reporter: Pete MacKinnon
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/
These builds are currently failing due to the builder configuration being
incomplete. After building, they specify the test command as:
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
However, it is not enough to specify the hadoop.version, the tests should
instead be run using the {hadoop-2.2, hadoop-2.3} profiles respectively.
For example:
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
These profiles will not only set the appropriate hadoop.version but also set
the version of protobuf-java required by yarn (2.5.0). Without the correct
profile set, the test run fails at:
*** RUN ABORTED ***
java.lang.VerifyError: class
org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
since it is getting the default version of protobuf-java (2.4.1) which has the
old incompatible version of getUnknownFields.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
