[
https://issues.apache.org/jira/browse/SPARK-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pete MacKinnon updated SPARK-2338:
----------------------------------
Description:
{noformat}
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/
{noformat}
These builds are currently failing due to the builder configuration being
incomplete. After building, they specify the test command as:
{noformat}
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
{noformat}
However, it is not enough to specify the hadoop.version, the tests should
instead be run using the hadoop-2.2 and hadoop-2.3 profiles respectively.
For example:
{noformat}
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
{noformat}
These profiles will not only set the appropriate hadoop.version but also set
the version of protobuf-java required by yarn (2.5.0). Without the correct
profile set, the test run fails at:
{noformat}
*** RUN ABORTED ***
java.lang.VerifyError: class
org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
{noformat}
since it is getting the default version of protobuf-java (2.4.1) which has the
old incompatible version of getUnknownFields.
was:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/
These builds are currently failing due to the builder configuration being
incomplete. After building, they specify the test command as:
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
However, it is not enough to specify the hadoop.version, the tests should
instead be run using the {hadoop-2.2, hadoop-2.3} profiles respectively.
For example:
/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
-Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
These profiles will not only set the appropriate hadoop.version but also set
the version of protobuf-java required by yarn (2.5.0). Without the correct
profile set, the test run fails at:
*** RUN ABORTED ***
java.lang.VerifyError: class
org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
since it is getting the default version of protobuf-java (2.4.1) which has the
old incompatible version of getUnknownFields.
> Jenkins Spark-Master-Maven-with-YARN builds failing due to test
> misconfiguration
> --------------------------------------------------------------------------------
>
> Key: SPARK-2338
> URL: https://issues.apache.org/jira/browse/SPARK-2338
> Project: Spark
> Issue Type: Bug
> Components: Build, Project Infra, YARN
> Affects Versions: 1.0.0
> Environment: https://amplab.cs.berkeley.edu/jenkins
> Reporter: Pete MacKinnon
> Labels: hadoop2, jenkins, maven, protobuf, yarn
>
> {noformat}
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/
> {noformat}
> These builds are currently failing due to the builder configuration being
> incomplete. After building, they specify the test command as:
> {noformat}
> /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
> -Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
> {noformat}
> However, it is not enough to specify the hadoop.version, the tests should
> instead be run using the hadoop-2.2 and hadoop-2.3 profiles respectively.
> For example:
> {noformat}
> /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn
> -Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
> {noformat}
> These profiles will not only set the appropriate hadoop.version but also set
> the version of protobuf-java required by yarn (2.5.0). Without the correct
> profile set, the test run fails at:
> {noformat}
> *** RUN ABORTED ***
> java.lang.VerifyError: class
> org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final
> method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
> {noformat}
> since it is getting the default version of protobuf-java (2.4.1) which has
> the old incompatible version of getUnknownFields.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
