I'm always running sbt clean before building Spark. And I'm seeing the errors when I build Spark first time after downloading and extracting spark-0.9.0-incubating.tgz<http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5/spark-0.9.0-incubating.tgz>
Just in case, I deleted the test.jar files (in work/app-201401XXXXXX-0000 directories) which didn't get deleted by sbt clean. But I still see the errors. This is not a blocker for the release at all. I've tested the RC5 and don't find any other issue right now. I'm curious if anyone else sees this. On Sun, Jan 26, 2014 at 10:58 PM, Patrick Wendell <pwend...@gmail.com>wrote: > Hey Taka, > > If you build a second version you need to clean the existing assembly jar. > > The reference implementation of the tests are the ones on the U.C. > Berkeley Jenkins. These are passing for Branch 0.9 for both Hadoop 1 > and Hadoop 2 versions, so I'm inclined to think it's an issue with > your test env or setup. > > https://amplab.cs.berkeley.edu/jenkins/view/Spark/ > > - Patrick > > On Sun, Jan 26, 2014 at 10:52 PM, Reynold Xin <r...@databricks.com> wrote: > > It is possible that you have generated the assembly jar using one version > > of Hadoop, and then another assembly jar with another version. Those > tests > > that failed are all using a local cluster that sets up multiple > processes, > > which would require launching Spark worker processes using the assembly > > jar. If that's indeed the problem, removing the extra assembly jars > should > > fix them. > > > > > > On Sun, Jan 26, 2014 at 10:49 PM, Taka Shinagawa <taka.epsi...@gmail.com > >wrote: > > > >> If I build Spark for Hadoop 1.0.4 (either "SPARK_HADOOP_VERSION=1.0.4 > >> sbt/sbt assembly" or "sbt/sbt assembly") or use the binary > distribution, > >> 'sbt/sbt test' runs successfully. > >> > >> However, if I build Spark targeting any other Hadoop versions (e.g. > >> "SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly", > "SPARK_HADOOP_VERSION=2.2.0 > >> sbt/sbt assembly"), I'm getting the following errors with 'sbt/sbt > test': > >> > >> 1) type mismatch errors with JavaPairDStream.scala > >> 2) following test failures > >> [error] Failed tests: > >> [error] org.apache.spark.ShuffleNettySuite > >> [error] org.apache.spark.ShuffleSuite > >> [error] org.apache.spark.FileServerSuite > >> [error] org.apache.spark.DistributedSuite > >> > >> I don't have Hadoop 1.0.4 installed on my test systems (but the test > >> succeeds, and failed with the installed Hadoop versions). I'm seeing > these > >> sbt test errors with the previous 0.9.0 RCs and 0.8.1, too. > >> > >> I'm wondering if anyone else has seen this problem or I'm missing > something > >> to run the test correctly. > >> > >> Thanks, > >> Taka > >> > >> > >> > >> > >> On Sat, Jan 25, 2014 at 5:00 PM, Sean McNamara > >> <sean.mcnam...@webtrends.com>wrote: > >> > >> > +1 > >> > > >> > On 1/25/14, 4:04 PM, "Mark Hamstra" <m...@clearstorydata.com> wrote: > >> > > >> > >+1 > >> > > > >> > > > >> > >On Sat, Jan 25, 2014 at 2:37 PM, Andy Konwinski > >> > ><andykonwin...@gmail.com>wrote: > >> > > > >> > >> +1 > >> > >> > >> > >> > >> > >> On Sat, Jan 25, 2014 at 2:27 PM, Reynold Xin <r...@databricks.com> > >> > >>wrote: > >> > >> > >> > >> > +1 > >> > >> > > >> > >> > > On Jan 25, 2014, at 12:07 PM, Hossein <fal...@gmail.com> > wrote: > >> > >> > > > >> > >> > > +1 > >> > >> > > > >> > >> > > Compiled and tested on Mavericks. > >> > >> > > > >> > >> > > --Hossein > >> > >> > > > >> > >> > > > >> > >> > > On Sat, Jan 25, 2014 at 11:38 AM, Patrick Wendell > >> > >><pwend...@gmail.com > >> > >> > >wrote: > >> > >> > > > >> > >> > >> I'll kick of the voting with a +1. > >> > >> > >> > >> > >> > >> On Thu, Jan 23, 2014 at 11:33 PM, Patrick Wendell > >> > >><pwend...@gmail.com > >> > >> > > >> > >> > >> wrote: > >> > >> > >>> Please vote on releasing the following candidate as Apache > Spark > >> > >> > >>> (incubating) version 0.9.0. > >> > >> > >>> > >> > >> > >>> A draft of the release notes along with the changes file is > >> > >>attached > >> > >> > >>> to this e-mail. > >> > >> > >>> > >> > >> > >>> The tag to be voted on is v0.9.0-incubating (commit > 95d28ff3): > >> > >> > >> > >> > >> > > >> > >> > >> > >> > >> > > >> > https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h= > >> > >>95d28ff3d0d20d9c583e184f9e2c5ae842d8a4d9 > >> > >> > >>> > >> > >> > >>> The release files, including signatures, digests, etc can be > >> found > >> > >> at: > >> > >> > >>> > http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5 > >> > >> > >>> > >> > >> > >>> Release artifacts are signed with the following key: > >> > >> > >>> https://people.apache.org/keys/committer/pwendell.asc > >> > >> > >>> > >> > >> > >>> The staging repository for this release can be found at: > >> > >> > >>> > >> > >> > > >> > >> > >> https://repository.apache.org/content/repositories/orgapachespark-1006/ > >> > >> > >>> > >> > >> > >>> The documentation corresponding to this release can be found > at: > >> > >> > >>> > >> > >>http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5-docs/ > >> > >> > >>> > >> > >> > >>> Please vote on releasing this package as Apache Spark > >> > >> 0.9.0-incubating! > >> > >> > >>> > >> > >> > >>> The vote is open until Monday, January 27, at 07:30 UTC and > >> passes > >> > >> ifa > >> > >> > >>> majority of at least 3 +1 PPMC votes are cast. > >> > >> > >>> > >> > >> > >>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating > >> > >> > >>> [ ] -1 Do not release this package because ... > >> > >> > >>> > >> > >> > >>> To learn more about Apache Spark, please see > >> > >> > >>> http://spark.incubator.apache.org/ > >> > >> > >> > >> > >> > > >> > >> > -- > >> > >> > You received this message because you are subscribed to the > Google > >> > >>Groups > >> > >> > "Unofficial Apache Spark Dev Mailing List Mirror" group. > >> > >> > To unsubscribe from this group and stop receiving emails from it, > >> > >>send an > >> > >> > email to apache-spark-dev-mirror+unsubscr...@googlegroups.com. > >> > >> > For more options, visit https://groups.google.com/groups/opt_out > . > >> > >> > > >> > >> > >> > > >> > > >> >
