just rerun my test on rc5 everything works
build applications with sbt and the spark-*.jar which is compiled with Hadoop 2.3 +1 -- Nan Zhu On Sunday, May 18, 2014 at 11:07 PM, witgo wrote: > How to reproduce this bug? > > > ------------------ Original ------------------ > From: "Patrick Wendell";<pwend...@gmail.com (mailto:pwend...@gmail.com)>; > Date: Mon, May 19, 2014 10:08 AM > To: "dev@spark.apache.org (mailto:dev@spark.apache.org)"<dev@spark.apache.org > (mailto:dev@spark.apache.org)>; > Cc: "Tom Graves"<tgraves...@yahoo.com (mailto:tgraves...@yahoo.com)>; > Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9) > > > > Hey Matei - the issue you found is not related to security. This patch > a few days ago broke builds for Hadoop 1 with YARN support enabled. > The patch directly altered the way we deal with commons-lang > dependency, which is what is at the base of this stack trace. > > https://github.com/apache/spark/pull/754 > > - Patrick > > On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <matei.zaha...@gmail.com > (mailto:matei.zaha...@gmail.com)> wrote: > > Alright, I've opened https://github.com/apache/spark/pull/819 with the > > Windows fixes. I also found one other likely bug, > > https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages > > for Hadoop1 built in this RC. I think this is due to Hadoop 1's security > > code depending on a different version of org.apache.commons than Hadoop 2, > > but it needs investigation. Tom, any thoughts on this? > > > > Matei > > > > On May 18, 2014, at 12:33 PM, Matei Zaharia <matei.zaha...@gmail.com > > (mailto:matei.zaha...@gmail.com)> wrote: > > > > > I took the always fun task of testing it on Windows, and unfortunately, I > > > found some small problems with the prebuilt packages due to recent > > > changes to the launch scripts: bin/spark-class2.cmd looks in ./jars > > > instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't > > > quite match the master-setting behavior of the Unix based one. I'll send > > > a pull request to fix them soon. > > > > > > Matei > > > > > > > > > On May 17, 2014, at 11:32 AM, Sandy Ryza <sandy.r...@cloudera.com > > > (mailto:sandy.r...@cloudera.com)> wrote: > > > > > > > +1 > > > > > > > > Reran my tests from rc5: > > > > > > > > * Built the release from source. > > > > * Compiled Java and Scala apps that interact with HDFS against it. > > > > * Ran them in local mode. > > > > * Ran them against a pseudo-distributed YARN cluster in both yarn-client > > > > mode and yarn-cluster mode. > > > > > > > > > > > > On Sat, May 17, 2014 at 10:08 AM, Andrew Or <and...@databricks.com > > > > (mailto:and...@databricks.com)> wrote: > > > > > > > > > +1 > > > > > > > > > > > > > > > 2014-05-17 8:53 GMT-07:00 Mark Hamstra <m...@clearstorydata.com > > > > > (mailto:m...@clearstorydata.com)>: > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell > > > > > > <pwend...@gmail.com (mailto:pwend...@gmail.com) > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I'll start the voting with a +1. > > > > > > > > > > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell > > > > > > > <pwend...@gmail.com (mailto:pwend...@gmail.com)> > > > > > > > wrote: > > > > > > > > Please vote on releasing the following candidate as Apache Spark > > > > > > > > > > > > > > > > > > > > > > > > > > version > > > > > > > 1.0.0! > > > > > > > > This has one bug fix and one minor feature on top of rc8: > > > > > > > > SPARK-1864: https://github.com/apache/spark/pull/808 > > > > > > > > SPARK-1808: https://github.com/apache/spark/pull/799 > > > > > > > > > > > > > > > > The tag to be voted on is v1.0.0-rc9 (commit 920f947): > > > > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 > > > > > > > > > > > > > > > > The release files, including signatures, digests, etc. can be > > > > > > > > found > > > > > at: > > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9/ > > > > > > > > > > > > > > > > Release artifacts are signed with the following key: > > > > > > > > https://people.apache.org/keys/committer/pwendell.asc > > > > > > > > > > > > > > > > The staging repository for this release can be found at: > > > > > > https://repository.apache.org/content/repositories/orgapachespark-1017/ > > > > > > > > > > > > > > > > The documentation corresponding to this release can be found at: > > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ > > > > > > > > > > > > > > > > Please vote on releasing this package as Apache Spark 1.0.0! > > > > > > > > > > > > > > > > The vote is open until Tuesday, May 20, at 08:56 UTC and passes > > > > > > > > if > > > > > > > > amajority of at least 3 +1 PMC votes are cast. > > > > > > > > > > > > > > > > [ ] +1 Release this package as Apache Spark 1.0.0 > > > > > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > > > > > > > To learn more about Apache Spark, please see > > > > > > > > http://spark.apache.org/ > > > > > > > > > > > > > > > > == API Changes == > > > > > > > > We welcome users to compile Spark applications against 1.0. > > > > > > > > There are > > > > > > > > a few API changes in this release. Here are links to the > > > > > > > > associated > > > > > > > > upgrade guides - user facing changes have been kept as small as > > > > > > > > possible. > > > > > > > > > > > > > > > > changes to ML vector specification: > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 > > > > > > > > > > > > > > > > changes to the Java API: > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark > > > > > > > > > > > > > > > > changes to the streaming API: > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x > > > > > > > > > > > > > > > > changes to the GraphX API: > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 > > > > > > > > > > > > > > > > coGroup and related functions now return Iterable[T] instead of > > > > > Seq[T] > > > > > > > > ==> Call toSeq on the result to restore the old behavior > > > > > > > > > > > > > > > > SparkContext.jarOfClass returns Option[String] instead of > > > > > > > > Seq[String] > > > > > > > > ==> Call toSeq on the result to restore old behavior > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
