[RESULT] [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-17 Thread Patrick Wendell
The vote is now closed. This vote passes with 4 IPMC +1's and no 0 or -1 votes. +1 (4 Total) Marvin Humphrey Henry Saputra Chris Mattmann Roman Shaposhnik 0 (0 Total) -1 (0 Total) * = Binding Vote Thanks to everyone who helped vet this release. - Patrick

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-16 Thread Azuryy Yu
cember 16, 2013 10:48 AM > >>> To: dev@spark.incubator.apache.org > >>> Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) > >>> > >>> Hi Azuryy > >>> > >>> Please Check https://spark-project.atlassian.net/browse/SPARK-

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-16 Thread Matei Zaharia
gt;> problem, you might need to build mesos against 2.5.0, I don't test that, if >> you got time, mind take a test? >>> >>> Best Regards, >>> Raymond Liu >>> >>> >>> -Original Message- >>> From: Liu, Raymond [mailt

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Azuryy Yu
> > Best Regards, > > Raymond Liu > > > > > > -Original Message- > > From: Liu, Raymond [mailto:raymond....@intel.com] > > Sent: Monday, December 16, 2013 10:48 AM > > To: dev@spark.incubator.apache.org > > Subject: RE: [VOTE] Release Apache

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Matei Zaharia
; > > -Original Message- > From: Liu, Raymond [mailto:raymond@intel.com] > Sent: Monday, December 16, 2013 10:48 AM > To: dev@spark.incubator.apache.org > Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) > > Hi Azuryy > > Please Check https

RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Liu, Raymond
0:48 AM To: dev@spark.incubator.apache.org Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) Hi Azuryy Please Check https://spark-project.atlassian.net/browse/SPARK-995 for this protobuf version issue Best Regards, Raymond Liu -Original Message- From: Azuryy Yu [ma

RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Liu, Raymond
: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) Hi here, Do we have plan to upgrade protobuf from 2.4.1 to 2.5.0? PB has some uncompatable API between these two versions. Hadoop-2.x using protobuf-2.5.0 but if some guys want to run Spark on mesos, then mesos using protobuf-2.4.1 currently

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Azuryy Yu
Hi here, Do we have plan to upgrade protobuf from 2.4.1 to 2.5.0? PB has some uncompatable API between these two versions. Hadoop-2.x using protobuf-2.5.0 but if some guys want to run Spark on mesos, then mesos using protobuf-2.4.1 currently. so we may discuss here for a better solution. On Mo

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Azuryy Yu
Thanks Patrick. On 16 Dec 2013 02:43, "Patrick Wendell" wrote: > You can checkout the docs mentioned in the vote thread. There is also > a pre-build binary for hadoop2 that is compiled for YARN 2.2 > > - Patrick > > On Sun, Dec 15, 2013 at 4:31 AM, Azuryy Yu wrote: > > yarn 2.2, not yarn 0.22, I

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Patrick Wendell
You can checkout the docs mentioned in the vote thread. There is also a pre-build binary for hadoop2 that is compiled for YARN 2.2 - Patrick On Sun, Dec 15, 2013 at 4:31 AM, Azuryy Yu wrote: > yarn 2.2, not yarn 0.22, I am so sorry. > > > On Sun, Dec 15, 2013 at 8:31 PM, Azuryy Yu wrote: > >> H

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Azuryy Yu
Hi, Spark-0.8.1 supports yarn 0.22 right? where to find the release note? Thanks. On Sun, Dec 15, 2013 at 3:20 AM, Henry Saputra wrote: > Yeah seems like it. He was ok with our prev release. > Let's wait for his reply > > On Saturday, December 14, 2013, Patrick Wendell wrote: > > > Henry - from

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-15 Thread Azuryy Yu
yarn 2.2, not yarn 0.22, I am so sorry. On Sun, Dec 15, 2013 at 8:31 PM, Azuryy Yu wrote: > Hi, > Spark-0.8.1 supports yarn 0.22 right? where to find the release note? > Thanks. > > > On Sun, Dec 15, 2013 at 3:20 AM, Henry Saputra wrote: > >> Yeah seems like it. He was ok with our prev release.

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Henry Saputra
Yeah seems like it. He was ok with our prev release. Let's wait for his reply On Saturday, December 14, 2013, Patrick Wendell wrote: > Henry - from that thread it looks like sebb's concern was something > different than this. > > On Sat, Dec 14, 2013 at 11:08 AM, Henry Saputra > wrote: > > Hi Pa

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Patrick Wendell
Henry - from that thread it looks like sebb's concern was something different than this. On Sat, Dec 14, 2013 at 11:08 AM, Henry Saputra wrote: > Hi Patrick, > > Yeap I agree, but technically ASF VOTE release on source only, there > even debate about it =), so putting it in the vote staging artif

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Henry Saputra
Hi Patrick, Yeap I agree, but technically ASF VOTE release on source only, there even debate about it =), so putting it in the vote staging artifact could confuse people because in our case we do package 3rd party libraries in the binary jars. I have sent email to sebb asking clarification about

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Patrick Wendell
Hey Henry, One thing a lot of people do during the vote is test the binaries and make sure they work. This is really valuable. If you'd like I could add a caveat to the vote thread explaining that we are only voting on the source. - Patrick On Sat, Dec 14, 2013 at 10:40 AM, Henry Saputra wrote:

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Henry Saputra
Actually we should be fine putting the binaries there as long as the VOTE is for the source. Let's verify with sebb in the general@ list about his concern. - Henry On Sat, Dec 14, 2013 at 10:31 AM, Henry Saputra wrote: > Hi Patrick, as sebb has mentioned let's move the binaries from the > votin

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-14 Thread Henry Saputra
Hi Patrick, as sebb has mentioned let's move the binaries from the voting directory in your people.apache.org directory. ASF release voting is for source code and not binaries, and technically we provide binaries for convenience. And add link to the KEYS location in the dist[1] to let verify signa

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-13 Thread Patrick Wendell
The vote is now closed. This vote passes with 5 PPMC +1's and no 0 or -1 votes. +1 (5 Total) Matei Zaharia* Nick Pentreath* Patrick Wendell* Prashant Sharma* Tom Graves* 0 (0 Total) -1 (0 Total) * = Binding Vote As per the incubator release guide [1] I'll be sending this to the general incubat

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-12 Thread Evan Chan
I'd be personally fine with a standard workflow of assemble-deps + packaging just the Spark files as separate packages, if it speeds up everyone's development time. On Wed, Dec 11, 2013 at 1:10 PM, Mark Hamstra wrote: > I don't know how to make sense of the numbers, but here's what I've got > fr

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-12 Thread Tom Graves
+1.  Built spark on yarn for both hadoop 0.23 and hadoop 2.2.0 on redhat linux using maven.  Ran some tests on both a secure Hadoop 0.23 cluster and a secure Hadoop 2.2.0 cluster.  Verified signatures and md5. Tom On Tuesday, December 10, 2013 6:49 PM, Patrick Wendell wrote: Please vote

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-12 Thread Tom Graves
I re-downloaded the source tarball and it works now. Tom On Wednesday, December 11, 2013 6:27 PM, Patrick Wendell wrote: I also talked to a few people who got corrupted binaries when downloading from the people.apache HTTP. In that case the checksum failed but if they re-downloaded it worke

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-12 Thread Prashant Sharma
+1, built and tested on linux. On Thu, Dec 12, 2013 at 5:57 AM, Patrick Wendell wrote: > I also talked to a few people who got corrupted binaries when > downloading from the people.apache HTTP. In that case the checksum > failed but if they re-downloaded it worked. So maybe just re-download > a

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Patrick Wendell
I also talked to a few people who got corrupted binaries when downloading from the people.apache HTTP. In that case the checksum failed but if they re-downloaded it worked. So maybe just re-download and try again? On Wed, Dec 11, 2013 at 3:15 PM, Patrick Wendell wrote: > Hey Tom, > > I re-verifie

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Patrick Wendell
Hey Tom, I re-verified the signatures and got someone else to do it. It seemed fine. Here is what I did. gpg --recv-key 9E4FE3AF wget http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/spark-0.8.1-incubating.tgz.asc wget http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/s

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Mark Hamstra
I don't know how to make sense of the numbers, but here's what I've got from a very small sample size. For both v0.8.0-incubating and v0.8.1-incubating, building separate assemblies is faster than `./sbt/sbt assembly` and the times for building separate assemblies for 0.8.0 and 0.8.1 are about the

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Tom Graves
Hey folks, I'm trying to verify the signature on the rc4 but am getting a BAD signature, is it working for others?  Perhaps I messed up the import of the asc file. $ gpg --verify spark-0.8.1-incubating.tgz.asc spark-0.8.1-incubating.tgz gpg: Signature made Tue 10 Dec 2013 10:53:15 PM UTC using R

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Taka Shinagawa
On my Core 2 Duo (slower) Mac, I'm having the same problem with building v0.8.0-incubating as well. I'm suspecting the slow build problem is caused by sbt-assembly as reported here. https://github.com/sbt/sbt-assembly/issues/68 I tried to upgrade sbt-assembly to version 0.10.1 (and sbt to 0.13.0)

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Patrick Wendell
I'll +1 myself also. For anyone who has the slow build problem: does this issue happen when building v0.8.0-incubating also? Trying to figure out whether it's related to something we added in 0.8.1 or if it's a long standing issue. - Patrick On Wed, Dec 11, 2013 at 10:39 AM, Matei Zaharia wrote

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Matei Zaharia
Woah, weird, but definitely good to know. If you’re doing Spark development, there’s also a more convenient option added by Shivaram in the master branch. You can do sbt assemble-deps to package *just* the dependencies of each project in a special assembly JAR, and then use sbt compile to updat

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Nick Pentreath
- Successfully built via sbt/sbt assembly/assembly on Mac OS X, as well as on a dev Ubuntu EC2 box - Successfully tested via sbt/sbt test locally - Successfully built and tested using mvn package locally - I've tested my own Spark jobs (built against 0.8.0-incubating) on this RC a

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Prashant Sharma
I hope this PR https://github.com/apache/incubator-spark/pull/252 can help. Again this is not a blocker for the release from my side either. On Wed, Dec 11, 2013 at 2:14 PM, Mark Hamstra wrote: > Interesting, and confirmed: On my machine where `./sbt/sbt assembly` takes > a long, long, long

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Mark Hamstra
Interesting, and confirmed: On my machine where `./sbt/sbt assembly` takes a long, long, long time to complete (a MBP, in my case), building three separate assemblies (`./sbt/sbt assembly/assembly`, `./sbt/sbt examples/assembly`, `./sbt/sbt tools/assembly`) takes much, much less time. On Wed

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Taka Shinagawa
That's a good point. Although it's definitely not a blocker for this release, it would be more user friendly to mention sbt/sbt assembly/assembly as well as Maven build instructions in the README and quick-start files (at least until the time-consuming sbt assembly process on a regular hardware get

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Prashant Sharma
forgot to mention, after running sbt/sbt assembly/assembly running sbt/sbt examples/assembly takes just 37s. Not to mention my hardware is not really great. On Wed, Dec 11, 2013 at 1:28 PM, Prashant Sharma wrote: > Hi Patrick and Matei, > > Was trying out this and followed the quick start guide

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-11 Thread Prashant Sharma
Hi Patrick and Matei, Was trying out this and followed the quick start guide which says do sbt/sbt assembly, like few others I was also stuck for few minutes on linux. On the other hand if I use sbt/sbt assembly/assembly it is much faster. Should we change the documentation to reflect this. It wi

Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-10 Thread Matei Zaharia
+1 Built and tested it on Mac OS X. Matei On Dec 10, 2013, at 4:49 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark > (incubating) version 0.8.1. > > The tag to be voted on is v0.8.1-incubating (commit b87d31d): > https://git-wip-us.apache.org/rep

[VOTE] Release Apache Spark 0.8.1-incubating (rc4)

2013-12-10 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark (incubating) version 0.8.1. The tag to be voted on is v0.8.1-incubating (commit b87d31d): https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=commit;h=b87d31dd8eb4b4e47c0138e9242d0dd6922c8c4e The rel