Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

- spun up an EC2 cluster successfully using spark-ec2 - tested S3 file access from that cluster successfully +1 ​ On Tue, Jul 29, 2014 at 1:46 AM, Henry Saputra wrote: > NOTICE and LICENSE files look good > Hashes and sigs look good > No executable in the source distribution > Compile so

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

NOTICE and LICENSE files look good Hashes and sigs look good No executable in the source distribution Compile source and run standalone +1 - Henry On Fri, Jul 25, 2014 at 4:08 PM, Tathagata Das wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.0.2. > > This

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

+1 Tested basic spark-shell and pyspark operations and MLlib examples on a Mac. On Mon, Jul 28, 2014 at 8:29 PM, Mubarak Seyed wrote: > +1 (non-binding) > > Tested this on Mac OS X. > > > On Mon, Jul 28, 2014 at 6:52 PM, Andrew Or wrote: > >> +1 Tested on standalone and yarn clusters >> >> >> 2

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

+1 (non-binding) Tested this on Mac OS X. On Mon, Jul 28, 2014 at 6:52 PM, Andrew Or wrote: > +1 Tested on standalone and yarn clusters > > > 2014-07-28 14:59 GMT-07:00 Tathagata Das : > > > Let me add my vote as well. > > Did some basic tests by running simple projects with various Spark > >

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

+1 Tested on standalone and yarn clusters 2014-07-28 14:59 GMT-07:00 Tathagata Das : > Let me add my vote as well. > Did some basic tests by running simple projects with various Spark > modules. Tested checksums. > > +1 > > On Sun, Jul 27, 2014 at 4:52 PM, Matei Zaharia > wrote: > > +1 > > > >

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

Let me add my vote as well. Did some basic tests by running simple projects with various Spark modules. Tested checksums. +1 On Sun, Jul 27, 2014 at 4:52 PM, Matei Zaharia wrote: > +1 > > Tested this on Mac OS X. > > Matei > > On Jul 25, 2014, at 4:08 PM, Tathagata Das > wrote: > >> Please vot

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

Right, the scenario is, for example, that a class is added in release 2.5.0, but has been back-ported to a 2.4.1-based release. 2.4.1 isn't missing anything from 2.4.1. But a version of "2.4.1" doesn't tell you whether or not the class is there reliably. By the way, I just found there is already s

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

We could also do this, though it would be great if the Hadoop project provided this version number as at least a baseline. It's up to distributors to decide which version they report but I imagine they won't remove stuff that's in the reported version number. Matei On Jul 27, 2014, at 1:57 PM,

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

+1 Tested this on Mac OS X. Matei On Jul 25, 2014, at 4:08 PM, Tathagata Das wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.0.2. > > This release fixes a number of bugs in Spark 1.0.1. > Some of the notable ones are > - SPARK-2452: Known issue is Spark

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

Good idea, although it gets difficult in the context of multiple distributions. Say change X is not present in version A, but present in version B. If you depend on X, what version can you look for to detect it? The distribution will return "A" or "A+X" or somesuch, but testing for "A" will give an

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

For this particular issue, it would be good to know if Hadoop provides an API to determine the Hadoop version. If not, maybe that can be added to Hadoop in its next release, and we can check for it with reflection. We recently added a SparkContext.version() method in Spark to let you tell the ve

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

Hey Ted, We always intend Spark to work with the newer Hadoop versions and encourage Spark users to use the newest Hadoop versions for best performance. We do try to be liberal in terms of supporting older versions as well. This is because many people run older HDFS versions and we want Spark to

Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

Thanks for replying, Patrick. The intention of my first email was for utilizing newer hadoop releases for their bug fixes. I am still looking for clean way of passing hadoop release version number to individual classes. Using newer hadoop releases would encourage pushing bug fixes / new features u

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

my vote is not binding. > > > > > -- Original -- > From: "Andrew Ash";; > Date: Mon, Jul 28, 2014 01:47 AM > To: "dev"; > > Subject: Re:[VOTE] Release Apache Spark 1.0.2 (RC1) > > > > Is that a re

Re:[VOTE] Release Apache Spark 1.0.2 (RC1)

‍It's not sure. I only tested 1.0.1, 1.0.2 (RC) version.‍ In addition, my vote is not binding.‍‍ -- Original -- From: "Andrew Ash";; Date: Mon, Jul 28, 2014 01:47 AM To: "dev"; Subject: Re:[VOTE] Release Apache Spark 1.0.2 (RC1

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

; -- Original -- >> From: "Tathagata Das";; >> Date: Sat, Jul 26, 2014 07:08 AM >> To: "dev@spark.apache.org"; >> >> Subject: [VOTE] Release Apache Spark 1.0.2 (RC1) >> >> >> >> Please vote on releas

Re:[VOTE] Release Apache Spark 1.0.2 (RC1)

s";; > Date: Sat, Jul 26, 2014 07:08 AM > To: "dev@spark.apache.org"; > > Subject: [VOTE] Release Apache Spark 1.0.2 (RC1) > > > > Please vote on releasing the following candidate as Apache Spark version > 1.0.2. > > This release fixes a number of b

Re:[VOTE] Release Apache Spark 1.0.2 (RC1)

-1 The following bug should be fixed: https://issues.apache.org/jira/browse/SPARK-2677‍ -- Original -- From: "Tathagata Das";; Date: Sat, Jul 26, 2014 07:08 AM To: "dev@spark.apache.org"; Subject: [VOTE] Release Apache Spark 1.0.2 (RC

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

Ted - technically I think you are correct, although I wouldn't recommend disabling this lock. This lock is not expensive (acquired once per task, as are many other locks already). Also, we've seen some cases where Hadoop concurrency bugs ended up requiring multiple fixes - concurrency of client acc

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

HADOOP-10456 is fixed in hadoop 2.4.1 Does this mean that synchronization on HadoopRDD.CONFIGURATION_INSTANTIATION_LOCK can be bypassed for hadoop 2.4.1 ? Cheers On Fri, Jul 25, 2014 at 6:00 PM, Patrick Wendell wrote: > The most important issue in this release is actually an ammendment to > a

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

OK, thanks for the clarification. 2014년 7월 25일 금요일, Michael Armbrust님이 작성한 메시지: > That query is looking at "Fix Version" not "Target Version". The fact that > the first one is still open is only because the bug is not resolved in > master. It is fixed in 1.0.2. The second one is partially fixe

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

The most important issue in this release is actually an ammendment to an earlier fix. The original fix caused a deadlock which was a regression from 1.0.0->1.0.1: Issue: https://issues.apache.org/jira/browse/SPARK-1097 1.0.1 Fix: https://github.com/apache/spark/pull/1273/files (had a deadlock) 1

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

That query is looking at "Fix Version" not "Target Version". The fact that the first one is still open is only because the bug is not resolved in master. It is fixed in 1.0.2. The second one is partially fixed in 1.0.2, but is not worth blocking the release for. On Fri, Jul 25, 2014 at 4:23 PM

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

TD, there are a couple of unresolved issues slated for 1.0.2 . Should they be edited somehow? On Fri, Jul 25, 2014 at 7:08 PM, Ta

[VOTE] Release Apache Spark 1.0.2 (RC1)

Please vote on releasing the following candidate as Apache Spark version 1.0.2. This release fixes a number of bugs in Spark 1.0.1. Some of the notable ones are - SPARK-2452: Known issue is Spark 1.0.1 caused by attempted fix for SPARK-1199. The fix was reverted for 1.0.2. - SPARK-2576: NoClassDef