Re: [VOTE] Release Apache Spark 2.4.2

: Tuesday, April 30, 2019 1:52:53 PM To: Reynold Xin Cc: Jungtaek Lim; Dongjoon Hyun; Wenchen Fan; Michael Heuer; Terry Kim; dev; Xiao Li Subject: Re: [VOTE] Release Apache Spark 2.4.2 FWIW I'm OK with this even though I proposed the backport PR for discussion. It really is a tough call

Re: [VOTE] Release Apache Spark 2.4.2

Hi, All and Xiao (as a next release manager). In any case, can the release manager include the information about the used release script as a part of VOTE email officially? That information will be very helpful to reproduce Spark build (in the downstream environment) Currently, it's not clearly

Re: [VOTE] Release Apache Spark 2.4.2

> it could just be fixed in master rather than back-port and re-roll the RC I don't think the release script is part of the released product. That said, we can just fix the release script in branch 2.4 without creating a new RC. We can even create a new repo for the release script, like

Re: [VOTE] Release Apache Spark 2.4.2

I think this is a reasonable idea; I know @vanzin had suggested it was simpler to use the latest in case a bug was found in the release script and then it could just be fixed in master rather than back-port and re-roll the RC. That said I think we did / had to already drop the ability to build <=

Re: [VOTE] Release Apache Spark 2.4.2

As Scala 2.11 is the default for 2.4.x, we currently include _2.11 artifacts in our release. Our Python library depends on the Scala artifacts. With Homebrew, we have -submit and -shell script wrappers for spark-submit and spark-shell, and those will break at runtime if we're using _2.11

Re: [VOTE] Release Apache Spark 2.4.2

Yeah I don't think the pyspark change was intentional; I'm trying to help assess what the impact is though. It may be a dumb question, but, what problem does the change cause? is it beyond what I mentioned below? you have a project with interdependent Python and Scala components? On Fri, Apr 26,

Re: [VOTE] Release Apache Spark 2.4.2

We certainly can't be the only project downstream of Spark that includes Scala versioned artifacts in our release. Our python library on PyPI depends on pyspark, our Bioconda recipe depends on the pyspark Conda recipe, and our Homebrew formula depends on the apache-spark Homebrew formula.

Re: [VOTE] Release Apache Spark 2.4.2

To be clear, what's the nature of the problem there... just Pyspark apps that are using a Scala-based library? Trying to make sure we understand what is and isn't a problem here. On Fri, Apr 26, 2019 at 9:44 AM Michael Heuer wrote: > This will also cause problems in Conda builds that depend on

Re: [VOTE] Release Apache Spark 2.4.2

This will also cause problems in Conda builds that depend on pyspark https://anaconda.org/conda-forge/pyspark and Homebrew builds that depend on apache-spark, as that also uses the binary distribution.

Re: [VOTE] Release Apache Spark 2.4.2

Re: .NET, what's the particular issue in there that it's causing? 2.4.2 still builds for 2.11. I'd imagine you'd be pulling dependencies from Maven central (?) or if needed can build for 2.11 from source. I'm more concerned about pyspark because it builds in 2.12 jars. On Fri, Apr 26, 2019 at

Re: [VOTE] Release Apache Spark 2.4.2

I do feel it'd be better to not switch default Scala versions in a minor release. I don't know how much downstream this impacts. Dotnet is a good data point. Anybody else hit this issue? On Thu, Apr 25, 2019 at 11:36 PM, Terry Kim < yumin...@gmail.com > wrote: > > > > Very much interested

Re: [VOTE] Release Apache Spark 2.4.2

Very much interested in hearing what you folks decide. We currently have a couple asking us questions at https://github.com/dotnet/spark/issues. Thanks, Terry -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: [VOTE] Release Apache Spark 2.4.2

+1 I have tested it and looks good! Best Regards, Ryan On Sun, Apr 21, 2019 at 8:49 PM Wenchen Fan wrote: > Yea these should be mentioned in the 2.4.1 release notes. > > It seems we only have one ticket that is labeled as "release-notes" for > 2.4.2:

Re: [VOTE] Release Apache Spark 2.4.2

Yea these should be mentioned in the 2.4.1 release notes. It seems we only have one ticket that is labeled as "release-notes" for 2.4.2: https://issues.apache.org/jira/browse/SPARK-27419 . I'll mention it when I write release notes. On Mon, Apr 22, 2019 at 5:46 AM Sean Owen wrote: > One minor

Re: [VOTE] Release Apache Spark 2.4.2

One minor comment: for 2.4.1 we had a couple JIRAs marked 'release-notes': https://issues.apache.org/jira/browse/SPARK-27198?jql=project%20%3D%20SPARK%20and%20fixVersion%20%20in%20(2.4.1%2C%202.4.2)%20and%20labels%20%3D%20%27release-notes%27 They should be mentioned in

Re: [VOTE] Release Apache Spark 2.4.2

* Wenchen Fan > *Cc:* Spark dev list > *Subject:* Re: [VOTE] Release Apache Spark 2.4.2 > > +1 from me too. > > It seems like there is support for merging the Jackson change into > 2.4.x (and, I think, a few more minor dependency updates) but this > doesn't have to go into

Re: [VOTE] Release Apache Spark 2.4.2

+1 R tests, package tests on r-hub. Manually check commits under R, doc etc From: Sean Owen Sent: Saturday, April 20, 2019 11:27 AM To: Wenchen Fan Cc: Spark dev list Subject: Re: [VOTE] Release Apache Spark 2.4.2 +1 from me too. It seems like

Re: [VOTE] Release Apache Spark 2.4.2

+1 from me too. It seems like there is support for merging the Jackson change into 2.4.x (and, I think, a few more minor dependency updates) but this doesn't have to go into 2.4.2. That said, if there is another RC for any reason, I think we could include it. Otherwise can wait for 2.4.3. On

Re: [VOTE] Release Apache Spark 2.4.2

-1, as i'd like to be sure that the python test infra change for jenkins is included (https://github.com/apache/spark/pull/24379) On Fri, Apr 19, 2019 at 12:01 PM Michael Armbrust wrote: > +1 (binding), we've test this and it LGTM. > > On Thu, Apr 18, 2019 at 7:51 PM Wenchen Fan wrote: > >>

Re: [VOTE] Release Apache Spark 2.4.2

+1 (binding), we've test this and it LGTM. On Thu, Apr 18, 2019 at 7:51 PM Wenchen Fan wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.2. > > The vote is open until April 23 PST and passes if a majority +1 PMC votes > are cast, with > a minimum of 3 +1

[VOTE] Release Apache Spark 2.4.2

Please vote on releasing the following candidate as Apache Spark version 2.4.2. The vote is open until April 23 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.2 [ ] -1 Do not release this package because ... To