Re: [VOTE] SPARK 2.4.0 (RC2)

[shane knapp]

>
> Using Scala 2.12.7 is a not an infra change, but change to the build,
> but again it's not even specific to 2.12.7. We should use the latest
> if we can though.
>
> yep, exactly.  we don't even have scala install on any of our jenkins
nodes.  this is all taken care of via build/mvn at runtime.
-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: [VOTE] SPARK 2.4.0 (RC2)

[Shixiong(Ryan) Zhu]

-1. Found an issue in a new 2.4 Java API:
https://issues.apache.org/jira/browse/SPARK-25644 We should fix it in 2.4.0
to avoid future breaking changes.

Best Regards,
Ryan


On Mon, Oct 1, 2018 at 7:22 PM Michael Heuer  wrote:

> FYI I’ve open two new issues against 2.4.0 rc2
>
> https://issues.apache.org/jira/browse/SPARK-25587
> https://issues.apache.org/jira/browse/SPARK-25588
>
> that are regressions against 2.3.1, and may also be present in 2.3.2.
> They could use triage or review.
>
>michael
>
>
> On Oct 1, 2018, at 9:18 PM, Wenchen Fan  wrote:
>
> This RC fails because of the correctness bug: SPARK-25538
>
> I'll start a new RC once the fix(
> https://github.com/apache/spark/pull/22602) is merged.
>
> Thanks,
> Wenchen
>
> On Tue, Oct 2, 2018 at 1:21 AM Sean Owen  wrote:
>
>> Given that this release is probably still 2 weeks from landing, I don't
>> think that waiting on a spark-tensorflow-connector release with TF 1.12 in
>> mid-October is a big deal. Users can use the library with Spark 2.3.x for a
>> week or two before upgrading, if that's the case. I think this kind of bug
>> fix is appropriate for a minor release, while I could see trying to work
>> around to keep the buggy behavior in a maintenance release.
>> On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng 
>> wrote:
>>
>>>
>>> IMHO, the use case (spark-tensorflow-connector) is very important. But
>>> whether we need to fix it in 2.4 branch depends on the release timeline.
>>> See my comment in the JIRA:
>>> https://issues.apache.org/jira/browse/SPARK-25378
>>>
>>>
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Michael Heuer]

FYI I’ve open two new issues against 2.4.0 rc2

https://issues.apache.org/jira/browse/SPARK-25587 

https://issues.apache.org/jira/browse/SPARK-25588 


that are regressions against 2.3.1, and may also be present in 2.3.2.  They 
could use triage or review.

   michael


> On Oct 1, 2018, at 9:18 PM, Wenchen Fan  wrote:
> 
> This RC fails because of the correctness bug: SPARK-25538
> 
> I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602 
> ) is merged.
> 
> Thanks,
> Wenchen
> 
> On Tue, Oct 2, 2018 at 1:21 AM Sean Owen  > wrote:
> Given that this release is probably still 2 weeks from landing, I don't think 
> that waiting on a spark-tensorflow-connector release with TF 1.12 in 
> mid-October is a big deal. Users can use the library with Spark 2.3.x for a 
> week or two before upgrading, if that's the case. I think this kind of bug 
> fix is appropriate for a minor release, while I could see trying to work 
> around to keep the buggy behavior in a maintenance release.
> On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng  > wrote:
> 
> IMHO, the use case (spark-tensorflow-connector) is very important. But 
> whether we need to fix it in 2.4 branch depends on the release timeline. See 
> my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378 
> 
> 

Re: [VOTE] SPARK 2.4.0 (RC2)

[Wenchen Fan]

This RC fails because of the correctness bug: SPARK-25538

I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602)
is merged.

Thanks,
Wenchen

On Tue, Oct 2, 2018 at 1:21 AM Sean Owen  wrote:

> Given that this release is probably still 2 weeks from landing, I don't
> think that waiting on a spark-tensorflow-connector release with TF 1.12 in
> mid-October is a big deal. Users can use the library with Spark 2.3.x for a
> week or two before upgrading, if that's the case. I think this kind of bug
> fix is appropriate for a minor release, while I could see trying to work
> around to keep the buggy behavior in a maintenance release.
> On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng  wrote:
>
>>
>> IMHO, the use case (spark-tensorflow-connector) is very important. But
>> whether we need to fix it in 2.4 branch depends on the release timeline.
>> See my comment in the JIRA:
>> https://issues.apache.org/jira/browse/SPARK-25378
>>
>>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Xiangrui Meng]

On Mon, Oct 1, 2018 at 9:52 AM Holden Karau  wrote:

> Oh that does look like an important correctness issue.
> -1
>
> On Mon, Oct 1, 2018, 9:57 AM Marco Gaido  wrote:
>
>> -1, I was able to reproduce SPARK-25538 with the provided data.
>>
>> Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu  ha
>> scritto:
>>
>>> +1
>>>
>>>  Original message 
>>> From: Denny Lee 
>>> Date: 9/30/18 10:30 PM (GMT-08:00)
>>> To: Stavros Kontopoulos 
>>> Cc: Sean Owen , Wenchen Fan ,
>>> dev 
>>> Subject: Re: [VOTE] SPARK 2.4.0 (RC2)
>>>
>>> +1 (non-binding)
>>>
>>>
>>> On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <
>>> stavros.kontopou...@lightbend.com> wrote:
>>>
>>>> +1
>>>>
>>>> Stavros
>>>>
>>>> On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
>>>>
>>>>> +1, with comments:
>>>>>
>>>>> There are 5 critical issues for 2.4, and no blockers:
>>>>> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>>> SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
>>>>> SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
>>>>> SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration
>>>>> guide
>>>>> SPARK-25323 ML 2.4 QA: API: Python API coverage
>>>>>
>>>>> Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?
>>>>>
>>>>
IMHO, the use case (spark-tensorflow-connector) is very important. But
whether we need to fix it in 2.4 branch depends on the release timeline.
See my comment in the JIRA:
https://issues.apache.org/jira/browse/SPARK-25378


>
>>>>> I found two issues resolved for 2.4.1 that got into this RC, so marked
>>>>> them as resolved in 2.4.0.
>>>>>
>>>>> I checked the licenses and notice and they look correct now in source
>>>>> and binary builds.
>>>>>
>>>>> The 2.12 artifacts are as I'd expect.
>>>>>
>>>>> I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
>>>>> -Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan 
>>>>> wrote:
>>>>> >
>>>>> > Please vote on releasing the following candidate as Apache Spark
>>>>> version 2.4.0.
>>>>> >
>>>>> > The vote is open until October 1 PST and passes if a majority +1 PMC
>>>>> votes are cast, with
>>>>> > a minimum of 3 +1 votes.
>>>>> >
>>>>> > [ ] +1 Release this package as Apache Spark 2.4.0
>>>>> > [ ] -1 Do not release this package because ...
>>>>> >
>>>>> > To learn more about Apache Spark, please see
>>>>> http://spark.apache.org/
>>>>> >
>>>>> > The tag to be voted on is v2.4.0-rc2 (commit
>>>>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>>>>> > https://github.com/apache/spark/tree/v2.4.0-rc2
>>>>> >
>>>>> > The release files, including signatures, digests, etc. can be found
>>>>> at:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>>>> >
>>>>> > Signatures used for Spark RCs can be found in this file:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>> >
>>>>> > The staging repository for this release can be found at:
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>>>> >
>>>>> > The documentation corresponding to this release can be found at:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>>>> >
>>>>> > The list of bug fixes going into 2.4.0 can be found at the following
>>>>> URL:
>>>>> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>>>> >
>>>>> > FAQ
>>>>> >
>>>>> > =
>>>>> > How can I help test this release?
>>>>> > =
>>>>>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Holden Karau]

Oh that does look like an important correctness issue.
-1

On Mon, Oct 1, 2018, 9:57 AM Marco Gaido  wrote:

> -1, I was able to reproduce SPARK-25538 with the provided data.
>
> Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu  ha
> scritto:
>
>> +1
>>
>>  Original message 
>> From: Denny Lee 
>> Date: 9/30/18 10:30 PM (GMT-08:00)
>> To: Stavros Kontopoulos 
>> Cc: Sean Owen , Wenchen Fan , dev
>> 
>> Subject: Re: [VOTE] SPARK 2.4.0 (RC2)
>>
>> +1 (non-binding)
>>
>>
>> On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <
>> stavros.kontopou...@lightbend.com> wrote:
>>
>>> +1
>>>
>>> Stavros
>>>
>>> On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
>>>
>>>> +1, with comments:
>>>>
>>>> There are 5 critical issues for 2.4, and no blockers:
>>>> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>> SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
>>>> SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
>>>> SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration
>>>> guide
>>>> SPARK-25323 ML 2.4 QA: API: Python API coverage
>>>>
>>>> Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?
>>>>
>>>> I found two issues resolved for 2.4.1 that got into this RC, so marked
>>>> them as resolved in 2.4.0.
>>>>
>>>> I checked the licenses and notice and they look correct now in source
>>>> and binary builds.
>>>>
>>>> The 2.12 artifacts are as I'd expect.
>>>>
>>>> I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
>>>> -Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan 
>>>> wrote:
>>>> >
>>>> > Please vote on releasing the following candidate as Apache Spark
>>>> version 2.4.0.
>>>> >
>>>> > The vote is open until October 1 PST and passes if a majority +1 PMC
>>>> votes are cast, with
>>>> > a minimum of 3 +1 votes.
>>>> >
>>>> > [ ] +1 Release this package as Apache Spark 2.4.0
>>>> > [ ] -1 Do not release this package because ...
>>>> >
>>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>>> >
>>>> > The tag to be voted on is v2.4.0-rc2 (commit
>>>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>>>> > https://github.com/apache/spark/tree/v2.4.0-rc2
>>>> >
>>>> > The release files, including signatures, digests, etc. can be found
>>>> at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>>> >
>>>> > Signatures used for Spark RCs can be found in this file:
>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>> >
>>>> > The staging repository for this release can be found at:
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>>> >
>>>> > The documentation corresponding to this release can be found at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>>> >
>>>> > The list of bug fixes going into 2.4.0 can be found at the following
>>>> URL:
>>>> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>>> >
>>>> > FAQ
>>>> >
>>>> > =
>>>> > How can I help test this release?
>>>> > =
>>>> >
>>>> > If you are a Spark user, you can help us test this release by taking
>>>> > an existing Spark workload and running on this release candidate, then
>>>> > reporting any regressions.
>>>> >
>>>> > If you're working in PySpark you can set up a virtual env and install
>>>> > the current RC and see if anything important breaks, in the Java/Scala
>>>> > you can add the staging repository to your projects resolvers and test
>>>> > with the RC (make sure to clean up the artifact cache before/after so
>>>> > you don't end up building with a out of date RC going forward).
>>>> >
>>>> > ===
>>>> > What should happen to JIRA tickets still targeting 2.4.0?
>>>> > ===
>>>> >
>>>> > The current list of open tickets targeted at 2.4.0 can be found at:
>>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>>> Version/s" = 2.4.0
>>>> >
>>>> > Committers should look at those and triage. Extremely important bug
>>>> > fixes, documentation, and API tweaks that impact compatibility should
>>>> > be worked on immediately. Everything else please retarget to an
>>>> > appropriate release.
>>>> >
>>>> > ==
>>>> > But my bug isn't fixed?
>>>> > ==
>>>> >
>>>> > In order to make timely releases, we will typically not hold the
>>>> > release unless the bug in question is a regression from the previous
>>>> > release. That being said, if there is something which is a regression
>>>> > that has not been correctly targeted please ping me or a committer to
>>>> > help target the issue.
>>>>
>>>> -
>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>>
>>>>
>>>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Marco Gaido]

-1, I was able to reproduce SPARK-25538 with the provided data.

Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu  ha
scritto:

> +1
>
>  Original message 
> From: Denny Lee 
> Date: 9/30/18 10:30 PM (GMT-08:00)
> To: Stavros Kontopoulos 
> Cc: Sean Owen , Wenchen Fan , dev <
> dev@spark.apache.org>
> Subject: Re: [VOTE] SPARK 2.4.0 (RC2)
>
> +1 (non-binding)
>
>
> On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <
> stavros.kontopou...@lightbend.com> wrote:
>
>> +1
>>
>> Stavros
>>
>> On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
>>
>>> +1, with comments:
>>>
>>> There are 5 critical issues for 2.4, and no blockers:
>>> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>> SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
>>> SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
>>> SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration
>>> guide
>>> SPARK-25323 ML 2.4 QA: API: Python API coverage
>>>
>>> Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?
>>>
>>> I found two issues resolved for 2.4.1 that got into this RC, so marked
>>> them as resolved in 2.4.0.
>>>
>>> I checked the licenses and notice and they look correct now in source
>>> and binary builds.
>>>
>>> The 2.12 artifacts are as I'd expect.
>>>
>>> I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
>>> -Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.
>>>
>>>
>>>
>>>
>>> On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.0.
>>> >
>>> > The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> > a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.0
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v2.4.0-rc2 (commit
>>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>>> > https://github.com/apache/spark/tree/v2.4.0-rc2
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> > https://repository.apache.org/content/repositories/orgapachespark-1287
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>> >
>>> > The list of bug fixes going into 2.4.0 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 2.4.0?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 2.4.0 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==
>>> > But my bug isn't fixed?
>>> > ==
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Ted Yu]

+1
 Original message From: Denny Lee  Date: 
9/30/18  10:30 PM  (GMT-08:00) To: Stavros Kontopoulos 
 Cc: Sean Owen , Wenchen 
Fan , dev  Subject: Re: [VOTE] SPARK 
2.4.0 (RC2) 
+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos 
 wrote:
+1
Stavros
On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
+1, with comments:



There are 5 critical issues for 2.4, and no blockers:

SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4

SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs

SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella

SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide

SPARK-25323 ML 2.4 QA: API: Python API coverage



Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?



I found two issues resolved for 2.4.1 that got into this RC, so marked

them as resolved in 2.4.0.



I checked the licenses and notice and they look correct now in source

and binary builds.



The 2.12 artifacts are as I'd expect.



I ran all tests for 2.11 and 2.12 and they pass with -Pyarn

-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.









On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan  wrote:

>

> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.0.

>

> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> are cast, with

> a minimum of 3 +1 votes.

>

> [ ] +1 Release this package as Apache Spark 2.4.0

> [ ] -1 Do not release this package because ...

>

> To learn more about Apache Spark, please see http://spark.apache.org/

>

> The tag to be voted on is v2.4.0-rc2 (commit 
> 42f25f309e91c8cde1814e3720099ac1e64783da):

> https://github.com/apache/spark/tree/v2.4.0-rc2

>

> The release files, including signatures, digests, etc. can be found at:

> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/

>

> Signatures used for Spark RCs can be found in this file:

> https://dist.apache.org/repos/dist/dev/spark/KEYS

>

> The staging repository for this release can be found at:

> https://repository.apache.org/content/repositories/orgapachespark-1287

>

> The documentation corresponding to this release can be found at:

> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/

>

> The list of bug fixes going into 2.4.0 can be found at the following URL:

> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0

>

> FAQ

>

> =

> How can I help test this release?

> =

>

> If you are a Spark user, you can help us test this release by taking

> an existing Spark workload and running on this release candidate, then

> reporting any regressions.

>

> If you're working in PySpark you can set up a virtual env and install

> the current RC and see if anything important breaks, in the Java/Scala

> you can add the staging repository to your projects resolvers and test

> with the RC (make sure to clean up the artifact cache before/after so

> you don't end up building with a out of date RC going forward).

>

> ===

> What should happen to JIRA tickets still targeting 2.4.0?

> ===

>

> The current list of open tickets targeted at 2.4.0 can be found at:

> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.0

>

> Committers should look at those and triage. Extremely important bug

> fixes, documentation, and API tweaks that impact compatibility should

> be worked on immediately. Everything else please retarget to an

> appropriate release.

>

> ==

> But my bug isn't fixed?

> ==

>

> In order to make timely releases, we will typically not hold the

> release unless the bug in question is a regression from the previous

> release. That being said, if there is something which is a regression

> that has not been correctly targeted please ping me or a committer to

> help target the issue.



-

To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC2)

[Denny Lee]

+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> wrote:

> +1
>
> Stavros
>
> On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
>
>> +1, with comments:
>>
>> There are 5 critical issues for 2.4, and no blockers:
>> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>> SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
>> SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
>> SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
>> SPARK-25323 ML 2.4 QA: API: Python API coverage
>>
>> Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?
>>
>> I found two issues resolved for 2.4.1 that got into this RC, so marked
>> them as resolved in 2.4.0.
>>
>> I checked the licenses and notice and they look correct now in source
>> and binary builds.
>>
>> The 2.12 artifacts are as I'd expect.
>>
>> I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
>> -Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.
>>
>>
>>
>>
>> On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan  wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.0.
>> >
>> > The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> > a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.0
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v2.4.0-rc2 (commit
>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>> > https://github.com/apache/spark/tree/v2.4.0-rc2
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1287
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>> >
>> > The list of bug fixes going into 2.4.0 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 2.4.0?
>> > ===
>> >
>> > The current list of open tickets targeted at 2.4.0 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Stavros Kontopoulos]

+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:

> +1, with comments:
>
> There are 5 critical issues for 2.4, and no blockers:
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
> SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
> SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
> SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
> SPARK-25323 ML 2.4 QA: API: Python API coverage
>
> Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?
>
> I found two issues resolved for 2.4.1 that got into this RC, so marked
> them as resolved in 2.4.0.
>
> I checked the licenses and notice and they look correct now in source
> and binary builds.
>
> The 2.12 artifacts are as I'd expect.
>
> I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
> -Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.
>
>
>
>
> On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan  wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC
> votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc2 (commit
> 42f25f309e91c8cde1814e3720099ac1e64783da):
> > https://github.com/apache/spark/tree/v2.4.0-rc2
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1287
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Sean Owen]

+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 
> 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC2)

[Sean Owen]

Go ahead and file a JIRA to update to 2.12.7 with these details. We'll
assess whether it is a blocker.

On Fri, Sep 28, 2018 at 12:09 PM Darcy Shen  wrote:
>
> I agree it is a non-important Spark bug. I mean the Option and String 
> comparison. The bug is easy to fix and obvious to confirm. If the desc of PR 
> may not be accurate，feel free to edit the title or content. I am on a 
> vocation from 9.29   ：）
>
> But the scala bug of WrappedArray is severe. We should not provide a 
> pre-built spark packages with Scala 2.12.6 . The bug is not about compiler. 
> But about scala-library.
>
> If the prebuilt packages of Spark use scala-library 2.12.6 ， the bug exists 
> whatever scala version our application developer use.
>
> For Spark we should be serious about the minor Scala version. A preferred 
> Scala minor version should be officially stated.
>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC2)

[Darcy Shen]

I agree it is a non-important Spark bug. I mean the Option and 
String comparison. The bug is easy to fix and obvious to confirm. If the desc 
of PR may not be accurate，feel free to edit the title or content. I am on a 
vocation from 9.29   ：）But the scala bug of WrappedArray is severe. We should 
not provide a pre-built spark packages with Scala 2.12.6 . The bug is not about 
compiler. But about scala-library.If the prebuilt packages of Spark use 
scala-library 2.12.6 ， the bug exists whatever scala version our application 
developer use.For Spark we should be serious about the minor Scala version. A 
preferred Scala minor version should be officially stated. （hi wenchen sorry 
for the duplication of email ，just forgot to cc the list） On Fri, 28 Sep 
2018 22:38:05 +0800  Wenchen Fan wrote I don't think 
this bug is so serious to fail an RC, it's only about metrics IIUC, and it's 
not a regression in 2.4.I agree we should backport this fix to 2.3 and 2.4, and 
we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do 
you know how to change it?BTW end users can still use scala 2.12.7 with the 
Spark package built with scala 2.12.6, right?On Fri, Sep 28, 2018 at 4:22 PM 
Darcy Shen  wrote:-1 
see:https://github.com/apache/spark/pull/22577We should make sure that Spark 
works with Scala 2.12.7 .https://github.com/scala/bug/issues/11123This resolved 
bug of Scala 2.12.6 is severe and related to correctness.We should warn our 
aggressive users about the Scala version. Latest Scala (2.12.7) is preferred 
and should pass the unit tests at least. On Fri, 28 Sep 2018 10:59:41 +0800 
Wenchen Fan  wrote Please vote on releasing the 
following candidate as Apache Spark version 2.4.0.The vote is open until 
October 1 PST and passes if a majority +1 PMC votes are cast, witha minimum of 
3 +1 votes.[ ] +1 Release this package as Apache Spark 2.4.0[ ] -1 Do not 
release this package because ...To learn more about Apache Spark, please see 
http://spark.apache.org/The tag to be voted on is v2.4.0-rc2 (commit 
42f25f309e91c8cde1814e3720099ac1e64783da):https://github.com/apache/spark/tree/v2.4.0-rc2The
 release files, including signatures, digests, etc. can be found 
at:https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/Signatures used 
for Spark RCs can be found in this 
file:https://dist.apache.org/repos/dist/dev/spark/KEYSThe staging repository 
for this release can be found 
at:https://repository.apache.org/content/repositories/orgapachespark-1287The 
documentation corresponding to this release can be found 
at:https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/The list of bug 
fixes going into 2.4.0 can be found at the following 
URL:https://issues.apache.org/jira/projects/SPARK/versions/2.4.0FAQ=How
 can I help test this release?=If you are a Spark user, 
you can help us test this release by takingan existing Spark workload and 
running on this release candidate, thenreporting any regressions.If you're 
working in PySpark you can set up a virtual env and installthe current RC and 
see if anything important breaks, in the Java/Scalayou can add the staging 
repository to your projects resolvers and testwith the RC (make sure to clean 
up the artifact cache before/after soyou don't end up building with a out of 
date RC going forward).===What should 
happen to JIRA tickets still targeting 
2.4.0?===The current list of open 
tickets targeted at 2.4.0 can be found 
at:https://issues.apache.org/jira/projects/SPARK and search for "Target 
Version/s" = 2.4.0Committers should look at those and triage. Extremely 
important bugfixes, documentation, and API tweaks that impact compatibility 
shouldbe worked on immediately. Everything else please retarget to 
anappropriate release.==But my bug isn't 
fixed?==In order to make timely releases, we will typically not 
hold therelease unless the bug in question is a regression from the 
previousrelease. That being said, if there is something which is a 
regressionthat has not been correctly targeted please ping me or a committer 
tohelp target the issue. 

Re: [VOTE] SPARK 2.4.0 (RC2)

[Dongjoon Hyun]

Hi, Wenchen.

The current issue link seems to be out of order for me.

The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.0

Could you send out with the following issue link for next RCs?

https://issues.apache.org/jira/projects/SPARK/versions/12342385

Bests,
Dongjoon.

On Thu, Sep 27, 2018 at 8:00 PM Wenchen Fan  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit
> 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Sean Owen]

I don't even know how or if this manifests as a bug. The code is
indeed incorrect and the 2.12 compiler flags it. We fixed a number of
these in SPARK-25398. While I want to get this into 2.4 if we have
another RC, I don't see evidence this is a blocker. It is not specific
to Scala 2.12.

Using Scala 2.12.7 is a not an infra change, but change to the build,
but again it's not even specific to 2.12.7. We should use the latest
if we can though.

On Fri, Sep 28, 2018 at 9:38 AM Wenchen Fan  wrote:
>
> I don't think this bug is so serious to fail an RC, it's only about metrics 
> IIUC, and it's not a regression in 2.4.
>
> I agree we should backport this fix to 2.3 and 2.4, and we should update our 
> scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to 
> change it?
>
> BTW end users can still use scala 2.12.7 with the Spark package built with 
> scala 2.12.6, right?
>
> On Fri, Sep 28, 2018 at 4:22 PM Darcy Shen  wrote:
>>
>> -1
>>
>> see:
>>
>> https://github.com/apache/spark/pull/22577
>>
>> We should make sure that Spark works with Scala 2.12.7 .
>>
>> https://github.com/scala/bug/issues/11123
>>
>> This resolved bug of Scala 2.12.6 is severe and related to correctness.
>>
>> We should warn our aggressive users about the Scala version.
>> Latest Scala (2.12.7) is preferred and should pass the unit tests at least.
>>
>>
>>  On Fri, 28 Sep 2018 10:59:41 +0800 Wenchen Fan  
>> wrote 
>>
>> Please vote on releasing the following candidate as Apache Spark version 
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
>> are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc2 (commit 
>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>> https://github.com/apache/spark/tree/v2.4.0-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target 
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>>
>>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 2.4.0 (RC2)

[Wenchen Fan]

I don't think this bug is so serious to fail an RC, it's only about metrics
IIUC, and it's not a regression in 2.4.

I agree we should backport this fix to 2.3 and 2.4, and we should update
our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how
to change it?

BTW end users can still use scala 2.12.7 with the Spark package built with
scala 2.12.6, right?

On Fri, Sep 28, 2018 at 4:22 PM Darcy Shen  wrote:

> -1
>
> see:
>
> *https://github.com/apache/spark/pull/22577
> *
>
> We should make sure that Spark works with Scala 2.12.7 .
>
> https://github.com/scala/bug/issues/11123
>
> This resolved bug of Scala 2.12.6 is severe and related to correctness.
>
> We should warn our aggressive users about the Scala version.
> Latest Scala (2.12.7) is preferred and should pass the unit tests at least.
>
>
>  On Fri, 28 Sep 2018 10:59:41 +0800 *Wenchen Fan  >* wrote 
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit
> 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
>
>
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Wenchen Fan]

Yes, that's proposed by Sean. This time we should publish a Scala 2.12
build, both in maven and the download page.

On Fri, Sep 28, 2018 at 11:34 AM Saisai Shao  wrote:

> Only "without-hadoop" profile has 2.12 binary, is it expected?
>
> Thanks
> Saisai
>
> Wenchen Fan  于2018年9月28日周五 上午11:08写道：
>
>> I'm adding my own +1, since all the problems mentioned in the RC1 voting
>> email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.
>>
>> On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan  wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.4.0.
>>>
>>> The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.0-rc2 (commit
>>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>>> https://github.com/apache/spark/tree/v2.4.0-rc2
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>>
>>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>>
>>> FAQ
>>>
>>> =
>>> How can I help test this release?
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 2.4.0?
>>> ===
>>>
>>> The current list of open tickets targeted at 2.4.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Saisai Shao]

Only "without-hadoop" profile has 2.12 binary, is it expected?

Thanks
Saisai

Wenchen Fan  于2018年9月28日周五 上午11:08写道：

> I'm adding my own +1, since all the problems mentioned in the RC1 voting
> email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.
>
> On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc2 (commit
>> 42f25f309e91c8cde1814e3720099ac1e64783da):
>> https://github.com/apache/spark/tree/v2.4.0-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>

Re: [VOTE] SPARK 2.4.0 (RC2)

[Wenchen Fan]

I'm adding my own +1, since all the problems mentioned in the RC1 voting
email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.

On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit
> 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.0?
> ===
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>

[VOTE] SPARK 2.4.0 (RC2)

[Wenchen Fan]

Please vote on releasing the following candidate as Apache Spark version
2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes
are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit
42f25f309e91c8cde1814e3720099ac1e64783da):
https://github.com/apache/spark/tree/v2.4.0-rc2

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1287

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/

The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.0

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting 2.4.0?
===

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.