Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

I plan to cut RC 1 for Spark 3.1.1 this friday or next monday. Let me know
if there are any blocker.

Thanks guys.

2021년 1월 12일 (화) 오전 1:22, Dongjoon Hyun 님이 작성:

> Thank you, Hyukjin!
>
> Bests,
> Dongjoon.
>
> On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon  wrote:
>
>> I had a response from the INFRA team and Sonatype. Just to share, the
>> removal is possible as an exception, but it's best to go ahead for 3.1.1
>> for safety as we all discussed.
>> There are several non-regression but correctness issues being tracked
>> under https://issues.apache.org/jira/browse/SPARK-33507 that contains
>> few tickets Dongjoon mentioned.
>> I will monitor this ticket and proceed for 3.1.1 RC1 when there are no
>> more standing issues under this ticket.
>> In addition, I made a PR to share this to Spark users and community (
>> https://github.com/apache/spark-website/pull/301), and it's merged and
>> visible now (
>> https://spark.apache.org/news/next-official-release-spark-3.1.1.html)
>>
>> Thank you all for your support and feedback!
>>
>> 2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski 님이 작성:
>>
>>> Hi Sean,
>>>
>>> +1 to leave it. Makes so much more sense (as that's what really happened
>>> and the history of Apache Spark is...irreversible).
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> 
>>> https://about.me/JacekLaskowski
>>> "The Internals Of" Online Books 
>>> Follow me on https://twitter.com/jaceklaskowski
>>>
>>> 
>>>
>>>
>>> On Thu, Jan 7, 2021 at 3:44 PM Sean Owen  wrote:
>>>
 While we can delete the tag, maybe just leave it. As a general rule we
 would not remove anything pushed to the main git repo.

 On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:

> Hi,
>
> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead
> with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and
> we'll see 3.1.1-rc1?
>
> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Dongjoon Hyun]

Thank you, Hyukjin!

Bests,
Dongjoon.

On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon  wrote:

> I had a response from the INFRA team and Sonatype. Just to share, the
> removal is possible as an exception, but it's best to go ahead for 3.1.1
> for safety as we all discussed.
> There are several non-regression but correctness issues being tracked
> under https://issues.apache.org/jira/browse/SPARK-33507 that contains few
> tickets Dongjoon mentioned.
> I will monitor this ticket and proceed for 3.1.1 RC1 when there are no
> more standing issues under this ticket.
> In addition, I made a PR to share this to Spark users and community (
> https://github.com/apache/spark-website/pull/301), and it's merged and
> visible now (
> https://spark.apache.org/news/next-official-release-spark-3.1.1.html)
>
> Thank you all for your support and feedback!
>
> 2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski 님이 작성:
>
>> Hi Sean,
>>
>> +1 to leave it. Makes so much more sense (as that's what really happened
>> and the history of Apache Spark is...irreversible).
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>
>> On Thu, Jan 7, 2021 at 3:44 PM Sean Owen  wrote:
>>
>>> While we can delete the tag, maybe just leave it. As a general rule we
>>> would not remove anything pushed to the main git repo.
>>>
>>> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>>>
 Hi,

 BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead
 with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and
 we'll see 3.1.1-rc1?

 [1] https://github.com/apache/spark/tree/v3.1.0-rc1

 Pozdrawiam,
 Jacek Laskowski
 
 https://about.me/JacekLaskowski
 "The Internals Of" Online Books 
 Follow me on https://twitter.com/jaceklaskowski

 

>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

I had a response from the INFRA team and Sonatype. Just to share, the
removal is possible as an exception, but it's best to go ahead for 3.1.1
for safety as we all discussed.
There are several non-regression but correctness issues being tracked under
https://issues.apache.org/jira/browse/SPARK-33507 that contains few tickets
Dongjoon mentioned.
I will monitor this ticket and proceed for 3.1.1 RC1 when there are no more
standing issues under this ticket.
In addition, I made a PR to share this to Spark users and community (
https://github.com/apache/spark-website/pull/301), and it's merged and
visible now (
https://spark.apache.org/news/next-official-release-spark-3.1.1.html)

Thank you all for your support and feedback!

2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski 님이 작성:

> Hi Sean,
>
> +1 to leave it. Makes so much more sense (as that's what really happened
> and the history of Apache Spark is...irreversible).
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Thu, Jan 7, 2021 at 3:44 PM Sean Owen  wrote:
>
>> While we can delete the tag, maybe just leave it. As a general rule we
>> would not remove anything pushed to the main git repo.
>>
>> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>>
>>> Hi,
>>>
>>> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead
>>> with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and
>>> we'll see 3.1.1-rc1?
>>>
>>> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> 
>>> https://about.me/JacekLaskowski
>>> "The Internals Of" Online Books 
>>> Follow me on https://twitter.com/jaceklaskowski
>>>
>>> 
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened
and the history of Apache Spark is...irreversible).

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 3:44 PM Sean Owen  wrote:

> While we can delete the tag, maybe just leave it. As a general rule we
> would not remove anything pushed to the main git repo.
>
> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
>> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
>> see 3.1.1-rc1?
>>
>> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Xiao Li]

> I will prepare to upload news in spark-website to explain that 3.1.0 is
incompletely published because there was something wrong during the release
process, and we go to 3.1.1 right away.

+1

Sean Owen  于2021年1月7日周四 上午6:44写道：

> While we can delete the tag, maybe just leave it. As a general rule we
> would not remove anything pushed to the main git repo.
>
> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
>> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
>> see 3.1.1-rc1?
>>
>> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

While we can delete the tag, maybe just leave it. As a general rule we
would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:

> Hi,
>
> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
> see 3.1.1-rc1?
>
> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
see 3.1.1-rc1?

[1] https://github.com/apache/spark/tree/v3.1.0-rc1

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 3:12 PM Jacek Laskowski  wrote:

> Hi,
>
> I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that
> we're skipping that particular release. Gonna be more fun! :)
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon  wrote:
>
>> Thank you Holden and Wenchen!
>>
>> Let me:
>> - prepare a PR for news in spark-website first about 3.1.0 accident late
>> tonight (in KST)
>> - and start to prepare 3.1.1 probably in few more days like next monday
>> in case other people have different thoughts
>>
>>
>>
>> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>>
>>> I think that posting the 3.1.0 maven release was an accident and we're
>>> going to 3.1.1 RCs is the right step forward.
>>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>>> happens in prod too).
>>>
>>> I'd also like to echo the: it's totally ok we all make mistakes
>>> especially in partially manual & partially automated environments, I've
>>> created a bunch of RCs labels without recognizing they were getting pushed
>>> automatically.
>>>
>>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>>
 I agree with Jungtaek that people are likely to be biased when testing
 3.1.0. At least this will not be the same community-blessed release as
 previous ones, because the voting is already affected by the fact that
 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

 On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
 wrote:

> Okay, let me just start to prepare 3.1.1. I think that will address
> all concerns except that 3.1.0 will remain in Maven as incomplete.
> By right, removal in the Maven repo is disallowed. Overwrite is
> possible as far as I know but other mirrors that maintain cache will get
> affected.
> Maven is one of the downstream publish channels, and we haven't
> officially announced and published it to Apache repo anyway.
> I will prepare to upload news in spark-website to explain that 3.1.0
> is incompletely published because there was something wrong during the
> release process, and we go to 3.1.1 right away.
> Are we all good with this?
>
>
>
> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>
>> I think that It would be great though if we have a clear blocker that
>> makes the release pointless if we want to drop this RC practically given
>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>> delivered to end users relatively fast.
>> That would make it clear which option we should take. I personally
>> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA 
>> team's
>> response anyway.
>>
>>
>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>
>>> I don't agree the first two are blockers for reasons I gave earlier.
>>> Those two do look like important issues - are they regressions from
>>> 3.0.1?
>>> I do agree we'd probably cut a new RC for those in any event, so
>>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>>> overwritten)
>>>
>>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Before we discover the pre-uploaded artifacts, both Jungtaek and
 Hyukjin already made two blockers shared here.
 IIUC, it meant implicitly RC1 failure at that time.

 In addition to that, there are two correctness issues. So, I made
 up my mind to cast -1 for this RC1 before joining this thread.

 SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh
 cache (committed after tagging)
 SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
 (PR is under review)

 Although the above issues are not regression, those are enough for
 me to give -1 for 3.1.0 RC1.

 On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it
> as normal? I did, you can too, etc.
> Or specifically what blocks 

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi,

I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that
we're skipping that particular release. Gonna be more fun! :)

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon  wrote:

> Thank you Holden and Wenchen!
>
> Let me:
> - prepare a PR for news in spark-website first about 3.1.0 accident late
> tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday in
> case other people have different thoughts
>
>
>
> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>
>> I think that posting the 3.1.0 maven release was an accident and we're
>> going to 3.1.1 RCs is the right step forward.
>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>> happens in prod too).
>>
>> I'd also like to echo the: it's totally ok we all make mistakes
>> especially in partially manual & partially automated environments, I've
>> created a bunch of RCs labels without recognizing they were getting pushed
>> automatically.
>>
>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>
>>> I agree with Jungtaek that people are likely to be biased when testing
>>> 3.1.0. At least this will not be the same community-blessed release as
>>> previous ones, because the voting is already affected by the fact that
>>> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>>>
>>> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
>>> wrote:
>>>
 Okay, let me just start to prepare 3.1.1. I think that will address all
 concerns except that 3.1.0 will remain in Maven as incomplete.
 By right, removal in the Maven repo is disallowed. Overwrite is
 possible as far as I know but other mirrors that maintain cache will get
 affected.
 Maven is one of the downstream publish channels, and we haven't
 officially announced and published it to Apache repo anyway.
 I will prepare to upload news in spark-website to explain that 3.1.0 is
 incompletely published because there was something wrong during the release
 process, and we go to 3.1.1 right away.
 Are we all good with this?



 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:

> I think that It would be great though if we have a clear blocker that
> makes the release pointless if we want to drop this RC practically given
> that we will schedule 3.1.1 faster - non-regression bug fixes will be
> delivered to end users relatively fast.
> That would make it clear which option we should take. I personally
> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
> response anyway.
>
>
> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>
>> I don't agree the first two are blockers for reasons I gave earlier.
>> Those two do look like important issues - are they regressions from
>> 3.0.1?
>> I do agree we'd probably cut a new RC for those in any event, so
>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>> overwritten)
>>
>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>> wrote:
>>
>>> Before we discover the pre-uploaded artifacts, both Jungtaek and
>>> Hyukjin already made two blockers shared here.
>>> IIUC, it meant implicitly RC1 failure at that time.
>>>
>>> In addition to that, there are two correctness issues. So, I made up
>>> my mind to cast -1 for this RC1 before joining this thread.
>>>
>>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>>> (committed after tagging)
>>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
>>> (PR is under review)
>>>
>>> Although the above issues are not regression, those are enough for
>>> me to give -1 for 3.1.0 RC1.
>>>
>>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>>
 I just don't see a reason to believe there's a rush? just test it
 as normal? I did, you can too, etc.
 Or specifically what blocks the current RC?

>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Gengliang Wang]

> - prepare a PR for news in spark-website first about 3.1.0 accident late
tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday
in case other people have different thoughts

+1

On Thu, Jan 7, 2021 at 7:54 PM Takeshi Yamamuro 
wrote:

> > I will prepare to upload news in spark-website to explain that 3.1.0 is
> incompletely published because there was something wrong during the release
> process, and we go to 3.1.1 right away.
>
> +1
>
> Thanks for working on this, Hyukjin!
>
> Bests,
> Takeshi
>
> On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon  wrote:
>
>> Thank you Holden and Wenchen!
>>
>> Let me:
>> - prepare a PR for news in spark-website first about 3.1.0 accident late
>> tonight (in KST)
>> - and start to prepare 3.1.1 probably in few more days like next monday
>> in case other people have different thoughts
>>
>>
>>
>> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>>
>>> I think that posting the 3.1.0 maven release was an accident and we're
>>> going to 3.1.1 RCs is the right step forward.
>>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>>> happens in prod too).
>>>
>>> I'd also like to echo the: it's totally ok we all make mistakes
>>> especially in partially manual & partially automated environments, I've
>>> created a bunch of RCs labels without recognizing they were getting pushed
>>> automatically.
>>>
>>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>>
 I agree with Jungtaek that people are likely to be biased when testing
 3.1.0. At least this will not be the same community-blessed release as
 previous ones, because the voting is already affected by the fact that
 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

 On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
 wrote:

> Okay, let me just start to prepare 3.1.1. I think that will address
> all concerns except that 3.1.0 will remain in Maven as incomplete.
> By right, removal in the Maven repo is disallowed. Overwrite is
> possible as far as I know but other mirrors that maintain cache will get
> affected.
> Maven is one of the downstream publish channels, and we haven't
> officially announced and published it to Apache repo anyway.
> I will prepare to upload news in spark-website to explain that 3.1.0
> is incompletely published because there was something wrong during the
> release process, and we go to 3.1.1 right away.
> Are we all good with this?
>
>
>
> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>
>> I think that It would be great though if we have a clear blocker that
>> makes the release pointless if we want to drop this RC practically given
>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>> delivered to end users relatively fast.
>> That would make it clear which option we should take. I personally
>> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA 
>> team's
>> response anyway.
>>
>>
>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>
>>> I don't agree the first two are blockers for reasons I gave earlier.
>>> Those two do look like important issues - are they regressions from
>>> 3.0.1?
>>> I do agree we'd probably cut a new RC for those in any event, so
>>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>>> overwritten)
>>>
>>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Before we discover the pre-uploaded artifacts, both Jungtaek and
 Hyukjin already made two blockers shared here.
 IIUC, it meant implicitly RC1 failure at that time.

 In addition to that, there are two correctness issues. So, I made
 up my mind to cast -1 for this RC1 before joining this thread.

 SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh
 cache (committed after tagging)
 SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
 (PR is under review)

 Although the above issues are not regression, those are enough for
 me to give -1 for 3.1.0 RC1.

 On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it
> as normal? I did, you can too, etc.
> Or specifically what blocks the current RC?
>

>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Takeshi Yamamuro]

> I will prepare to upload news in spark-website to explain that 3.1.0 is
incompletely published because there was something wrong during the release
process, and we go to 3.1.1 right away.

+1

Thanks for working on this, Hyukjin!

Bests,
Takeshi

On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon  wrote:

> Thank you Holden and Wenchen!
>
> Let me:
> - prepare a PR for news in spark-website first about 3.1.0 accident late
> tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday in
> case other people have different thoughts
>
>
>
> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>
>> I think that posting the 3.1.0 maven release was an accident and we're
>> going to 3.1.1 RCs is the right step forward.
>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>> happens in prod too).
>>
>> I'd also like to echo the: it's totally ok we all make mistakes
>> especially in partially manual & partially automated environments, I've
>> created a bunch of RCs labels without recognizing they were getting pushed
>> automatically.
>>
>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>
>>> I agree with Jungtaek that people are likely to be biased when testing
>>> 3.1.0. At least this will not be the same community-blessed release as
>>> previous ones, because the voting is already affected by the fact that
>>> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>>>
>>> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
>>> wrote:
>>>
 Okay, let me just start to prepare 3.1.1. I think that will address all
 concerns except that 3.1.0 will remain in Maven as incomplete.
 By right, removal in the Maven repo is disallowed. Overwrite is
 possible as far as I know but other mirrors that maintain cache will get
 affected.
 Maven is one of the downstream publish channels, and we haven't
 officially announced and published it to Apache repo anyway.
 I will prepare to upload news in spark-website to explain that 3.1.0 is
 incompletely published because there was something wrong during the release
 process, and we go to 3.1.1 right away.
 Are we all good with this?



 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:

> I think that It would be great though if we have a clear blocker that
> makes the release pointless if we want to drop this RC practically given
> that we will schedule 3.1.1 faster - non-regression bug fixes will be
> delivered to end users relatively fast.
> That would make it clear which option we should take. I personally
> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
> response anyway.
>
>
> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>
>> I don't agree the first two are blockers for reasons I gave earlier.
>> Those two do look like important issues - are they regressions from
>> 3.0.1?
>> I do agree we'd probably cut a new RC for those in any event, so
>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>> overwritten)
>>
>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>> wrote:
>>
>>> Before we discover the pre-uploaded artifacts, both Jungtaek and
>>> Hyukjin already made two blockers shared here.
>>> IIUC, it meant implicitly RC1 failure at that time.
>>>
>>> In addition to that, there are two correctness issues. So, I made up
>>> my mind to cast -1 for this RC1 before joining this thread.
>>>
>>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>>> (committed after tagging)
>>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
>>> (PR is under review)
>>>
>>> Although the above issues are not regression, those are enough for
>>> me to give -1 for 3.1.0 RC1.
>>>
>>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>>
 I just don't see a reason to believe there's a rush? just test it
 as normal? I did, you can too, etc.
 Or specifically what blocks the current RC?

>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

-- 
---
Takeshi Yamamuro

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late
tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in
case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:

> I think that posting the 3.1.0 maven release was an accident and we're
> going to 3.1.1 RCs is the right step forward.
> I'd ask for maybe a day before cutting the 3.1.1 release, I think
> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
> first I thought it was just a test issue, but Dongjoon pointed out the NPE
> happens in prod too).
>
> I'd also like to echo the: it's totally ok we all make mistakes especially
> in partially manual & partially automated environments, I've created a
> bunch of RCs labels without recognizing they were getting pushed
> automatically.
>
> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>
>> I agree with Jungtaek that people are likely to be biased when testing
>> 3.1.0. At least this will not be the same community-blessed release as
>> previous ones, because the voting is already affected by the fact that
>> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>>
>> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon  wrote:
>>
>>> Okay, let me just start to prepare 3.1.1. I think that will address all
>>> concerns except that 3.1.0 will remain in Maven as incomplete.
>>> By right, removal in the Maven repo is disallowed. Overwrite is possible
>>> as far as I know but other mirrors that maintain cache will get affected.
>>> Maven is one of the downstream publish channels, and we haven't
>>> officially announced and published it to Apache repo anyway.
>>> I will prepare to upload news in spark-website to explain that 3.1.0 is
>>> incompletely published because there was something wrong during the release
>>> process, and we go to 3.1.1 right away.
>>> Are we all good with this?
>>>
>>>
>>>
>>> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>>>
 I think that It would be great though if we have a clear blocker that
 makes the release pointless if we want to drop this RC practically given
 that we will schedule 3.1.1 faster - non-regression bug fixes will be
 delivered to end users relatively fast.
 That would make it clear which option we should take. I personally
 don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
 response anyway.


 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:

> I don't agree the first two are blockers for reasons I gave earlier.
> Those two do look like important issues - are they regressions from
> 3.0.1?
> I do agree we'd probably cut a new RC for those in any event, so agree
> with the plan to drop 3.1.0 (if the Maven release can't be overwritten)
>
> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
> wrote:
>
>> Before we discover the pre-uploaded artifacts, both Jungtaek and
>> Hyukjin already made two blockers shared here.
>> IIUC, it meant implicitly RC1 failure at that time.
>>
>> In addition to that, there are two correctness issues. So, I made up
>> my mind to cast -1 for this RC1 before joining this thread.
>>
>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>> (committed after tagging)
>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
>> (PR is under review)
>>
>> Although the above issues are not regression, those are enough for me
>> to give -1 for 3.1.0 RC1.
>>
>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>
>>> I just don't see a reason to believe there's a rush? just test it as
>>> normal? I did, you can too, etc.
>>> Or specifically what blocks the current RC?
>>>
>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Holden Karau]

I think that posting the 3.1.0 maven release was an accident and we're
going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think
https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
first I thought it was just a test issue, but Dongjoon pointed out the NPE
happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially
in partially manual & partially automated environments, I've created a
bunch of RCs labels without recognizing they were getting pushed
automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:

> I agree with Jungtaek that people are likely to be biased when testing
> 3.1.0. At least this will not be the same community-blessed release as
> previous ones, because the voting is already affected by the fact that
> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>
> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon  wrote:
>
>> Okay, let me just start to prepare 3.1.1. I think that will address all
>> concerns except that 3.1.0 will remain in Maven as incomplete.
>> By right, removal in the Maven repo is disallowed. Overwrite is possible
>> as far as I know but other mirrors that maintain cache will get affected.
>> Maven is one of the downstream publish channels, and we haven't
>> officially announced and published it to Apache repo anyway.
>> I will prepare to upload news in spark-website to explain that 3.1.0 is
>> incompletely published because there was something wrong during the release
>> process, and we go to 3.1.1 right away.
>> Are we all good with this?
>>
>>
>>
>> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>>
>>> I think that It would be great though if we have a clear blocker that
>>> makes the release pointless if we want to drop this RC practically given
>>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>>> delivered to end users relatively fast.
>>> That would make it clear which option we should take. I personally don't
>>> mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
>>> response anyway.
>>>
>>>
>>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>>
 I don't agree the first two are blockers for reasons I gave earlier.
 Those two do look like important issues - are they regressions from
 3.0.1?
 I do agree we'd probably cut a new RC for those in any event, so agree
 with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

 On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
 wrote:

> Before we discover the pre-uploaded artifacts, both Jungtaek and
> Hyukjin already made two blockers shared here.
> IIUC, it meant implicitly RC1 failure at that time.
>
> In addition to that, there are two correctness issues. So, I made up
> my mind to cast -1 for this RC1 before joining this thread.
>
> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
> (committed after tagging)
> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
> (PR is under review)
>
> Although the above issues are not regression, those are enough for me
> to give -1 for 3.1.0 RC1.
>
> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>
>> I just don't see a reason to believe there's a rush? just test it as
>> normal? I did, you can too, etc.
>> Or specifically what blocks the current RC?
>>
>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Wenchen Fan]

I agree with Jungtaek that people are likely to be biased when testing
3.1.0. At least this will not be the same community-blessed release as
previous ones, because the voting is already affected by the fact that
3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon  wrote:

> Okay, let me just start to prepare 3.1.1. I think that will address all
> concerns except that 3.1.0 will remain in Maven as incomplete.
> By right, removal in the Maven repo is disallowed. Overwrite is possible
> as far as I know but other mirrors that maintain cache will get affected.
> Maven is one of the downstream publish channels, and we haven't officially
> announced and published it to Apache repo anyway.
> I will prepare to upload news in spark-website to explain that 3.1.0 is
> incompletely published because there was something wrong during the release
> process, and we go to 3.1.1 right away.
> Are we all good with this?
>
>
>
> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>
>> I think that It would be great though if we have a clear blocker that
>> makes the release pointless if we want to drop this RC practically given
>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>> delivered to end users relatively fast.
>> That would make it clear which option we should take. I personally don't
>> mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
>> response anyway.
>>
>>
>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>
>>> I don't agree the first two are blockers for reasons I gave earlier.
>>> Those two do look like important issues - are they regressions from
>>> 3.0.1?
>>> I do agree we'd probably cut a new RC for those in any event, so agree
>>> with the plan to drop 3.1.0 (if the Maven release can't be overwritten)
>>>
>>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>>> wrote:
>>>
 Before we discover the pre-uploaded artifacts, both Jungtaek and
 Hyukjin already made two blockers shared here.
 IIUC, it meant implicitly RC1 failure at that time.

 In addition to that, there are two correctness issues. So, I made up my
 mind to cast -1 for this RC1 before joining this thread.

 SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
 (committed after tagging)
 SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR
 is under review)

 Although the above issues are not regression, those are enough for me
 to give -1 for 3.1.0 RC1.

 On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it as
> normal? I did, you can too, etc.
> Or specifically what blocks the current RC?
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Okay, let me just start to prepare 3.1.1. I think that will address all
concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as
far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially
announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is
incompletely published because there was something wrong during the release
process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:

> I think that It would be great though if we have a clear blocker that
> makes the release pointless if we want to drop this RC practically given
> that we will schedule 3.1.1 faster - non-regression bug fixes will be
> delivered to end users relatively fast.
> That would make it clear which option we should take. I personally don't
> mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
> response anyway.
>
>
> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>
>> I don't agree the first two are blockers for reasons I gave earlier.
>> Those two do look like important issues - are they regressions from 3.0.1?
>> I do agree we'd probably cut a new RC for those in any event, so agree
>> with the plan to drop 3.1.0 (if the Maven release can't be overwritten)
>>
>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>> wrote:
>>
>>> Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin
>>> already made two blockers shared here.
>>> IIUC, it meant implicitly RC1 failure at that time.
>>>
>>> In addition to that, there are two correctness issues. So, I made up my
>>> mind to cast -1 for this RC1 before joining this thread.
>>>
>>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>>> (committed after tagging)
>>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR
>>> is under review)
>>>
>>> Although the above issues are not regression, those are enough for me to
>>> give -1 for 3.1.0 RC1.
>>>
>>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>>
 I just don't see a reason to believe there's a rush? just test it as
 normal? I did, you can too, etc.
 Or specifically what blocks the current RC?

>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

I think that It would be great though if we have a clear blocker that makes
the release pointless if we want to drop this RC practically given that we
will schedule 3.1.1 faster - non-regression bug fixes will be delivered to
end users relatively fast.
That would make it clear which option we should take. I personally don't
mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:

> I don't agree the first two are blockers for reasons I gave earlier.
> Those two do look like important issues - are they regressions from 3.0.1?
> I do agree we'd probably cut a new RC for those in any event, so agree
> with the plan to drop 3.1.0 (if the Maven release can't be overwritten)
>
> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
> wrote:
>
>> Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin
>> already made two blockers shared here.
>> IIUC, it meant implicitly RC1 failure at that time.
>>
>> In addition to that, there are two correctness issues. So, I made up my
>> mind to cast -1 for this RC1 before joining this thread.
>>
>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>> (committed after tagging)
>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR
>> is under review)
>>
>> Although the above issues are not regression, those are enough for me to
>> give -1 for 3.1.0 RC1.
>>
>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>
>>> I just don't see a reason to believe there's a rush? just test it as
>>> normal? I did, you can too, etc.
>>> Or specifically what blocks the current RC?
>>>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with
the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
wrote:

> Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin
> already made two blockers shared here.
> IIUC, it meant implicitly RC1 failure at that time.
>
> In addition to that, there are two correctness issues. So, I made up my
> mind to cast -1 for this RC1 before joining this thread.
>
> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
> (committed after tagging)
> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is
> under review)
>
> Although the above issues are not regression, those are enough for me to
> give -1 for 3.1.0 RC1.
>
> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>
>> I just don't see a reason to believe there's a rush? just test it as
>> normal? I did, you can too, etc.
>> Or specifically what blocks the current RC?
>>
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Dongjoon Hyun]

Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin
already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my
mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
(committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is
under review)

Although the above issues are not regression, those are enough for me to
give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it as
> normal? I did, you can too, etc.
> Or specifically what blocks the current RC?
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

I just don't see a reason to believe there's a rush? just test it as
normal? I did, you can too, etc.
Or specifically what blocks the current RC?

On Wed, Jan 6, 2021 at 5:46 PM Jungtaek Lim 
wrote:

> No worries about the accident. We're human beings, and everyone can make a
> mistake. Let's wait and see the response of INFRA-21266.
>
> Just a 2 cents, I'm actually leaning toward to skip 3.1.0 and start the
> release process for 3.1.1, as anyone could be some sort of "rushing" on
> verification on 3.1.0. As we're already biased by the fact the release is
> already available, the RC might not be tested intensively and extensively.
> I'm also OK to continue verification on RC1 and consider this as official
> 3.1.0 once vote passes if we are sure to do the normal release candidate QA
> without bias. (I mean skipping some verifications they normally did, or
> consider serious bugs to be "later" based on the expectation that 3.1.1
> comes pretty soon.)
>
> On Thu, Jan 7, 2021 at 6:56 AM Hyukjin Kwon  wrote:
>
>> Thanks Dongjoon, Sean and Tom. I just thought that we could have some
>> more bug fixes or some changes if RC1 passes as a regular release due to
>> the relatively fewer RCs.
>> I agree that if this RC passes, it's just that an RC passed normally per
>> the regular process, and there's nothing wrong here. By right, there
>> shouldn't be any special treatment or difference in 3.1.1.
>> I more meant a practical point that we might happen to face some more bug
>> fixes or breaking changes (of course as an exception) that happens
>> sometimes.
>>
>>
>> 2021년 1월 7일 (목) 오전 6:44, Tom Graves 님이 작성:
>>
>>> I think it makes sense to wait and see what they say on INFRA-21266
>>> .
>>>
>>> In the mean time hopefully people can start testing it and if no other
>>> problems found and vote passes can stay published.  It seems like the 2
>>> issues above wouldn't be blockers in my opinion and could be handled in a
>>> 3.1.1 but others can chime too.
>>>
>>> If we find other issues with it in testing and they can't revert in
>>> INFRA-21266 - I assume we handle by putting some documentation out there
>>> telling people not to use it and we go to 3.1.1.
>>>
>>> One thing I didn't follow was the comment: "release 3.1.1 fast that
>>> exceptionally allows a bit of breaking changes" - what do you mean by that?
>>>
>>> if there is anything we can add to our release process documentation to
>>> prevent in the future that would be great as well.
>>>
>>> Tom
>>>
>>> On Wednesday, January 6, 2021, 03:07:26 PM CST, Hyukjin Kwon <
>>> gurwls...@gmail.com> wrote:
>>>
>>>
>>> Yes, it was my mistake. I faced the same issue as INFRA-20651
>>> , and it is worse in
>>> my case because I misunderstood that RC and releases are separately
>>> released out.
>>> Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
>>> . We can wait and
>>> see how it goes.
>>>
>>> Though, I know it’s impossible to remove by right. It is possible to
>>> overwrite but it will affect people who already have it in their cache.
>>> I am thinkthing two options:
>>>
>>>- Skip 3.1.0 and release 3.1.1 right away since the release isn’t
>>>officially out to the main Apache repo/mirrors but only one of the
>>>downstream channels. We can just say that there was something wrong 
>>> during
>>>the 3.1.0 release so it became 3.1.1 right away.
>>>
>>>
>>>- Release 3.1.0 out, of course, based on the vote results here. We
>>>could release 3.1.1 fast that exceptionally allows a bit of breaking
>>>changes with properly documenting it in a release note and migration 
>>> guide.
>>>
>>> I would appreciate it if I could hear other people' opinions.
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jungtaek Lim]

No worries about the accident. We're human beings, and everyone can make a
mistake. Let's wait and see the response of INFRA-21266.

Just a 2 cents, I'm actually leaning toward to skip 3.1.0 and start the
release process for 3.1.1, as anyone could be some sort of "rushing" on
verification on 3.1.0. As we're already biased by the fact the release is
already available, the RC might not be tested intensively and extensively.
I'm also OK to continue verification on RC1 and consider this as official
3.1.0 once vote passes if we are sure to do the normal release candidate QA
without bias. (I mean skipping some verifications they normally did, or
consider serious bugs to be "later" based on the expectation that 3.1.1
comes pretty soon.)

On Thu, Jan 7, 2021 at 6:56 AM Hyukjin Kwon  wrote:

> Thanks Dongjoon, Sean and Tom. I just thought that we could have some more
> bug fixes or some changes if RC1 passes as a regular release due to the
> relatively fewer RCs.
> I agree that if this RC passes, it's just that an RC passed normally per
> the regular process, and there's nothing wrong here. By right, there
> shouldn't be any special treatment or difference in 3.1.1.
> I more meant a practical point that we might happen to face some more bug
> fixes or breaking changes (of course as an exception) that happens
> sometimes.
>
>
> 2021년 1월 7일 (목) 오전 6:44, Tom Graves 님이 작성:
>
>> I think it makes sense to wait and see what they say on INFRA-21266
>> .
>>
>> In the mean time hopefully people can start testing it and if no other
>> problems found and vote passes can stay published.  It seems like the 2
>> issues above wouldn't be blockers in my opinion and could be handled in a
>> 3.1.1 but others can chime too.
>>
>> If we find other issues with it in testing and they can't revert in
>> INFRA-21266 - I assume we handle by putting some documentation out there
>> telling people not to use it and we go to 3.1.1.
>>
>> One thing I didn't follow was the comment: "release 3.1.1 fast that
>> exceptionally allows a bit of breaking changes" - what do you mean by that?
>>
>> if there is anything we can add to our release process documentation to
>> prevent in the future that would be great as well.
>>
>> Tom
>>
>> On Wednesday, January 6, 2021, 03:07:26 PM CST, Hyukjin Kwon <
>> gurwls...@gmail.com> wrote:
>>
>>
>> Yes, it was my mistake. I faced the same issue as INFRA-20651
>> , and it is worse in
>> my case because I misunderstood that RC and releases are separately
>> released out.
>> Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
>> . We can wait and see
>> how it goes.
>>
>> Though, I know it’s impossible to remove by right. It is possible to
>> overwrite but it will affect people who already have it in their cache.
>> I am thinkthing two options:
>>
>>- Skip 3.1.0 and release 3.1.1 right away since the release isn’t
>>officially out to the main Apache repo/mirrors but only one of the
>>downstream channels. We can just say that there was something wrong during
>>the 3.1.0 release so it became 3.1.1 right away.
>>
>>
>>- Release 3.1.0 out, of course, based on the vote results here. We
>>could release 3.1.1 fast that exceptionally allows a bit of breaking
>>changes with properly documenting it in a release note and migration 
>> guide.
>>
>> I would appreciate it if I could hear other people' opinions.
>>
>> Thanks.
>>
>>
>>
>>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Thanks Dongjoon, Sean and Tom. I just thought that we could have some more
bug fixes or some changes if RC1 passes as a regular release due to the
relatively fewer RCs.
I agree that if this RC passes, it's just that an RC passed normally per
the regular process, and there's nothing wrong here. By right, there
shouldn't be any special treatment or difference in 3.1.1.
I more meant a practical point that we might happen to face some more bug
fixes or breaking changes (of course as an exception) that happens
sometimes.


2021년 1월 7일 (목) 오전 6:44, Tom Graves 님이 작성:

> I think it makes sense to wait and see what they say on INFRA-21266
> .
>
> In the mean time hopefully people can start testing it and if no other
> problems found and vote passes can stay published.  It seems like the 2
> issues above wouldn't be blockers in my opinion and could be handled in a
> 3.1.1 but others can chime too.
>
> If we find other issues with it in testing and they can't revert in
> INFRA-21266 - I assume we handle by putting some documentation out there
> telling people not to use it and we go to 3.1.1.
>
> One thing I didn't follow was the comment: "release 3.1.1 fast that
> exceptionally allows a bit of breaking changes" - what do you mean by that?
>
> if there is anything we can add to our release process documentation to
> prevent in the future that would be great as well.
>
> Tom
>
> On Wednesday, January 6, 2021, 03:07:26 PM CST, Hyukjin Kwon <
> gurwls...@gmail.com> wrote:
>
>
> Yes, it was my mistake. I faced the same issue as INFRA-20651
> , and it is worse in
> my case because I misunderstood that RC and releases are separately
> released out.
> Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
> . We can wait and see
> how it goes.
>
> Though, I know it’s impossible to remove by right. It is possible to
> overwrite but it will affect people who already have it in their cache.
> I am thinkthing two options:
>
>- Skip 3.1.0 and release 3.1.1 right away since the release isn’t
>officially out to the main Apache repo/mirrors but only one of the
>downstream channels. We can just say that there was something wrong during
>the 3.1.0 release so it became 3.1.1 right away.
>
>
>- Release 3.1.0 out, of course, based on the vote results here. We
>could release 3.1.1 fast that exceptionally allows a bit of breaking
>changes with properly documenting it in a release note and migration guide.
>
> I would appreciate it if I could hear other people' opinions.
>
> Thanks.
>
>
>
>
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Tom Graves]

 I think it makes sense to wait and see what they say on INFRA-21266.  
In the mean time hopefully people can start testing it and if no other problems 
found and vote passes can stay published.  It seems like the 2 issues above 
wouldn't be blockers in my opinion and could be handled in a 3.1.1 but others 
can chime too.
If we find other issues with it in testing and they can't revert in INFRA-21266 
- I assume we handle by putting some documentation out there telling people not 
to use it and we go to 3.1.1.  
One thing I didn't follow was the comment: "release 3.1.1 fast that 
exceptionally allows a bit of breaking changes" - what do you mean by that?
if there is anything we can add to our release process documentation to prevent 
in the future that would be great as well.
Tom
On Wednesday, January 6, 2021, 03:07:26 PM CST, Hyukjin Kwon 
 wrote:  
 
 
Yes, it was my mistake. I faced the same issue as INFRA-20651, and it is worse 
in my case because I misunderstood that RC and releases are separately released 
out.
Right after this, I filed an INFRA JIRA to revert this at INFRA-21266. We can 
wait and see how it goes.

Though, I know it’s impossible to remove by right. It is possible to overwrite 
but it will affect people who already have it in their cache.
I am thinkthing two options:
   
   - Skip 3.1.0 and release 3.1.1 right away since the release isn’t officially 
out to the main Apache repo/mirrors but only one of the downstream channels. We 
can just say that there was something wrong during the 3.1.0 release so it 
became 3.1.1 right away.
   
   - Release 3.1.0 out, of course, based on the vote results here. We could 
release 3.1.1 fast that exceptionally allows a bit of breaking changes with 
properly documenting it in a release note and migration guide.
I would appreciate it if I could hear other people' opinions.

Thanks.



  

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

OK, we'll have to update the release page to clarify there was never a real
3.1.0 release then.

But I'm not suggesting releasing 3.1.0 _because_ it was published
accidentally.
I'm suggesting we figure out normally whether we would have released it,
and if so, great. If not, fine we must skip the version.

On Wed, Jan 6, 2021 at 3:36 PM Dongjoon Hyun 
wrote:

> Thank you, Jacek, Sean, and Hyukjin.
>
> The release is a human-driven process. Everyone can make mistakes.
>
> For example, I released Apache Spark 2.2.3 with a missing pandoc, but we
> didn't touch it because it's a community-blessed official version.
>
>https://pypi.org/project/pyspark/2.2.3/
>
> For this incident, given the situation, I'm +1 for `Skipping 3.1.0 and
> starting 3.1.1 RC1` instead of making it official.
>
> It's because
>
> 1. The vote is important in the Apache project management process.
> The existing 3.1.0 artifacts in Maven Central are not a
> community-blessed version.
> 2. Since this is the first incident, we had better build a rule to handle
> this kind of accident.
> If we approve `3.1.0` because it's published accidently, it could be a
> bad practice.
> In the worst case, a release manager can publish Spark 10.1.1
> accidently in the future without votes.
>
> BTW, thank you, Hyukjin, for your all efforts to prepare 3.1.0 as a
> release manager.
> We know that you devote lots of your time to make it happen.
>
> Bests,
> Dongjoon.
>
>
> On Wed, Jan 6, 2021 at 1:07 PM Hyukjin Kwon  wrote:
>
>> Yes, it was my mistake. I faced the same issue as INFRA-20651
>> , and it is worse in
>> my case because I misunderstood that RC and releases are separately
>> released out.
>> Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
>> . We can wait and see
>> how it goes.
>>
>> Though, I know it’s impossible to remove by right. It is possible to
>> overwrite but it will affect people who already have it in their cache.
>> I am thinkthing two options:
>>
>>- Skip 3.1.0 and release 3.1.1 right away since the release isn’t
>>officially out to the main Apache repo/mirrors but only one of the
>>downstream channels. We can just say that there was something wrong during
>>the 3.1.0 release so it became 3.1.1 right away.
>>
>>
>>- Release 3.1.0 out, of course, based on the vote results here. We
>>could release 3.1.1 fast that exceptionally allows a bit of breaking
>>changes with properly documenting it in a release note and migration 
>> guide.
>>
>> I would appreciate it if I could hear other people' opinions.
>>
>> Thanks.
>>
>>
>>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Dongjoon Hyun]

Thank you, Jacek, Sean, and Hyukjin.

The release is a human-driven process. Everyone can make mistakes.

For example, I released Apache Spark 2.2.3 with a missing pandoc, but we
didn't touch it because it's a community-blessed official version.

   https://pypi.org/project/pyspark/2.2.3/

For this incident, given the situation, I'm +1 for `Skipping 3.1.0 and
starting 3.1.1 RC1` instead of making it official.

It's because

1. The vote is important in the Apache project management process.
The existing 3.1.0 artifacts in Maven Central are not a
community-blessed version.
2. Since this is the first incident, we had better build a rule to handle
this kind of accident.
If we approve `3.1.0` because it's published accidently, it could be a
bad practice.
In the worst case, a release manager can publish Spark 10.1.1
accidently in the future without votes.

BTW, thank you, Hyukjin, for your all efforts to prepare 3.1.0 as a release
manager.
We know that you devote lots of your time to make it happen.

Bests,
Dongjoon.


On Wed, Jan 6, 2021 at 1:07 PM Hyukjin Kwon  wrote:

> Yes, it was my mistake. I faced the same issue as INFRA-20651
> , and it is worse in
> my case because I misunderstood that RC and releases are separately
> released out.
> Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
> . We can wait and see
> how it goes.
>
> Though, I know it’s impossible to remove by right. It is possible to
> overwrite but it will affect people who already have it in their cache.
> I am thinkthing two options:
>
>- Skip 3.1.0 and release 3.1.1 right away since the release isn’t
>officially out to the main Apache repo/mirrors but only one of the
>downstream channels. We can just say that there was something wrong during
>the 3.1.0 release so it became 3.1.1 right away.
>
>
>- Release 3.1.0 out, of course, based on the vote results here. We
>could release 3.1.1 fast that exceptionally allows a bit of breaking
>changes with properly documenting it in a release note and migration guide.
>
> I would appreciate it if I could hear other people' opinions.
>
> Thanks.
>
>
>
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Yes, it was my mistake. I faced the same issue as INFRA-20651
, and it is worse in my
case because I misunderstood that RC and releases are separately released
out.
Right after this, I filed an INFRA JIRA to revert this at INFRA-21266
. We can wait and see
how it goes.

Though, I know it’s impossible to remove by right. It is possible to
overwrite but it will affect people who already have it in their cache.
I am thinkthing two options:

   - Skip 3.1.0 and release 3.1.1 right away since the release isn’t
   officially out to the main Apache repo/mirrors but only one of the
   downstream channels. We can just say that there was something wrong during
   the 3.1.0 release so it became 3.1.1 right away.


   - Release 3.1.0 out, of course, based on the vote results here. We could
   release 3.1.1 fast that exceptionally allows a bit of breaking changes with
   properly documenting it in a release note and migration guide.

I would appreciate it if I could hear other people' opinions.

Thanks.

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

Er, yeah uh oh. Did the staging repo accidentally get closed/released?
Maybe I'm also missing something.
If so then one way or the other we can't undo that as the 3.1.0 release in
Maven, as far as I know. We can make a 3.1.1, but then it's kind of weird
there was never any 3.1.0 source release.

We may well decide that, well, that 3.1.0 release was fine enough actually,
that the blockers aren't actually that blocking. And the two mentioned so
far don't sound so much like blockers.
- If the Kafka problem happened in 3.0, then it isn't a regression from 3.0
to 3.1, technically
- If CRAN wasn't working in 3.0.1, that's not a regression

Naturally, of course, we may wish to fix those fast. And, we can create a
3.1.1 release fast if desired. But if that's the extent of it - no serious
regressions - we might decide to just go ahead with this RC as the 3.1.0
release and move on to 3.1.1 quickly. At the least, again, would be weird
to have no 3.1.0 actual release on the ASF if we can't undo this.

FWIW the release looks OK to me as is, +1. I don't think the above are
blockers.



On Wed, Jan 6, 2021 at 1:38 PM Jacek Laskowski  wrote:

> Hi,
>
> I'm curious why Spark 3.1.0 is already available in repo1.maven.org?
>
> https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.0/
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Wed, Jan 6, 2021 at 1:01 AM Hyukjin Kwon  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 3.1.0.
>>
>> The vote is open until January 8th 4PM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.1.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.1.0-rc1 (commit
>> 97340c1e34cfd84de445b6b7545cfa466a1baaf6):
>> https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1363/
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/
>>
>> The list of bug fixes going into 3.1.0 can be found at the following URL:
>> https://s.apache.org/ldzzl
>>
>> This release is using the release script of the tag v3.1.0-rc1.
>>
>> FAQ
>>
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC via "pip install
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
>> "
>> and see if anything important breaks.
>> In the Java/Scala, you can add the staging repository to your projects
>> resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with an out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.1.0?
>> ===
>>
>> The current list of open tickets targeted at 3.1.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.1.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi,

I'm curious why Spark 3.1.0 is already available in repo1.maven.org?

https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.0/

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Wed, Jan 6, 2021 at 1:01 AM Hyukjin Kwon  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 3.1.0.
>
> The vote is open until January 8th 4PM PST and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.1.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v3.1.0-rc1 (commit
> 97340c1e34cfd84de445b6b7545cfa466a1baaf6):
> https://github.com/apache/spark/tree/v3.1.0-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1363/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/
>
> The list of bug fixes going into 3.1.0 can be found at the following URL:
> https://s.apache.org/ldzzl
>
> This release is using the release script of the tag v3.1.0-rc1.
>
> FAQ
>
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC via "pip install
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
> "
> and see if anything important breaks.
> In the Java/Scala, you can add the staging repository to your projects
> resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 3.1.0?
> ===
>
> The current list of open tickets targeted at 3.1.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.1.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Seems like we have two PRs for both blockers, and one is already merged,
nice.
I will wait for a couple of days more before starting a new RC to make sure
we catch more regressions before the new RC.
Please keep testing this RC. I would appreciate it :-).

2021년 1월 6일 (수) 오후 2:28, Hyukjin Kwon 님이 작성:

> Actually, I will mark https://issues.apache.org/jira/browse/SPARK-34021
> as a blocker too. For CRAN submission, we should fix it.
>
> 2021년 1월 6일 (수) 오후 1:47, Jungtaek Lim 님이 작성:
>
>> There's an issue SPARK-33635 [1] reported due to performance regression
>> on Kafka read between Spark 2.4 vs 3.0, which sounds like a blocker. I'll
>> mark this as a blocker, unless anyone has different opinions.
>>
>> 1. https://issues.apache.org/jira/browse/SPARK-33635
>>
>> On Wed, Jan 6, 2021 at 9:01 AM Hyukjin Kwon  wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 3.1.0.
>>>
>>> The vote is open until January 8th 4PM PST and passes if a majority +1
>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 3.1.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v3.1.0-rc1 (commit
>>> 97340c1e34cfd84de445b6b7545cfa466a1baaf6):
>>> https://github.com/apache/spark/tree/v3.1.0-rc1
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1363/
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/
>>>
>>> The list of bug fixes going into 3.1.0 can be found at the following URL:
>>> https://s.apache.org/ldzzl
>>>
>>> This release is using the release script of the tag v3.1.0-rc1.
>>>
>>> FAQ
>>>
>>>
>>> =
>>> How can I help test this release?
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC via "pip install
>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
>>> "
>>> and see if anything important breaks.
>>> In the Java/Scala, you can add the staging repository to your projects
>>> resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with an out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 3.1.0?
>>> ===
>>>
>>> The current list of open tickets targeted at 3.1.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.1.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Actually, I will mark https://issues.apache.org/jira/browse/SPARK-34021 as
a blocker too. For CRAN submission, we should fix it.

2021년 1월 6일 (수) 오후 1:47, Jungtaek Lim 님이 작성:

> There's an issue SPARK-33635 [1] reported due to performance regression on
> Kafka read between Spark 2.4 vs 3.0, which sounds like a blocker. I'll mark
> this as a blocker, unless anyone has different opinions.
>
> 1. https://issues.apache.org/jira/browse/SPARK-33635
>
> On Wed, Jan 6, 2021 at 9:01 AM Hyukjin Kwon  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 3.1.0.
>>
>> The vote is open until January 8th 4PM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.1.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.1.0-rc1 (commit
>> 97340c1e34cfd84de445b6b7545cfa466a1baaf6):
>> https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1363/
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/
>>
>> The list of bug fixes going into 3.1.0 can be found at the following URL:
>> https://s.apache.org/ldzzl
>>
>> This release is using the release script of the tag v3.1.0-rc1.
>>
>> FAQ
>>
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC via "pip install
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
>> "
>> and see if anything important breaks.
>> In the Java/Scala, you can add the staging repository to your projects
>> resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with an out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.1.0?
>> ===
>>
>> The current list of open tickets targeted at 3.1.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.1.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jungtaek Lim]

There's an issue SPARK-33635 [1] reported due to performance regression on
Kafka read between Spark 2.4 vs 3.0, which sounds like a blocker. I'll mark
this as a blocker, unless anyone has different opinions.

1. https://issues.apache.org/jira/browse/SPARK-33635

On Wed, Jan 6, 2021 at 9:01 AM Hyukjin Kwon  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 3.1.0.
>
> The vote is open until January 8th 4PM PST and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.1.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v3.1.0-rc1 (commit
> 97340c1e34cfd84de445b6b7545cfa466a1baaf6):
> https://github.com/apache/spark/tree/v3.1.0-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1363/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/
>
> The list of bug fixes going into 3.1.0 can be found at the following URL:
> https://s.apache.org/ldzzl
>
> This release is using the release script of the tag v3.1.0-rc1.
>
> FAQ
>
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC via "pip install
> https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
> "
> and see if anything important breaks.
> In the Java/Scala, you can add the staging repository to your projects
> resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 3.1.0?
> ===
>
> The current list of open tickets targeted at 3.1.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.1.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
>

[VOTE] Release Spark 3.1.0 (RC1)

[Hyukjin Kwon]

Please vote on releasing the following candidate as Apache Spark version
3.1.0.

The vote is open until January 8th 4PM PST and passes if a majority +1 PMC
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.0-rc1 (commit
97340c1e34cfd84de445b6b7545cfa466a1baaf6):
https://github.com/apache/spark/tree/v3.1.0-rc1

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1363/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/

The list of bug fixes going into 3.1.0 can be found at the following URL:
https://s.apache.org/ldzzl

This release is using the release script of the tag v3.1.0-rc1.

FAQ


=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install
https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-bin/pyspark-3.1.0.tar.gz
"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects
resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===
What should happen to JIRA tickets still targeting 3.1.0?
===

The current list of open tickets targeted at 3.1.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 3.1.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==
But my bug isn't fixed?
==

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.