[no subject]

[iriv kang]

Unsubscribe

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened
and the history of Apache Spark is...irreversible).

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 3:44 PM Sean Owen  wrote:

> While we can delete the tag, maybe just leave it. As a general rule we
> would not remove anything pushed to the main git repo.
>
> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
>> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
>> see 3.1.1-rc1?
>>
>> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Xiao Li]

> I will prepare to upload news in spark-website to explain that 3.1.0 is
incompletely published because there was something wrong during the release
process, and we go to 3.1.1 right away.

+1

Sean Owen  于2021年1月7日周四 上午6:44写道：

> While we can delete the tag, maybe just leave it. As a general rule we
> would not remove anything pushed to the main git repo.
>
> On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
>> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
>> see 3.1.1-rc1?
>>
>> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Sean Owen]

While we can delete the tag, maybe just leave it. As a general rule we
would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski  wrote:

> Hi,
>
> BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
> 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
> see 3.1.1-rc1?
>
> [1] https://github.com/apache/spark/tree/v3.1.0-rc1
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with
3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll
see 3.1.1-rc1?

[1] https://github.com/apache/spark/tree/v3.1.0-rc1

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 3:12 PM Jacek Laskowski  wrote:

> Hi,
>
> I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that
> we're skipping that particular release. Gonna be more fun! :)
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon  wrote:
>
>> Thank you Holden and Wenchen!
>>
>> Let me:
>> - prepare a PR for news in spark-website first about 3.1.0 accident late
>> tonight (in KST)
>> - and start to prepare 3.1.1 probably in few more days like next monday
>> in case other people have different thoughts
>>
>>
>>
>> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>>
>>> I think that posting the 3.1.0 maven release was an accident and we're
>>> going to 3.1.1 RCs is the right step forward.
>>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>>> happens in prod too).
>>>
>>> I'd also like to echo the: it's totally ok we all make mistakes
>>> especially in partially manual & partially automated environments, I've
>>> created a bunch of RCs labels without recognizing they were getting pushed
>>> automatically.
>>>
>>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>>
 I agree with Jungtaek that people are likely to be biased when testing
 3.1.0. At least this will not be the same community-blessed release as
 previous ones, because the voting is already affected by the fact that
 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

 On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
 wrote:

> Okay, let me just start to prepare 3.1.1. I think that will address
> all concerns except that 3.1.0 will remain in Maven as incomplete.
> By right, removal in the Maven repo is disallowed. Overwrite is
> possible as far as I know but other mirrors that maintain cache will get
> affected.
> Maven is one of the downstream publish channels, and we haven't
> officially announced and published it to Apache repo anyway.
> I will prepare to upload news in spark-website to explain that 3.1.0
> is incompletely published because there was something wrong during the
> release process, and we go to 3.1.1 right away.
> Are we all good with this?
>
>
>
> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>
>> I think that It would be great though if we have a clear blocker that
>> makes the release pointless if we want to drop this RC practically given
>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>> delivered to end users relatively fast.
>> That would make it clear which option we should take. I personally
>> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA 
>> team's
>> response anyway.
>>
>>
>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>
>>> I don't agree the first two are blockers for reasons I gave earlier.
>>> Those two do look like important issues - are they regressions from
>>> 3.0.1?
>>> I do agree we'd probably cut a new RC for those in any event, so
>>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>>> overwritten)
>>>
>>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Before we discover the pre-uploaded artifacts, both Jungtaek and
 Hyukjin already made two blockers shared here.
 IIUC, it meant implicitly RC1 failure at that time.

 In addition to that, there are two correctness issues. So, I made
 up my mind to cast -1 for this RC1 before joining this thread.

 SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh
 cache (committed after tagging)
 SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
 (PR is under review)

 Although the above issues are not regression, those are enough for
 me to give -1 for 3.1.0 RC1.

 On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it
> as normal? I did, you can too, etc.
> Or specifically what blocks 

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Jacek Laskowski]

Hi,

I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that
we're skipping that particular release. Gonna be more fun! :)

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon  wrote:

> Thank you Holden and Wenchen!
>
> Let me:
> - prepare a PR for news in spark-website first about 3.1.0 accident late
> tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday in
> case other people have different thoughts
>
>
>
> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>
>> I think that posting the 3.1.0 maven release was an accident and we're
>> going to 3.1.1 RCs is the right step forward.
>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>> happens in prod too).
>>
>> I'd also like to echo the: it's totally ok we all make mistakes
>> especially in partially manual & partially automated environments, I've
>> created a bunch of RCs labels without recognizing they were getting pushed
>> automatically.
>>
>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>
>>> I agree with Jungtaek that people are likely to be biased when testing
>>> 3.1.0. At least this will not be the same community-blessed release as
>>> previous ones, because the voting is already affected by the fact that
>>> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>>>
>>> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
>>> wrote:
>>>
 Okay, let me just start to prepare 3.1.1. I think that will address all
 concerns except that 3.1.0 will remain in Maven as incomplete.
 By right, removal in the Maven repo is disallowed. Overwrite is
 possible as far as I know but other mirrors that maintain cache will get
 affected.
 Maven is one of the downstream publish channels, and we haven't
 officially announced and published it to Apache repo anyway.
 I will prepare to upload news in spark-website to explain that 3.1.0 is
 incompletely published because there was something wrong during the release
 process, and we go to 3.1.1 right away.
 Are we all good with this?



 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:

> I think that It would be great though if we have a clear blocker that
> makes the release pointless if we want to drop this RC practically given
> that we will schedule 3.1.1 faster - non-regression bug fixes will be
> delivered to end users relatively fast.
> That would make it clear which option we should take. I personally
> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
> response anyway.
>
>
> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>
>> I don't agree the first two are blockers for reasons I gave earlier.
>> Those two do look like important issues - are they regressions from
>> 3.0.1?
>> I do agree we'd probably cut a new RC for those in any event, so
>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>> overwritten)
>>
>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>> wrote:
>>
>>> Before we discover the pre-uploaded artifacts, both Jungtaek and
>>> Hyukjin already made two blockers shared here.
>>> IIUC, it meant implicitly RC1 failure at that time.
>>>
>>> In addition to that, there are two correctness issues. So, I made up
>>> my mind to cast -1 for this RC1 before joining this thread.
>>>
>>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>>> (committed after tagging)
>>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
>>> (PR is under review)
>>>
>>> Although the above issues are not regression, those are enough for
>>> me to give -1 for 3.1.0 RC1.
>>>
>>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>>
 I just don't see a reason to believe there's a rush? just test it
 as normal? I did, you can too, etc.
 Or specifically what blocks the current RC?

>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

[no subject]

[rahul c]

Unsubscribe

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Gengliang Wang]

> - prepare a PR for news in spark-website first about 3.1.0 accident late
tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday
in case other people have different thoughts

+1

On Thu, Jan 7, 2021 at 7:54 PM Takeshi Yamamuro 
wrote:

> > I will prepare to upload news in spark-website to explain that 3.1.0 is
> incompletely published because there was something wrong during the release
> process, and we go to 3.1.1 right away.
>
> +1
>
> Thanks for working on this, Hyukjin!
>
> Bests,
> Takeshi
>
> On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon  wrote:
>
>> Thank you Holden and Wenchen!
>>
>> Let me:
>> - prepare a PR for news in spark-website first about 3.1.0 accident late
>> tonight (in KST)
>> - and start to prepare 3.1.1 probably in few more days like next monday
>> in case other people have different thoughts
>>
>>
>>
>> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>>
>>> I think that posting the 3.1.0 maven release was an accident and we're
>>> going to 3.1.1 RCs is the right step forward.
>>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>>> happens in prod too).
>>>
>>> I'd also like to echo the: it's totally ok we all make mistakes
>>> especially in partially manual & partially automated environments, I've
>>> created a bunch of RCs labels without recognizing they were getting pushed
>>> automatically.
>>>
>>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>>
 I agree with Jungtaek that people are likely to be biased when testing
 3.1.0. At least this will not be the same community-blessed release as
 previous ones, because the voting is already affected by the fact that
 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

 On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
 wrote:

> Okay, let me just start to prepare 3.1.1. I think that will address
> all concerns except that 3.1.0 will remain in Maven as incomplete.
> By right, removal in the Maven repo is disallowed. Overwrite is
> possible as far as I know but other mirrors that maintain cache will get
> affected.
> Maven is one of the downstream publish channels, and we haven't
> officially announced and published it to Apache repo anyway.
> I will prepare to upload news in spark-website to explain that 3.1.0
> is incompletely published because there was something wrong during the
> release process, and we go to 3.1.1 right away.
> Are we all good with this?
>
>
>
> 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:
>
>> I think that It would be great though if we have a clear blocker that
>> makes the release pointless if we want to drop this RC practically given
>> that we will schedule 3.1.1 faster - non-regression bug fixes will be
>> delivered to end users relatively fast.
>> That would make it clear which option we should take. I personally
>> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA 
>> team's
>> response anyway.
>>
>>
>> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>>
>>> I don't agree the first two are blockers for reasons I gave earlier.
>>> Those two do look like important issues - are they regressions from
>>> 3.0.1?
>>> I do agree we'd probably cut a new RC for those in any event, so
>>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>>> overwritten)
>>>
>>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Before we discover the pre-uploaded artifacts, both Jungtaek and
 Hyukjin already made two blockers shared here.
 IIUC, it meant implicitly RC1 failure at that time.

 In addition to that, there are two correctness issues. So, I made
 up my mind to cast -1 for this RC1 before joining this thread.

 SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh
 cache (committed after tagging)
 SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
 (PR is under review)

 Although the above issues are not regression, those are enough for
 me to give -1 for 3.1.0 RC1.

 On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:

> I just don't see a reason to believe there's a rush? just test it
> as normal? I did, you can too, etc.
> Or specifically what blocks the current RC?
>

>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [VOTE] Release Spark 3.1.0 (RC1)

[Takeshi Yamamuro]

> I will prepare to upload news in spark-website to explain that 3.1.0 is
incompletely published because there was something wrong during the release
process, and we go to 3.1.1 right away.

+1

Thanks for working on this, Hyukjin!

Bests,
Takeshi

On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon  wrote:

> Thank you Holden and Wenchen!
>
> Let me:
> - prepare a PR for news in spark-website first about 3.1.0 accident late
> tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday in
> case other people have different thoughts
>
>
>
> 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성:
>
>> I think that posting the 3.1.0 maven release was an accident and we're
>> going to 3.1.1 RCs is the right step forward.
>> I'd ask for maybe a day before cutting the 3.1.1 release, I think
>> https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at
>> first I thought it was just a test issue, but Dongjoon pointed out the NPE
>> happens in prod too).
>>
>> I'd also like to echo the: it's totally ok we all make mistakes
>> especially in partially manual & partially automated environments, I've
>> created a bunch of RCs labels without recognizing they were getting pushed
>> automatically.
>>
>> On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan  wrote:
>>
>>> I agree with Jungtaek that people are likely to be biased when testing
>>> 3.1.0. At least this will not be the same community-blessed release as
>>> previous ones, because the voting is already affected by the fact that
>>> 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.
>>>
>>> On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon 
>>> wrote:
>>>
 Okay, let me just start to prepare 3.1.1. I think that will address all
 concerns except that 3.1.0 will remain in Maven as incomplete.
 By right, removal in the Maven repo is disallowed. Overwrite is
 possible as far as I know but other mirrors that maintain cache will get
 affected.
 Maven is one of the downstream publish channels, and we haven't
 officially announced and published it to Apache repo anyway.
 I will prepare to upload news in spark-website to explain that 3.1.0 is
 incompletely published because there was something wrong during the release
 process, and we go to 3.1.1 right away.
 Are we all good with this?



 2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon 님이 작성:

> I think that It would be great though if we have a clear blocker that
> makes the release pointless if we want to drop this RC practically given
> that we will schedule 3.1.1 faster - non-regression bug fixes will be
> delivered to end users relatively fast.
> That would make it clear which option we should take. I personally
> don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's
> response anyway.
>
>
> 2021년 1월 7일 (목) 오후 1:03, Sean Owen 님이 작성:
>
>> I don't agree the first two are blockers for reasons I gave earlier.
>> Those two do look like important issues - are they regressions from
>> 3.0.1?
>> I do agree we'd probably cut a new RC for those in any event, so
>> agree with the plan to drop 3.1.0 (if the Maven release can't be
>> overwritten)
>>
>> On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun 
>> wrote:
>>
>>> Before we discover the pre-uploaded artifacts, both Jungtaek and
>>> Hyukjin already made two blockers shared here.
>>> IIUC, it meant implicitly RC1 failure at that time.
>>>
>>> In addition to that, there are two correctness issues. So, I made up
>>> my mind to cast -1 for this RC1 before joining this thread.
>>>
>>> SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
>>> (committed after tagging)
>>> SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
>>> (PR is under review)
>>>
>>> Although the above issues are not regression, those are enough for
>>> me to give -1 for 3.1.0 RC1.
>>>
>>> On Wed, Jan 6, 2021 at 3:52 PM Sean Owen  wrote:
>>>
 I just don't see a reason to believe there's a rush? just test it
 as normal? I did, you can too, etc.
 Or specifically what blocks the current RC?

>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

-- 
---
Takeshi Yamamuro