Re: Initial Decom PR for Spark 3?

[Hyukjin Kwon]

See https://spark.apache.org/improvement-proposals.html

On Tue, 23 Jun 2020, 07:01 Stephen Boesch,  wrote:

> I guess I missed that "community decision" where the existing design
> document that had been reviewed was put aside and a new SPIP  document was
> required.
>
> On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon  wrote:
>
>> Yeah, I believe the community decided to do a SPIP for such significant
>> changes. It would be best if we stick to the standard approaches.
>>
>> 2020년 6월 21일 (일) 오전 8:52, Holden Karau 님이 작성:
>>
>>> I believe so, however since Hyukjin is a committer and has asked for an
>>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>>> draft for comment by the end of Spark summit. I'll be using the same design
>>> document for the design component, so if anyone has input on the design
>>> document feel free to start leaving comments there now.
>>>
>>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch 
>>> wrote:
>>>
 Hi given there is a design doc (contrary to that common) - is this
 going to move forward?

 On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
>> wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>>> on it if folks don’t think it’s worth it. I’ve been working on it since
>>> 2017 and it was really close in November but then I had the crash and 
>>> had
>>> to step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been
>>> outstanding for awhile so those points are in its favour. It does 
>>> however
>>> change things in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

[Stephen Boesch]

I guess I missed that "community decision" where the existing design
document that had been reviewed was put aside and a new SPIP  document was
required.

On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon  wrote:

> Yeah, I believe the community decided to do a SPIP for such significant
> changes. It would be best if we stick to the standard approaches.
>
> 2020년 6월 21일 (일) 오전 8:52, Holden Karau 님이 작성:
>
>> I believe so, however since Hyukjin is a committer and has asked for an
>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>> draft for comment by the end of Spark summit. I'll be using the same design
>> document for the design component, so if anyone has input on the design
>> document feel free to start leaving comments there now.
>>
>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch  wrote:
>>
>>> Hi given there is a design doc (contrary to that common) - is this going
>>> to move forward?
>>>
>>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:
>>>
 Looks it had to be with SPIP and a proper design doc to discuss.

 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:

> I'd be willing to pull this in, unless others have concerns post
> branch-cut.
>
> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
> wrote:
>
>> Hi Y’all,
>>
>> I’ve got a K8s graceful decom PR (
>> https://github.com/apache/spark/pull/26440
>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>> on it if folks don’t think it’s worth it. I’ve been working on it since
>> 2017 and it was really close in November but then I had the crash and had
>> to step back for awhile.
>>
>> It’s effectiveness is behind a feature flag and it’s been outstanding
>> for awhile so those points are in its favour. It does however change 
>> things
>> in core which is not great.
>>
>> Cheers,
>>
>> Holden
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Initial Decom PR for Spark 3?

[Hyukjin Kwon]

Yeah, I believe the community decided to do a SPIP for such significant
changes. It would be best if we stick to the standard approaches.

2020년 6월 21일 (일) 오전 8:52, Holden Karau 님이 작성:

> I believe so, however since Hyukjin is a committer and has asked for an
> SPIP I'll be making an SPIP for this next week. I hope to send out the
> draft for comment by the end of Spark summit. I'll be using the same design
> document for the design component, so if anyone has input on the design
> document feel free to start leaving comments there now.
>
> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch  wrote:
>
>> Hi given there is a design doc (contrary to that common) - is this going
>> to move forward?
>>
>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:
>>
>>> Looks it had to be with SPIP and a proper design doc to discuss.
>>>
>>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>>>
 I'd be willing to pull this in, unless others have concerns post
 branch-cut.

 On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
 wrote:

> Hi Y’all,
>
> I’ve got a K8s graceful decom PR (
> https://github.com/apache/spark/pull/26440
>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
> it if folks don’t think it’s worth it. I’ve been working on it since 2017
> and it was really close in November but then I had the crash and had to
> step back for awhile.
>
> It’s effectiveness is behind a feature flag and it’s been outstanding
> for awhile so those points are in its favour. It does however change 
> things
> in core which is not great.
>
> Cheers,
>
> Holden
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Initial Decom PR for Spark 3?

[Holden Karau]

I believe so, however since Hyukjin is a committer and has asked for an
SPIP I'll be making an SPIP for this next week. I hope to send out the
draft for comment by the end of Spark summit. I'll be using the same design
document for the design component, so if anyone has input on the design
document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch  wrote:

> Hi given there is a design doc (contrary to that common) - is this going
> to move forward?
>
> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:
>
>> Looks it had to be with SPIP and a proper design doc to discuss.
>>
>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>>
>>> I'd be willing to pull this in, unless others have concerns post
>>> branch-cut.
>>>
>>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
>>> wrote:
>>>
 Hi Y’all,

 I’ve got a K8s graceful decom PR (
 https://github.com/apache/spark/pull/26440
  ) I’d love to try and get in for Spark 3, but I don’t want to push on
 it if folks don’t think it’s worth it. I’ve been working on it since 2017
 and it was really close in November but then I had the crash and had to
 step back for awhile.

 It’s effectiveness is behind a feature flag and it’s been outstanding
 for awhile so those points are in its favour. It does however change things
 in core which is not great.

 Cheers,

 Holden
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Initial Decom PR for Spark 3?

[Stephen Boesch]

Hi given there is a design doc (contrary to that common) - is this going to
move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau  wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

[Stephen Boesch]

Second paragraph of the PR lists the design doc.

> There is a design document at
https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau  wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

[Holden Karau]

For follow up while I've backported this in some internal releases I'm not
considering a candidate for backporting to Spark 3 anymore. I should have
updated the thread with that. The design doc is linked in the PR.

On Thu, Jun 18, 2020 at 6:05 PM Hyukjin Kwon  wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau  wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Initial Decom PR for Spark 3?

[Hyukjin Kwon]

Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:

> I'd be willing to pull this in, unless others have concerns post
> branch-cut.
>
> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau  wrote:
>
>> Hi Y’all,
>>
>> I’ve got a K8s graceful decom PR (
>> https://github.com/apache/spark/pull/26440
>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on it
>> if folks don’t think it’s worth it. I’ve been working on it since 2017 and
>> it was really close in November but then I had the crash and had to step
>> back for awhile.
>>
>> It’s effectiveness is behind a feature flag and it’s been outstanding for
>> awhile so those points are in its favour. It does however change things in
>> core which is not great.
>>
>> Cheers,
>>
>> Holden
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Initial Decom PR for Spark 3?

[Erik Erlandson]

I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau  wrote:

> Hi Y’all,
>
> I’ve got a K8s graceful decom PR (
> https://github.com/apache/spark/pull/26440
>  ) I’d love to try and get in for Spark 3, but I don’t want to push on it
> if folks don’t think it’s worth it. I’ve been working on it since 2017 and
> it was really close in November but then I had the crash and had to step
> back for awhile.
>
> It’s effectiveness is behind a feature flag and it’s been outstanding for
> awhile so those points are in its favour. It does however change things in
> core which is not great.
>
> Cheers,
>
> Holden
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Initial Decom PR for Spark 3?

[Holden Karau]

Hi Y’all,

I’ve got a K8s graceful decom PR (
https://github.com/apache/spark/pull/26440
 ) I’d love to try and get in for Spark 3, but I don’t want to push on it
if folks don’t think it’s worth it. I’ve been working on it since 2017 and
it was really close in November but then I had the crash and had to step
back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for
awhile so those points are in its favour. It does however change things in
core which is not great.

Cheers,

Holden
-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau