Re: [Reminder] Spark 3.5 Branch Cut

[Jia Fan]

Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to
JdbcDialect into 3.5.0?
https://github.com/apache/spark/pull/41855
Since this is the last major version update of 3.x, I think we need to make
sure JdbcDialect can support more databases.


Gengliang Wang  于2023年7月15日周六 05:20写道：

> Hi Yuanjian,
>
> Besides the abovementioned changes, it would be great to include the UI
> page for Spakr Connect: SPARK-44394
> .
>
> Best Regards,
> Gengliang
>
> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
>  wrote:
>
>> Thank you,
>> My changes that you listed are tracked under this Epic:
>> https://issues.apache.org/jira/browse/SPARK-43754
>> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
>> didn't mention it before because I have hopes that this one will make it
>> before the cut.
>>
>> (Unrelated) My colleague is also working on
>> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
>> https://github.com/apache/spark/pull/41443, so I hope that that one will
>> also make it before the cut.
>>
>> Best regards,
>> Juliusz Sompolski
>>
>> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li 
>> wrote:
>>
>>> Hi everyone,
>>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>>
>>> Please plan your PR merge accordingly with the given timeline.
>>> Currently, we have received the following exception merge requests:
>>>
>>>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>>>mechanism)
>>>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>>>client)
>>>- SPARK-44424:  Reattach to existing execute in Spark Connect
>>>(python client)
>>>
>>> If there are any other exception feature requests, please reply to this
>>> email. We will not merge any new features in 3.5 after the branch cut.
>>>
>>> Best,
>>> Yuanjian
>>>
>>

Re: [Reminder] Spark 3.5 Branch Cut

[Gengliang Wang]

Hi Yuanjian,

Besides the abovementioned changes, it would be great to include the UI
page for Spakr Connect: SPARK-44394
.

Best Regards,
Gengliang

On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
 wrote:

> Thank you,
> My changes that you listed are tracked under this Epic:
> https://issues.apache.org/jira/browse/SPARK-43754
> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
> didn't mention it before because I have hopes that this one will make it
> before the cut.
>
> (Unrelated) My colleague is also working on
> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
> https://github.com/apache/spark/pull/41443, so I hope that that one will
> also make it before the cut.
>
> Best regards,
> Juliusz Sompolski
>
> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li 
> wrote:
>
>> Hi everyone,
>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>
>> Please plan your PR merge accordingly with the given timeline. Currently,
>> we have received the following exception merge requests:
>>
>>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>>mechanism)
>>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>>client)
>>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>>client)
>>
>> If there are any other exception feature requests, please reply to this
>> email. We will not merge any new features in 3.5 after the branch cut.
>>
>> Best,
>> Yuanjian
>>
>

Re: [Reminder] Spark 3.5 Branch Cut

[Julek Sompolski]

Thank you,
My changes that you listed are tracked under this Epic:
https://issues.apache.org/jira/browse/SPARK-43754
I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
didn't mention it before because I have hopes that this one will make it
before the cut.

(Unrelated) My colleague is also working on
https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
https://github.com/apache/spark/pull/41443, so I hope that that one will
also make it before the cut.

Best regards,
Juliusz Sompolski

On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li  wrote:

> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>mechanism)
>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>client)
>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>

Re: [Reminder] Spark 3.5 Branch Cut

[Raghu Angadi]

Thank you. We plan to get remaining major pieces for Streaming Spark
Connect (Epic SPARK-42938
).
I would like to request a warmfix exception for the following tweaks and
improvements over the next two weeks (all in the same epic).

   - SPARK-42944 : (Will mostly hit Monday deadline, just in case) Python
   foreachBatch
   - SPARK-42941 : (WIP, but might slip Monday deadline): Python streaming
   listener
   - SPARK-44400 : Improve session access in connect Scala
   StreamingQueryListener
   - SPARK-44432 : Allow timeout of sessions when client disconnects and
   terminate queries
   - SPARK-44433 : Improve termination logic for Python processes for
   foreachBatch & query listener
   - SPARK-44434 : More Scala tests for foreachBatch & query listener
   - SPARK-44435 : More Python tests for foreachBatch & query listener
   - SPARK-44436 : Use Connect DataFrame for Scala foreachBatch in Connect


On Fri, Jul 14, 2023 at 10:33 AM Yuanjian Li  wrote:

> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>mechanism)
>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>client)
>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>

Re: Time for Spark v3.5.0 release

[Yuanjian Li]

Thanks for raising all the requests. Let's stick to the previously agreed
branch cut time. Based on past practice, let's label the above requests as
exception features.

I have just sent out a branch cut reminder titled "[Reminder] Spark 3.5
Branch Cut." Please ensure that all your requests are included.

Best,
Yuanjian

Julek Sompolski  于2023年7月14日周五 09:07写道：

> I am working on SPARK-44421, SPARK-44423 and SPARK-44424 in Spark Connect
> to support execution reconnection. A week or two of warmfix grace period
> would be much appreciated for this work.
>
> Best regards,
> Juliusz Sompolski
>
> On Fri, Jul 14, 2023 at 5:40 PM Raghu Angadi
>  wrote:
>
>> We have a bunch of work in progress for Spark Connect trying to meet the
>> branch cut deadline.
>>
>> Moving to 17th is certainly welcome.
>>
>> Is it feasible to extend it by a couple of more days?
>> Alternatively, we could have a relaxed warmfix process for Spark Connect
>> code for a week or two since it does not affect core Spark.
>>
>> Thank you.
>> Raghu.
>>
>> On Tue, Jul 4, 2023 at 3:42 PM Xinrong Meng  wrote:
>>
>>> +1
>>>
>>> Thank you!
>>>
>>> On Tue, Jul 4, 2023 at 3:04 PM Jungtaek Lim <
>>> kabhwan.opensou...@gmail.com> wrote:
>>>
 +1

 On Wed, Jul 5, 2023 at 2:23 AM L. C. Hsieh  wrote:

> +1
>
> Thanks Yuanjian.
>
> On Tue, Jul 4, 2023 at 7:45 AM yangjie01  wrote:
> >
> > +1
> >
> >
> >
> > 发件人: Maxim Gekk 
> > 日期: 2023年7月4日 星期二 17:24
> > 收件人: Kent Yao 
> > 抄送: "dev@spark.apache.org" 
> > 主题: Re: Time for Spark v3.5.0 release
> >
> >
> >
> > +1
> >
> > On Tue, Jul 4, 2023 at 11:55 AM Kent Yao  wrote:
> >
> > +1， thank you
> >
> > Kent
> >
> > On 2023/07/04 05:32:52 Dongjoon Hyun wrote:
> > > +1
> > >
> > > Thank you, Yuanjian
> > >
> > > Dongjoon
> > >
> > > On Tue, Jul 4, 2023 at 1:03 AM Hyukjin Kwon 
> wrote:
> > >
> > > > Yeah one day postponed shouldn't be a big deal.
> > > >
> > > > On Tue, Jul 4, 2023 at 7:10 AM Yuanjian Li <
> xyliyuanj...@gmail.com> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> According to the Spark versioning policy at
> > > >> https://spark.apache.org/versioning-policy.html, should we cut
> > > >> *branch-3.5* on *July 17th, 2023*? (We initially proposed
> January 16th,
> > > >> but since it's a Sunday, I suggest we postpone it by one day).
> > > >>
> > > >> I would like to volunteer as the release manager for Apache
> Spark 3.5.0.
> > > >>
> > > >> Best,
> > > >> Yuanjian
> > > >>
> > > >
> > >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

[Reminder] Spark 3.5 Branch Cut

[Yuanjian Li]

Hi everyone,
As discussed earlier in "Time for Spark v3.5.0 release", I will cut
branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.

Please plan your PR merge accordingly with the given timeline. Currently,
we have received the following exception merge requests:

   - SPARK-44421: Reattach to existing execute in Spark Connect (server
   mechanism)
   - SPARK-44423:  Reattach to existing execute in Spark Connect (scala
   client)
   - SPARK-44424:  Reattach to existing execute in Spark Connect (python
   client)

If there are any other exception feature requests, please reply to this
email. We will not merge any new features in 3.5 after the branch cut.

Best,
Yuanjian

Re: Time for Spark v3.5.0 release

[Julek Sompolski]

I am working on SPARK-44421, SPARK-44423 and SPARK-44424 in Spark Connect
to support execution reconnection. A week or two of warmfix grace period
would be much appreciated for this work.

Best regards,
Juliusz Sompolski

On Fri, Jul 14, 2023 at 5:40 PM Raghu Angadi
 wrote:

> We have a bunch of work in progress for Spark Connect trying to meet the
> branch cut deadline.
>
> Moving to 17th is certainly welcome.
>
> Is it feasible to extend it by a couple of more days?
> Alternatively, we could have a relaxed warmfix process for Spark Connect
> code for a week or two since it does not affect core Spark.
>
> Thank you.
> Raghu.
>
> On Tue, Jul 4, 2023 at 3:42 PM Xinrong Meng  wrote:
>
>> +1
>>
>> Thank you!
>>
>> On Tue, Jul 4, 2023 at 3:04 PM Jungtaek Lim 
>> wrote:
>>
>>> +1
>>>
>>> On Wed, Jul 5, 2023 at 2:23 AM L. C. Hsieh  wrote:
>>>
 +1

 Thanks Yuanjian.

 On Tue, Jul 4, 2023 at 7:45 AM yangjie01  wrote:
 >
 > +1
 >
 >
 >
 > 发件人: Maxim Gekk 
 > 日期: 2023年7月4日 星期二 17:24
 > 收件人: Kent Yao 
 > 抄送: "dev@spark.apache.org" 
 > 主题: Re: Time for Spark v3.5.0 release
 >
 >
 >
 > +1
 >
 > On Tue, Jul 4, 2023 at 11:55 AM Kent Yao  wrote:
 >
 > +1， thank you
 >
 > Kent
 >
 > On 2023/07/04 05:32:52 Dongjoon Hyun wrote:
 > > +1
 > >
 > > Thank you, Yuanjian
 > >
 > > Dongjoon
 > >
 > > On Tue, Jul 4, 2023 at 1:03 AM Hyukjin Kwon 
 wrote:
 > >
 > > > Yeah one day postponed shouldn't be a big deal.
 > > >
 > > > On Tue, Jul 4, 2023 at 7:10 AM Yuanjian Li <
 xyliyuanj...@gmail.com> wrote:
 > > >
 > > >> Hi All,
 > > >>
 > > >> According to the Spark versioning policy at
 > > >> https://spark.apache.org/versioning-policy.html, should we cut
 > > >> *branch-3.5* on *July 17th, 2023*? (We initially proposed
 January 16th,
 > > >> but since it's a Sunday, I suggest we postpone it by one day).
 > > >>
 > > >> I would like to volunteer as the release manager for Apache
 Spark 3.5.0.
 > > >>
 > > >> Best,
 > > >> Yuanjian
 > > >>
 > > >
 > >
 >
 > -
 > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Time for Spark v3.5.0 release

[Raghu Angadi]

We have a bunch of work in progress for Spark Connect trying to meet the
branch cut deadline.

Moving to 17th is certainly welcome.

Is it feasible to extend it by a couple of more days?
Alternatively, we could have a relaxed warmfix process for Spark Connect
code for a week or two since it does not affect core Spark.

Thank you.
Raghu.

On Tue, Jul 4, 2023 at 3:42 PM Xinrong Meng  wrote:

> +1
>
> Thank you!
>
> On Tue, Jul 4, 2023 at 3:04 PM Jungtaek Lim 
> wrote:
>
>> +1
>>
>> On Wed, Jul 5, 2023 at 2:23 AM L. C. Hsieh  wrote:
>>
>>> +1
>>>
>>> Thanks Yuanjian.
>>>
>>> On Tue, Jul 4, 2023 at 7:45 AM yangjie01  wrote:
>>> >
>>> > +1
>>> >
>>> >
>>> >
>>> > 发件人: Maxim Gekk 
>>> > 日期: 2023年7月4日 星期二 17:24
>>> > 收件人: Kent Yao 
>>> > 抄送: "dev@spark.apache.org" 
>>> > 主题: Re: Time for Spark v3.5.0 release
>>> >
>>> >
>>> >
>>> > +1
>>> >
>>> > On Tue, Jul 4, 2023 at 11:55 AM Kent Yao  wrote:
>>> >
>>> > +1， thank you
>>> >
>>> > Kent
>>> >
>>> > On 2023/07/04 05:32:52 Dongjoon Hyun wrote:
>>> > > +1
>>> > >
>>> > > Thank you, Yuanjian
>>> > >
>>> > > Dongjoon
>>> > >
>>> > > On Tue, Jul 4, 2023 at 1:03 AM Hyukjin Kwon 
>>> wrote:
>>> > >
>>> > > > Yeah one day postponed shouldn't be a big deal.
>>> > > >
>>> > > > On Tue, Jul 4, 2023 at 7:10 AM Yuanjian Li 
>>> wrote:
>>> > > >
>>> > > >> Hi All,
>>> > > >>
>>> > > >> According to the Spark versioning policy at
>>> > > >> https://spark.apache.org/versioning-policy.html, should we cut
>>> > > >> *branch-3.5* on *July 17th, 2023*? (We initially proposed January
>>> 16th,
>>> > > >> but since it's a Sunday, I suggest we postpone it by one day).
>>> > > >>
>>> > > >> I would like to volunteer as the release manager for Apache Spark
>>> 3.5.0.
>>> > > >>
>>> > > >> Best,
>>> > > >> Yuanjian
>>> > > >>
>>> > > >
>>> > >
>>> >
>>> > -
>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>