Re: [Reminder] Spark 3.5 Branch Cut

[Herman van Hovell]

Hi Yuanjian,

For the ongoing encoder work for the connect scala client I'd like to get
the following tickets in:

   - SPARK-44396  :
   Direct Arrow Deserialization
   - SPARK-9  :
   Upcasting for Arrow Deserialization
   - SPARK-44450  : Make
   direct Arrow encoding work with SQL/API.

Cheers,
Herman

On Sat, Jul 15, 2023 at 7:53 AM Enrico Minack 
wrote:

> Speaking of JdbcDialect, is there any interest in getting upserts for JDBC
> into 3.5.0?
>
> [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC:
> https://github.com/apache/spark/pull/41518
> [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using
> MERGE INTO with temp table: https://github.com/apache/spark/pull/41611
>
> Enrico
>
>
> Am 15.07.23 um 04:10 schrieb Jia Fan:
>
> Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to
> JdbcDialect into 3.5.0?
> https://github.com/apache/spark/pull/41855
> Since this is the last major version update of 3.x, I think we need to
> make sure JdbcDialect can support more databases.
>
>
> Gengliang Wang  于2023年7月15日周六 05:20写道：
>
>> Hi Yuanjian,
>>
>> Besides the abovementioned changes, it would be great to include the UI
>> page for Spakr Connect: SPARK-44394
>> .
>>
>> Best Regards,
>> Gengliang
>>
>> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
>>   wrote:
>>
>>> Thank you,
>>> My changes that you listed are tracked under this Epic:
>>> https://issues.apache.org/jira/browse/SPARK-43754
>>> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
>>> didn't mention it before because I have hopes that this one will make it
>>> before the cut.
>>>
>>> (Unrelated) My colleague is also working on
>>> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
>>> https://github.com/apache/spark/pull/41443, so I hope that that one
>>> will also make it before the cut.
>>>
>>> Best regards,
>>> Juliusz Sompolski
>>>
>>> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li 
>>> wrote:
>>>
 Hi everyone,
 As discussed earlier in "Time for Spark v3.5.0 release", I will cut
 branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.

 Please plan your PR merge accordingly with the given timeline.
 Currently, we have received the following exception merge requests:

- SPARK-44421: Reattach to existing execute in Spark Connect
(server mechanism)
- SPARK-44423:  Reattach to existing execute in Spark Connect
(scala client)
- SPARK-44424:  Reattach to existing execute in Spark Connect
(python client)

 If there are any other exception feature requests, please reply to this
 email. We will not merge any new features in 3.5 after the branch cut.

 Best,
 Yuanjian

>>>
>

Re: [Reminder] Spark 3.5 Branch Cut

[Enrico Minack]

Speaking of JdbcDialect, is there any interest in getting upserts for 
JDBC into 3.5.0?


[SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC: 
https://github.com/apache/spark/pull/41518
[SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using 
MERGE INTO with temp table: https://github.com/apache/spark/pull/41611


Enrico


Am 15.07.23 um 04:10 schrieb Jia Fan:
Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` 
to JdbcDialect into 3.5.0?

https://github.com/apache/spark/pull/41855
Since this is the last major version update of 3.x, I think we need to 
make sure JdbcDialect can support more databases.



Gengliang Wang  于2023年7月15日周六 05:20写道：

Hi Yuanjian,

Besides the abovementioned changes, it would be great to include
the UI page for Spakr Connect: SPARK-44394
.

Best Regards,
Gengliang

On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
 wrote:

Thank you,
My changes that you listed are tracked under this Epic:
https://issues.apache.org/jira/browse/SPARK-43754
I am also working on
https://issues.apache.org/jira/browse/SPARK-44422, didn't
mention it before because I have hopes that this one will make
it before the cut.

(Unrelated) My colleague is also working on
https://issues.apache.org/jira/browse/SPARK-43923 and I am
reviewing https://github.com/apache/spark/pull/41443, so I
hope that that one will also make it before the cut.

Best regards,
Juliusz Sompolski

On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li
 wrote:

Hi everyone,
As discussed earlier in "Time for Spark v3.5.0 release", I
will cut branch-3.5 on *Monday, July 17th at 1 pm PST* as
scheduled.

Please plan your PR merge accordingly with the given
timeline. Currently, we have received the following
exception merge requests:

  * SPARK-44421: Reattach to existing execute in Spark
Connect (server mechanism)
  * SPARK-44423:  Reattach to existing execute in Spark
Connect (scala client)
  * SPARK-44424:  Reattach to existing execute in Spark
Connect (python client)

If there are any other exception feature requests, please
reply to this email. We will not merge any new features in
3.5 after the branch cut.

Best,
Yuanjian

Re: [Reminder] Spark 3.5 Branch Cut

[Jia Fan]

Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to
JdbcDialect into 3.5.0?
https://github.com/apache/spark/pull/41855
Since this is the last major version update of 3.x, I think we need to make
sure JdbcDialect can support more databases.


Gengliang Wang  于2023年7月15日周六 05:20写道：

> Hi Yuanjian,
>
> Besides the abovementioned changes, it would be great to include the UI
> page for Spakr Connect: SPARK-44394
> .
>
> Best Regards,
> Gengliang
>
> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
>  wrote:
>
>> Thank you,
>> My changes that you listed are tracked under this Epic:
>> https://issues.apache.org/jira/browse/SPARK-43754
>> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
>> didn't mention it before because I have hopes that this one will make it
>> before the cut.
>>
>> (Unrelated) My colleague is also working on
>> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
>> https://github.com/apache/spark/pull/41443, so I hope that that one will
>> also make it before the cut.
>>
>> Best regards,
>> Juliusz Sompolski
>>
>> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li 
>> wrote:
>>
>>> Hi everyone,
>>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>>
>>> Please plan your PR merge accordingly with the given timeline.
>>> Currently, we have received the following exception merge requests:
>>>
>>>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>>>mechanism)
>>>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>>>client)
>>>- SPARK-44424:  Reattach to existing execute in Spark Connect
>>>(python client)
>>>
>>> If there are any other exception feature requests, please reply to this
>>> email. We will not merge any new features in 3.5 after the branch cut.
>>>
>>> Best,
>>> Yuanjian
>>>
>>

Re: [Reminder] Spark 3.5 Branch Cut

[Gengliang Wang]

Hi Yuanjian,

Besides the abovementioned changes, it would be great to include the UI
page for Spakr Connect: SPARK-44394
.

Best Regards,
Gengliang

On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
 wrote:

> Thank you,
> My changes that you listed are tracked under this Epic:
> https://issues.apache.org/jira/browse/SPARK-43754
> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
> didn't mention it before because I have hopes that this one will make it
> before the cut.
>
> (Unrelated) My colleague is also working on
> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
> https://github.com/apache/spark/pull/41443, so I hope that that one will
> also make it before the cut.
>
> Best regards,
> Juliusz Sompolski
>
> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li 
> wrote:
>
>> Hi everyone,
>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>
>> Please plan your PR merge accordingly with the given timeline. Currently,
>> we have received the following exception merge requests:
>>
>>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>>mechanism)
>>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>>client)
>>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>>client)
>>
>> If there are any other exception feature requests, please reply to this
>> email. We will not merge any new features in 3.5 after the branch cut.
>>
>> Best,
>> Yuanjian
>>
>

Re: [Reminder] Spark 3.5 Branch Cut

[Julek Sompolski]

Thank you,
My changes that you listed are tracked under this Epic:
https://issues.apache.org/jira/browse/SPARK-43754
I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
didn't mention it before because I have hopes that this one will make it
before the cut.

(Unrelated) My colleague is also working on
https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
https://github.com/apache/spark/pull/41443, so I hope that that one will
also make it before the cut.

Best regards,
Juliusz Sompolski

On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li  wrote:

> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>mechanism)
>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>client)
>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>

Re: [Reminder] Spark 3.5 Branch Cut

[Raghu Angadi]

Thank you. We plan to get remaining major pieces for Streaming Spark
Connect (Epic SPARK-42938
).
I would like to request a warmfix exception for the following tweaks and
improvements over the next two weeks (all in the same epic).

   - SPARK-42944 : (Will mostly hit Monday deadline, just in case) Python
   foreachBatch
   - SPARK-42941 : (WIP, but might slip Monday deadline): Python streaming
   listener
   - SPARK-44400 : Improve session access in connect Scala
   StreamingQueryListener
   - SPARK-44432 : Allow timeout of sessions when client disconnects and
   terminate queries
   - SPARK-44433 : Improve termination logic for Python processes for
   foreachBatch & query listener
   - SPARK-44434 : More Scala tests for foreachBatch & query listener
   - SPARK-44435 : More Python tests for foreachBatch & query listener
   - SPARK-44436 : Use Connect DataFrame for Scala foreachBatch in Connect


On Fri, Jul 14, 2023 at 10:33 AM Yuanjian Li  wrote:

> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
>- SPARK-44421: Reattach to existing execute in Spark Connect (server
>mechanism)
>- SPARK-44423:  Reattach to existing execute in Spark Connect (scala
>client)
>- SPARK-44424:  Reattach to existing execute in Spark Connect (python
>client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>

[Reminder] Spark 3.5 Branch Cut

[Yuanjian Li]

Hi everyone,
As discussed earlier in "Time for Spark v3.5.0 release", I will cut
branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.

Please plan your PR merge accordingly with the given timeline. Currently,
we have received the following exception merge requests:

   - SPARK-44421: Reattach to existing execute in Spark Connect (server
   mechanism)
   - SPARK-44423:  Reattach to existing execute in Spark Connect (scala
   client)
   - SPARK-44424:  Reattach to existing execute in Spark Connect (python
   client)

If there are any other exception feature requests, please reply to this
email. We will not merge any new features in 3.5 after the branch cut.

Best,
Yuanjian