Re: [DISCUSS] Spark 4.0.0 release

[Wenchen Fan]

Thank you all for the replies!

To @Nicholas Chammas  : Thanks for cleaning up
the error terminology and documentation! I've merged the first PR and let's
finish others before the 4.0 release.
To @Dongjoon Hyun  : Thanks for driving the ANSI
on by default effort! Now the vote has passed, let's flip the config and
finish the DataFrame error context feature before 4.0.
To @Jungtaek Lim  : Ack. We can treat the
Streaming state store data source as completed for 4.0 then.
To @Cheng Pan  : Yea we definitely should have a
preview release. Let's collect more feedback on the ongoing projects and
then we can propose a date for the preview release.

On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan  wrote:

> will we have preview release for 4.0.0 like we did for 2.0.0 and 3.0.0?
>
> Thanks,
> Cheng Pan
>
>
> > On Apr 15, 2024, at 09:58, Jungtaek Lim 
> wrote:
> >
> > W.r.t. state data source - reader (SPARK-45511), there are several
> follow-up tickets, but we don't plan to address them soon. The current
> implementation is the final shape for Spark 4.0.0, unless there are demands
> on the follow-up tickets.
> >
> > We may want to check the plan for transformWithState - my understanding
> is that we want to release the feature to 4.0.0, but there are several
> remaining works to be done. While the tentative timeline for releasing is
> June 2024, what would be the tentative timeline for the RC cut?
> > (cc. Anish to add more context on the plan for transformWithState)
> >
> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan  wrote:
> > Hi all,
> >
> > It's close to the previously proposed 4.0.0 release date (June 2024),
> and I think it's time to prepare for it and discuss the ongoing projects:
> > •
> > ANSI by default
> > • Spark Connect GA
> > • Structured Logging
> > • Streaming state store data source
> > • new data type VARIANT
> > • STRING collation support
> > • Spark k8s operator versioning
> > Please help to add more items to this list that are missed here. I would
> like to volunteer as the release manager for Apache Spark 4.0.0 if there is
> no objection. Thank you all for the great work that fills Spark 4.0!
> >
> > Wenchen Fan
>
>

Re: [DISCUSS] Spark 4.0.0 release

[Cheng Pan]

will we have preview release for 4.0.0 like we did for 2.0.0 and 3.0.0?

Thanks,
Cheng Pan


> On Apr 15, 2024, at 09:58, Jungtaek Lim  wrote:
> 
> W.r.t. state data source - reader (SPARK-45511), there are several follow-up 
> tickets, but we don't plan to address them soon. The current implementation 
> is the final shape for Spark 4.0.0, unless there are demands on the follow-up 
> tickets.
> 
> We may want to check the plan for transformWithState - my understanding is 
> that we want to release the feature to 4.0.0, but there are several remaining 
> works to be done. While the tentative timeline for releasing is June 2024, 
> what would be the tentative timeline for the RC cut?
> (cc. Anish to add more context on the plan for transformWithState)
> 
> On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan  wrote:
> Hi all,
> 
> It's close to the previously proposed 4.0.0 release date (June 2024), and I 
> think it's time to prepare for it and discuss the ongoing projects:
> • 
> ANSI by default
> • Spark Connect GA
> • Structured Logging
> • Streaming state store data source
> • new data type VARIANT
> • STRING collation support
> • Spark k8s operator versioning
> Please help to add more items to this list that are missed here. I would like 
> to volunteer as the release manager for Apache Spark 4.0.0 if there is no 
> objection. Thank you all for the great work that fills Spark 4.0!
> 
> Wenchen Fan


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] Spark 4.0.0 release

[Jungtaek Lim]

W.r.t. state data source - reader (SPARK-45511
), there are several
follow-up tickets, but we don't plan to address them soon. The current
implementation is the final shape for Spark 4.0.0, unless there are demands
on the follow-up tickets.

We may want to check the plan for transformWithState - my understanding is
that we want to release the feature to 4.0.0, but there are several
remaining works to be done. While the tentative timeline for releasing is
June 2024, what would be the tentative timeline for the RC cut?
(cc. Anish to add more context on the plan for transformWithState)

On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan  wrote:

> Hi all,
>
> It's close to the previously proposed 4.0.0 release date (June 2024), and
> I think it's time to prepare for it and discuss the ongoing projects:
>
>- ANSI by default
>- Spark Connect GA
>- Structured Logging
>- Streaming state store data source
>- new data type VARIANT
>- STRING collation support
>- Spark k8s operator versioning
>
> Please help to add more items to this list that are missed here. I would
> like to volunteer as the release manager for Apache Spark 4.0.0 if there is
> no objection. Thank you all for the great work that fills Spark 4.0!
>
> Wenchen Fan
>

Re: [DISCUSS] Spark 4.0.0 release

[Dongjoon Hyun]

Thank you for volunteering, Wenchen.

Dongjoon.

On 2024/04/12 15:11:04 Wenchen Fan wrote:
> Hi all,
> 
> It's close to the previously proposed 4.0.0 release date (June 2024), and I
> think it's time to prepare for it and discuss the ongoing projects:
> 
>- ANSI by default
>- Spark Connect GA
>- Structured Logging
>- Streaming state store data source
>- new data type VARIANT
>- STRING collation support
>- Spark k8s operator versioning
> 
> Please help to add more items to this list that are missed here. I would
> like to volunteer as the release manager for Apache Spark 4.0.0 if there is
> no objection. Thank you all for the great work that fills Spark 4.0!
> 
> Wenchen Fan
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[DISCUSS] Spark 4.0.0 release

[Wenchen Fan]

Hi all,

It's close to the previously proposed 4.0.0 release date (June 2024), and I
think it's time to prepare for it and discuss the ongoing projects:

   - ANSI by default
   - Spark Connect GA
   - Structured Logging
   - Streaming state store data source
   - new data type VARIANT
   - STRING collation support
   - Spark k8s operator versioning

Please help to add more items to this list that are missed here. I would
like to volunteer as the release manager for Apache Spark 4.0.0 if there is
no objection. Thank you all for the great work that fills Spark 4.0!

Wenchen Fan