Hey all, Reviving this thread, but Spark master has already accumulated a huge amount of changes. As a downstream project maintainer, I want to really start testing the new features and other breaking changes, and it's hard to do that without a Preview release. So the sooner we make a Preview release, the faster we can start getting feedback for fixing things for a great Spark 4.0 final release.
So I urge the community to produce a Spark 4.0 Preview soon even if certain features targeting the Delta 4.0 release are still incomplete. Thanks! On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan <cloud0...@gmail.com> wrote: > Thank you all for the replies! > > To @Nicholas Chammas <nicholas.cham...@gmail.com> : Thanks for cleaning > up the error terminology and documentation! I've merged the first PR and > let's finish others before the 4.0 release. > To @Dongjoon Hyun <dongjoon.h...@gmail.com> : Thanks for driving the ANSI > on by default effort! Now the vote has passed, let's flip the config and > finish the DataFrame error context feature before 4.0. > To @Jungtaek Lim <kabhwan.opensou...@gmail.com> : Ack. We can treat the > Streaming state store data source as completed for 4.0 then. > To @Cheng Pan <cheng...@apache.org> : Yea we definitely should have a > preview release. Let's collect more feedback on the ongoing projects and > then we can propose a date for the preview release. > > On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan <pan3...@gmail.com> wrote: > >> will we have preview release for 4.0.0 like we did for 2.0.0 and 3.0.0? >> >> Thanks, >> Cheng Pan >> >> >> > On Apr 15, 2024, at 09:58, Jungtaek Lim <kabhwan.opensou...@gmail.com> >> wrote: >> > >> > W.r.t. state data source - reader (SPARK-45511), there are several >> follow-up tickets, but we don't plan to address them soon. The current >> implementation is the final shape for Spark 4.0.0, unless there are demands >> on the follow-up tickets. >> > >> > We may want to check the plan for transformWithState - my understanding >> is that we want to release the feature to 4.0.0, but there are several >> remaining works to be done. While the tentative timeline for releasing is >> June 2024, what would be the tentative timeline for the RC cut? >> > (cc. Anish to add more context on the plan for transformWithState) >> > >> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan <cloud0...@gmail.com> >> wrote: >> > Hi all, >> > >> > It's close to the previously proposed 4.0.0 release date (June 2024), >> and I think it's time to prepare for it and discuss the ongoing projects: >> > • >> > ANSI by default >> > • Spark Connect GA >> > • Structured Logging >> > • Streaming state store data source >> > • new data type VARIANT >> > • STRING collation support >> > • Spark k8s operator versioning >> > Please help to add more items to this list that are missed here. I >> would like to volunteer as the release manager for Apache Spark 4.0.0 if >> there is no objection. Thank you all for the great work that fills Spark >> 4.0! >> > >> > Wenchen Fan >> >>
