W.r.t. state data source - reader (SPARK-45511 <https://issues.apache.org/jira/browse/SPARK-45511>), there are several follow-up tickets, but we don't plan to address them soon. The current implementation is the final shape for Spark 4.0.0, unless there are demands on the follow-up tickets.
We may want to check the plan for transformWithState - my understanding is that we want to release the feature to 4.0.0, but there are several remaining works to be done. While the tentative timeline for releasing is June 2024, what would be the tentative timeline for the RC cut? (cc. Anish to add more context on the plan for transformWithState) On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan <cloud0...@gmail.com> wrote: > Hi all, > > It's close to the previously proposed 4.0.0 release date (June 2024), and > I think it's time to prepare for it and discuss the ongoing projects: > > - ANSI by default > - Spark Connect GA > - Structured Logging > - Streaming state store data source > - new data type VARIANT > - STRING collation support > - Spark k8s operator versioning > > Please help to add more items to this list that are missed here. I would > like to volunteer as the release manager for Apache Spark 4.0.0 if there is > no objection. Thank you all for the great work that fills Spark 4.0! > > Wenchen Fan >
