Hi all, Thanks for sharing the progress of ongoing projects! Let me summarize them here: - Add Spark Connect config to allow simple switch [PR <https://github.com/apache/spark/pull/49107>] - ML algorithms on Spark Connect (doesn't block 4.0) [JIRA <https://issues.apache.org/jira/browse/SPARK-50812>] - DataFrame conversion to table argument in Spark Connect [PR <https://github.com/apache/spark/pull/49424>] - SQL user-defined functions (last PR <https://github.com/apache/spark/pull/49471> to support TVF) - unified Scala interface shared by classic and Connect (main PR merged) [ JIRA <https://issues.apache.org/jira/browse/SPARK-48918>] - Structured Streaming - Arbitrary State API v2 (only a few items left) [ JIRA <https://issues.apache.org/jira/browse/SPARK-46815>] - Recursive CTE (last PR <https://github.com/apache/spark/pull/49571> to support execution) - String collation support (only a few items left) [JIRA <https://issues.apache.org/jira/projects/SPARK/issues/SPARK-46830>] - SQL Scripting (only a few items left) [JIRA <https://issues.apache.org/jira/browse/SPARK-48338>] - AQE OOM [JIRA <https://issues.apache.org/jira/browse/SPARK-50992>]
Please let me know if you have other projects not tracked here. We can still backport commits for them to 4.0 before the RC cut (Feb 15). If you think your projects may not be finished before the RC cut, let's discuss and see how we can re-target your projects to 4.1 or have a simpler working version in 4.0. For other new projects, please do not merge them to branch-4.0. Thanks, Wenchen On Wed, Jan 22, 2025 at 7:03 PM Milan Cupac <milan.cu...@databricks.com.invalid> wrote: > I am working on recursive CTEs. > > Two final PRs should be merged soon: > https://github.com/apache/spark/pull/49518 > https://github.com/apache/spark/pull/49571 > > 2025/01/15 13:41:07 Wenchen Fan wrote: > > Hi all, > > > > We have cut the "branch-4.0" and I'm sending this email to collect the > > information for ongoing projects targeting Spark 4.0. Please reply to > this > > email to share the project progress with the community. > > > > Note that, the scheduled code freeze date is Feb 1, and RC1 cut date is > Feb > > 15. > > > > Thanks, > > Wenchen > > >
