Hi Tanveer, The approach with Spark Connect from Dangjoon Hyun seems like a good start, if we want to run Scala 3 Applications with a Spark backend
However I would also like to see a Scala 3 Build of Spark itself, as it would migrating existing applications easier. For that, it’s maybe a good Idea to just start with a small fork to gather more information: - Update https://github.com/apache/spark/pull/50474 - There doesn’t seem to be too much Scala Macros in the Codebase. Also there is no Shapeless. Good. - UDFs, DataSet, Encoders, ScalaReflection etc. are using Typetag to encode Decoders. This should be exchanged into some Spark-owned Typeclass, which can then describe Scala 2/Scala 3 specific ways. The Scala 2 Code can then still rely on TypeTags - Enabling Scala 3.3.x on the code and see what breaks. At least Scala with SBT supports Scala-Version specific Code paths (e.g. src/main/scala-3, Scala-2). I am sure, Maven can do this too. Scala-2-Specific Code goes to scala-2. Stubs should make it possible, to compile in Scala-3. - Implementing the stubs for Scala 3 and see how it goes. Typetags should possible be replaceable by a combination of ClassTag and Mirror.ProductOf (guessing) This could also be possible in a sub-project-wise fashion. The Scala 3 Code style should be as similar as the existing Scala 2 Style, in order to not make it more complicated, so Brace-Style and no unnecessary new futures. Note: I am not deep in the Spark source code. Kind Regards, Norbert > Am 04.11.2025 um 12:10 schrieb Tanveer Zia <[email protected]>: > > Hi everyone, > > I’m Tanveer from Scala Teams. We’re interested in contributing to the Scala 3 > migration of Apache Spark, as referenced in SPARK-54150 > <https://issues.apache.org/jira/browse/SPARK-54150>. > > Could you please share the current status or any existing roadmap for this > migration? We’d also appreciate guidance on how external contributors can > best get involved or coordinate with the core team on next steps. > > Best regards, > Tanveer Zia > Scala Teams > -- Reactive Core GmbH | Paul-Lincke-Ufer 8b | 10999 Berlin Fon: +49 30 9832 4666 | Web: www.reactivecore.de <http://www.reactivecore.de> Handelsregister: Amtsgericht Charlottenburg HRB 156696 B Sitz: Berlin | Geschäftsführer: Norbert Schultz
