It's great to hear from you that you are interested in the discussion of Apache Spark 4.0 scope and timeframe. :)
This is the initial thread which includes upgrading the default Scala version to 2.13 or 3.3 is the one of the reasons. As we know, Apache Spark 4.0 is not limited to this. For example, dropping Java 8 is also discussed in "JDK version support policy?" thread separately. In principle, this thread aims to avoid blocking the Apache Spark community by saying 'No for all; (1) No until Apache Spark 4.0 and (2) No 4.0 because we don't have much". So, could you share your opinion for Apache Spark 4.0 here? Best, Dongjoon. On Wed, May 31, 2023 at 6:02 PM Dongjoon Hyun <dongj...@apache.org> wrote: > Hi, All. > > I'd like to propose to start to prepare Apache Spark 4.0 after creating > branch-3.5 on July 16th. > > - https://spark.apache.org/versioning-policy.html > > Historically, the Apache Spark release dates have the following timeframes > and we already have Spark 3.5 plan which will be maintained up to 2026. > > Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3) > Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8) > Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x) > Spark 4: 2024.06 (4.0.0, NEW) > > As we discussed in the previous email thread, `Apache Spark 3.5.0 > Expectations`, we cannot deliver some features without Apache Spark 4. > > - "I wonder if it’s safer to do it in Spark 4 (which I believe will be > discussed soon)." > - "I would make it the default at 4.0, myself." > > Although there exist more other features, let's focus on Scala language > support history. > > Spark 2.0: SPARK-6363 Make Scala 2.11 the default Scala version (2016.07) > Spark 3.0: SPARK-25956 Make Scala 2.12 as default Scala version in Spark > 3.0 (2020.06) > > In addition, the Scala community released Scala 3.3.0 LTS yesterday. > > - https://scala-lang.org/blog/2023/05/30/scala-3.3.0-released.html > > If we decide to start, I believe we can support Scala 2.13 or Scala 3.3 > next year with Apache Spark 4 while supporting Spark 3.4 and 3.5 for Scala > 2.12 users. > > WDYT? > > Thanks, > Dongjoon. >
