What is the functionality that would go into a 2.5.0 release, that can't be in a 2.4.7 release? I think that's the key question. 2.4.x is the 2.x maintenance branch, and I personally could imagine being open to more freely backporting a few new features for 2.x users, whereas usually it's only bug fixes. Making 2.5.0 implies that 2.5.x is the 2.x maintenance branch but there's something too big for a 'normal' maintenance release, and I think the whole question turns on what that is.
If it's things like JDK 11 support, I think that is unfortunately fairly 'breaking' because of dependency updates. But maybe that's not it. On Fri, Jun 12, 2020 at 4:38 PM Holden Karau <hol...@pigscanfly.ca> wrote: > Hi Folks, > > As we're getting closer to Spark 3 I'd like to revisit a Spark 2.5 > release. Spark 3 brings a number of important changes, and by its nature is > not backward compatible. I think we'd all like to have as smooth an upgrade > experience to Spark 3 as possible, and I believe that having a Spark 2 > release some of the new functionality while continuing to support the older > APIs and current Scala version would make the upgrade path smoother. > > This pattern is not uncommon in other Hadoop ecosystem projects, like > Hadoop itself and HBase. > > I know that Ryan Blue has indicated he is already going to be maintaining > something like that internally at Netflix, and we'll be doing the same > thing at Apple. It seems like having a transitional release could benefit > the community with easy migrations and help avoid duplicated work. > > I want to be clear I'm volunteering to do the work of managing a 2.5 > release, so hopefully, this wouldn't create any substantial burdens on the > community. > > Cheers, > > Holden > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
