Would that mean supporting both 2.12 and 2.11? Could be a while before some of our libraries are off of 2.11.
Thanks, Justin > On Jan 19, 2018, at 10:53 AM, Koert Kuipers <ko...@tresata.com> wrote: > > i was expecting to be able to move to scala 2.12 sometime this year > > if this cannot be done in spark 2.x then that could be a compelling reason to > move spark 3 up to 2018 i think > > hadoop 3 sounds great but personally i have no use case for it yet > > On Fri, Jan 19, 2018 at 12:31 PM, Sean Owen <so...@cloudera.com > <mailto:so...@cloudera.com>> wrote: > Forking this thread to muse about Spark 3. Like Spark 2, I assume it would be > more about making all those accumulated breaking changes and updating lots of > dependencies. Hadoop 3 looms large in that list as well as Scala 2.12. > > Spark 1 was release in May 2014, and Spark 2 in July 2016. If Spark 2.3 is > out in Feb 2018 and it takes the now-usual 6 months until a next release, > Spark 3 could reasonably be next. > > However the release cycles are naturally slowing down, and it could also be > said that 2019 would be more on schedule for Spark 3. > > Nothing particularly urgent about deciding, but I'm curious if anyone had an > opinion on whether to move on to Spark 3 next or just continue with 2.4 later > this year. > > On Fri, Jan 19, 2018 at 11:13 AM Sean Owen <so...@cloudera.com > <mailto:so...@cloudera.com>> wrote: > Yeah, if users are using Kryo directly, they should be insulated from a > Spark-side change because of shading. > However this also entails updating (unshaded) Chill from 0.8.x to 0.9.x. I am > not sure if that causes problems for apps. > > Normally I'd avoid any major-version change in a minor release. This one > looked potentially entirely internal. > I think if there are any doubts, we can leave it for Spark 3. There was a bug > report that needed a fix from Kryo 4, but it might be minor after all. > >
