[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841287#comment-16841287 ]
Lukas Rytz commented on SPARK-25075: ------------------------------------ In our own interest of testing Scala the 2.13 RCs, I took a stab at compiling spark core on 2.13. My goal was *not* to get something running, just compiling and seeing what kind of breaking changes there are. So I chose a slightly adventurous methodology: I updated the {{scalaVersion}} to 2.13.0-pre-06392a5-SNAPSHOT (a very recent local build), but forced the {{scalaBinaryVersion}} to 2.12, so that the 2.12 dependencies end up on the classpath. That way I didn't have to worry about missing / incompatible dependencies. The first step was to avoid using {{scala.Seq}}, I used scalafix to rewrite all references of the type {{scala.Seq}} to {{scala.collection.Seq}}. As discussed on https://issues.apache.org/jira/browse/SPARK-27681, this is not necessarily the best solution, but the easiest. Here's a list of other breaking changes: * {{foo(someMutableOrGenericCollection: _*)}} no longer works, because varargs de-sugars to {{scala.Seq}}, so an immutable collection is now required. Calling {{.toSeq}} works, but is inefficient. Better build an immutable collection from the beginning. For arrays, {{immutable.ArraySeq.unsafeWrapArray}} can be used. Maybe the standard library should provide an unsafe {{immutable.SeqWrapper}} that wraps a {{collection.Seq}} for the cases when the users are certain it's safe. * Views are quite different in 2.13, for example {{seq.view}} is no longer a {{Seq}}, views are a separate hierarchy. This needs some adjustments, not too difficult. * Parallel collections are a separate module now ([https://github.com/scala/scala-parallel-collections]), no longer in the standard library. However, you might want to use StreamConverters instead to do parallel processing via a Java stream ([https://github.com/scala/scala/blob/2.13.x/src/library/scala/jdk/StreamConverters.scala]). * Subclasses of collections need some adjustments: {{BoundedPriorityQueue}}, {{TimeStampedHashMap}}. For example, {{++=}} cannot be overridden anymore, as it's a final alias for {{addAll}} now. The other changes are relatively minor. The brannch is here - for reference, too hacky to be actually useful: [https://github.com/lrytz/spark/commits/2.13-experiment]. {{sbt core/compile}} passes (except for scalastyle). Overall, this is more or less what we expected in terms of breaking changes. We definietly want to use the time between now and 2.13.0 final to improve migration documentation ([https://docs.scala-lang.org/overviews/core/collections-migration-213.html]) and scalafix rules ([https://github.com/scala/scala-rewrites] is in its early days). > Build and test Spark against Scala 2.13 > --------------------------------------- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, Project Infra > Affects Versions: 3.0.0 > Reporter: Guillaume Massé > Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org