Alright I just merged this in - so Spark is officially "Scala 2.10" from here forward.
For reference I cut a new branch called scala-2.9 with the commit immediately prior to the merge: https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9 - Patrick On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Hey Reymond, > > Let's move this discussion out of this thread and into the associated JIRA. > I'll write up our current approach over there. > > https://spark-project.atlassian.net/browse/SPARK-995 > > - Patrick > > > On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <raymond....@intel.com> wrote: >> >> Hi Patrick >> >> So what's the plan for support Yarn 2.2 in 0.9? As far as I can >> see, if you want to support both 2.2 and 2.0 , due to protobuf version >> incompatible issue. You need two version of akka anyway. >> >> Akka 2.3-M1 looks like have a little bit change in API, we >> probably could isolate the code like what we did on yarn part API. I >> remember that it is mentioned that to use reflection for different API is >> preferred. So the purpose to use reflection is to use one release bin jar to >> support both version of Hadoop/Yarn on runtime, instead of build different >> bin jar on compile time? >> >> Then all code related to hadoop will also be built in separate >> modules for loading on demand? This sounds to me involve a lot of works. And >> you still need to have shim layer and separate code for different version >> API and depends on different version Akka etc. Sounds like and even strict >> demands versus our current approaching on master, and with dynamic class >> loader in addition, And the problem we are facing now are still there? >> >> Best Regards, >> Raymond Liu >> >> -----Original Message----- >> From: Patrick Wendell [mailto:pwend...@gmail.com] >> Sent: Thursday, December 12, 2013 5:13 PM >> To: dev@spark.incubator.apache.org >> Subject: Re: Scala 2.10 Merge >> >> Also - the code is still there because of a recent merge that took in some >> newer changes... we'll be removing it for the final merge. >> >> >> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> >> > Hey Raymond, >> > >> > This won't work because AFAIK akka 2.3-M1 is not binary compatible >> > with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need >> > to still use the older protobuf library, so we'd need to support both. >> > >> > I'd also be concerned about having a reference to a non-released >> > version of akka. Akka is the source of our hardest-to-find bugs and >> > simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting. >> > Of course, if you are building off of master you can maintain a fork >> > that uses this. >> > >> > - Patrick >> > >> > >> > On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond >> > <raymond....@intel.com>wrote: >> > >> >> Hi Patrick >> >> >> >> What does that means for drop YARN 2.2? seems codes are still >> >> there. You mean if build upon 2.2 it will break, and won't and work >> >> right? >> >> Since the home made akka build on scala 2.10 are not there. While, if >> >> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5 >> >> for replacement? >> >> >> >> Best Regards, >> >> Raymond Liu >> >> >> >> >> >> -----Original Message----- >> >> From: Patrick Wendell [mailto:pwend...@gmail.com] >> >> Sent: Thursday, December 12, 2013 4:21 PM >> >> To: dev@spark.incubator.apache.org >> >> Subject: Scala 2.10 Merge >> >> >> >> Hi Developers, >> >> >> >> In the next few days we are planning to merge Scala 2.10 support into >> >> Spark. For those that haven't been following this, Prashant Sharma >> >> has been maintaining the scala-2.10 branch of Spark for several >> >> months. This branch is current with master and has been reviewed for >> >> merging: >> >> >> >> https://github.com/apache/incubator-spark/tree/scala-2.10 >> >> >> >> Scala 2.10 support is one of the most requested features for Spark - >> >> it will be great to get this into Spark 0.9! Please note that *Scala >> >> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I >> >> wanted to give a few heads-up/requests to developers: >> >> >> >> If you are developing applications on top of Spark's master branch, >> >> those will need to migrate to Scala 2.10. You may want to download >> >> and test the current scala-2.10 branch in order to make sure you will >> >> be okay as Spark developments move forward. Of course, you can always >> >> stick with the current master commit and be fine (I'll cut a tag when >> >> we do the merge in order to delineate where the version changes). >> >> Please open new threads on the dev list to report and discuss any >> >> issues. >> >> >> >> This merge will temporarily drop support for YARN 2.2 on the master >> >> branch. >> >> This is because the workaround we used was only compiled for Scala 2.9. >> >> We are going to come up with a more robust solution to YARN 2.2 >> >> support before releasing 0.9. >> >> >> >> Going forward, we will continue to make maintenance releases on >> >> branch-0.8 which will remain compatible with Scala 2.9. >> >> >> >> For those interested, the primary code changes in this merge are >> >> upgrading the akka version, changing the use of Scala 2.9's >> >> ClassManifest construct to Scala 2.10's ClassTag, and updating the >> >> spark shell to work with Scala 2.10's repl. >> >> >> >> - Patrick >> >> >> > >> > > >
