Great job everyone! A big step forward.
On Sat, Dec 14, 2013 at 2:37 AM, andy.petre...@gmail.com < andy.petre...@gmail.com> wrote: > That's a very good news! > Congrats > > Envoyé depuis mon HTC > > ----- Reply message ----- > De : "Sam Bessalah" <samkil...@gmail.com> > Pour : "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org> > Objet : Scala 2.10 Merge > Date : sam., déc. 14, 2013 11:03 > > > Yes. Awesome. > Great job guys. > > Sam Bessalah > > > On Dec 14, 2013, at 9:59 AM, Patrick Wendell <pwend...@gmail.com> wrote: > > > > Alright I just merged this in - so Spark is officially "Scala 2.10" > > from here forward. > > > > For reference I cut a new branch called scala-2.9 with the commit > > immediately prior to the merge: > > > https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9 > > > > - Patrick > > > >> On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> Hey Reymond, > >> > >> Let's move this discussion out of this thread and into the associated > JIRA. > >> I'll write up our current approach over there. > >> > >> https://spark-project.atlassian.net/browse/SPARK-995 > >> > >> - Patrick > >> > >> > >>> On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <raymond....@intel.com> > wrote: > >>> > >>> Hi Patrick > >>> > >>> So what's the plan for support Yarn 2.2 in 0.9? As far as I can > >>> see, if you want to support both 2.2 and 2.0 , due to protobuf version > >>> incompatible issue. You need two version of akka anyway. > >>> > >>> Akka 2.3-M1 looks like have a little bit change in API, we > >>> probably could isolate the code like what we did on yarn part API. I > >>> remember that it is mentioned that to use reflection for different API > is > >>> preferred. So the purpose to use reflection is to use one release bin > jar to > >>> support both version of Hadoop/Yarn on runtime, instead of build > different > >>> bin jar on compile time? > >>> > >>> Then all code related to hadoop will also be built in separate > >>> modules for loading on demand? This sounds to me involve a lot of > works. And > >>> you still need to have shim layer and separate code for different > version > >>> API and depends on different version Akka etc. Sounds like and even > strict > >>> demands versus our current approaching on master, and with dynamic > class > >>> loader in addition, And the problem we are facing now are still there? > >>> > >>> Best Regards, > >>> Raymond Liu > >>> > >>> -----Original Message----- > >>> From: Patrick Wendell [mailto:pwend...@gmail.com] > >>> Sent: Thursday, December 12, 2013 5:13 PM > >>> To: dev@spark.incubator.apache.org > >>> Subject: Re: Scala 2.10 Merge > >>> > >>> Also - the code is still there because of a recent merge that took in > some > >>> newer changes... we'll be removing it for the final merge. > >>> > >>> > >>> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <pwend...@gmail.com> > >>> wrote: > >>> > >>>> Hey Raymond, > >>>> > >>>> This won't work because AFAIK akka 2.3-M1 is not binary compatible > >>>> with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need > >>>> to still use the older protobuf library, so we'd need to support both. > >>>> > >>>> I'd also be concerned about having a reference to a non-released > >>>> version of akka. Akka is the source of our hardest-to-find bugs and > >>>> simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting. > >>>> Of course, if you are building off of master you can maintain a fork > >>>> that uses this. > >>>> > >>>> - Patrick > >>>> > >>>> > >>>> On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond > >>>> <raymond....@intel.com>wrote: > >>>> > >>>>> Hi Patrick > >>>>> > >>>>> What does that means for drop YARN 2.2? seems codes are still > >>>>> there. You mean if build upon 2.2 it will break, and won't and work > >>>>> right? > >>>>> Since the home made akka build on scala 2.10 are not there. While, if > >>>>> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5 > >>>>> for replacement? > >>>>> > >>>>> Best Regards, > >>>>> Raymond Liu > >>>>> > >>>>> > >>>>> -----Original Message----- > >>>>> From: Patrick Wendell [mailto:pwend...@gmail.com] > >>>>> Sent: Thursday, December 12, 2013 4:21 PM > >>>>> To: dev@spark.incubator.apache.org > >>>>> Subject: Scala 2.10 Merge > >>>>> > >>>>> Hi Developers, > >>>>> > >>>>> In the next few days we are planning to merge Scala 2.10 support into > >>>>> Spark. For those that haven't been following this, Prashant Sharma > >>>>> has been maintaining the scala-2.10 branch of Spark for several > >>>>> months. This branch is current with master and has been reviewed for > >>>>> merging: > >>>>> > >>>>> https://github.com/apache/incubator-spark/tree/scala-2.10 > >>>>> > >>>>> Scala 2.10 support is one of the most requested features for Spark - > >>>>> it will be great to get this into Spark 0.9! Please note that *Scala > >>>>> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I > >>>>> wanted to give a few heads-up/requests to developers: > >>>>> > >>>>> If you are developing applications on top of Spark's master branch, > >>>>> those will need to migrate to Scala 2.10. You may want to download > >>>>> and test the current scala-2.10 branch in order to make sure you will > >>>>> be okay as Spark developments move forward. Of course, you can always > >>>>> stick with the current master commit and be fine (I'll cut a tag when > >>>>> we do the merge in order to delineate where the version changes). > >>>>> Please open new threads on the dev list to report and discuss any > >>>>> issues. > >>>>> > >>>>> This merge will temporarily drop support for YARN 2.2 on the master > >>>>> branch. > >>>>> This is because the workaround we used was only compiled for Scala > 2.9. > >>>>> We are going to come up with a more robust solution to YARN 2.2 > >>>>> support before releasing 0.9. > >>>>> > >>>>> Going forward, we will continue to make maintenance releases on > >>>>> branch-0.8 which will remain compatible with Scala 2.9. > >>>>> > >>>>> For those interested, the primary code changes in this merge are > >>>>> upgrading the akka version, changing the use of Scala 2.9's > >>>>> ClassManifest construct to Scala 2.10's ClassTag, and updating the > >>>>> spark shell to work with Scala 2.10's repl. > >>>>> > >>>>> - Patrick > >> > >> > -- -- Evan Chan Staff Engineer e...@ooyala.com | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
