Maintaining both a 1.7 and 2.0 is too much work for the project, which is over-stretched now. This means that after 1.6 it's just small maintenance releases in 1.x and no substantial features or evolution. This means that the "in progress" APIs in 1.x that will stay that way, unless one updates to 2.x. It's not unreasonable, but means the update to the 2.x line isn't going to be that optional for users.
Scala 2.10 is already EOL right? Supporting it in 2.x means supporting it for a couple years, note. 2.10 is still used today, but that's the point of the current stable 1.x release in general: if you want to stick to current dependencies, stick to the current release. Although I think that's the right way to think about support across major versions in general, I can see that 2.x is more of a required update for those following the project's fixes and releases. Hence may indeed be important to just keep supporting 2.10. I can't see supporting 2.12 at the same time (right?). Is that a concern? it will be long since GA by the time 2.x is first released. There's another fairly coherent worldview where development continues in 1.7 and focuses on finishing the loose ends and lots of bug fixing. 2.0 is delayed somewhat into next year, and by that time supporting 2.11+2.12 and Java 8 looks more feasible and more in tune with currently deployed versions. I can't say I have a strong view but I personally hadn't imagined 2.x would start now. On Thu, Nov 26, 2015 at 7:00 AM, Reynold Xin <r...@databricks.com> wrote: > I don't think we should drop support for Scala 2.10, or make it harder in > terms of operations for people to upgrade. > > If there are further objections, I'm going to bump remove the 1.7 version > and retarget things to 2.0 on JIRA. > > > On Wed, Nov 25, 2015 at 12:54 AM, Sandy Ryza <sandy.r...@cloudera.com> > wrote: >> >> I see. My concern is / was that cluster operators will be reluctant to >> upgrade to 2.0, meaning that developers using those clusters need to stay on >> 1.x, and, if they want to move to DataFrames, essentially need to port their >> app twice. >> >> I misunderstood and thought part of the proposal was to drop support for >> 2.10 though. If your broad point is that there aren't changes in 2.0 that >> will make it less palatable to cluster administrators than releases in the >> 1.x line, then yes, 2.0 as the next release sounds fine to me. >> >> -Sandy >> >> >> On Tue, Nov 24, 2015 at 11:55 AM, Matei Zaharia <matei.zaha...@gmail.com> >> wrote: >>> >>> What are the other breaking changes in 2.0 though? Note that we're not >>> removing Scala 2.10, we're just making the default build be against Scala >>> 2.11 instead of 2.10. There seem to be very few changes that people would >>> worry about. If people are going to update their apps, I think it's better >>> to make the other small changes in 2.0 at the same time than to update once >>> for Dataset and another time for 2.0. >>> >>> BTW just refer to Reynold's original post for the other proposed API >>> changes. >>> >>> Matei >>> >>> On Nov 24, 2015, at 12:27 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote: >>> >>> I think that Kostas' logic still holds. The majority of Spark users, and >>> likely an even vaster majority of people running vaster jobs, are still on >>> RDDs and on the cusp of upgrading to DataFrames. Users will probably want >>> to upgrade to the stable version of the Dataset / DataFrame API so they >>> don't need to do so twice. Requiring that they absorb all the other ways >>> that Spark breaks compatibility in the move to 2.0 makes it much more >>> difficult for them to make this transition. >>> >>> Using the same set of APIs also means that it will be easier to backport >>> critical fixes to the 1.x line. >>> >>> It's not clear to me that avoiding breakage of an experimental API in the >>> 1.x line outweighs these issues. >>> >>> -Sandy >>> >>> On Mon, Nov 23, 2015 at 10:51 PM, Reynold Xin <r...@databricks.com> >>> wrote: >>>> >>>> I actually think the next one (after 1.6) should be Spark 2.0. The >>>> reason is that I already know we have to break some part of the >>>> DataFrame/Dataset API as part of the Dataset design. (e.g. DataFrame.map >>>> should return Dataset rather than RDD). In that case, I'd rather break this >>>> sooner (in one release) than later (in two releases). so the damage is >>>> smaller. >>>> >>>> I don't think whether we call Dataset/DataFrame experimental or not >>>> matters too much for 2.0. We can still call Dataset experimental in 2.0 and >>>> then mark them as stable in 2.1. Despite being "experimental", there has >>>> been no breaking changes to DataFrame from 1.3 to 1.6. >>>> >>>> >>>> >>>> On Wed, Nov 18, 2015 at 3:43 PM, Mark Hamstra <m...@clearstorydata.com> >>>> wrote: >>>>> >>>>> Ah, got it; by "stabilize" you meant changing the API, not just bug >>>>> fixing. We're on the same page now. >>>>> >>>>> On Wed, Nov 18, 2015 at 3:39 PM, Kostas Sakellis <kos...@cloudera.com> >>>>> wrote: >>>>>> >>>>>> A 1.6.x release will only fix bugs - we typically don't change APIs in >>>>>> z releases. The Dataset API is experimental and so we might be changing >>>>>> the >>>>>> APIs before we declare it stable. This is why I think it is important to >>>>>> first stabilize the Dataset API with a Spark 1.7 release before moving to >>>>>> Spark 2.0. This will benefit users that would like to use the new Dataset >>>>>> APIs but can't move to Spark 2.0 because of the backwards incompatible >>>>>> changes, like removal of deprecated APIs, Scala 2.11 etc. >>>>>> >>>>>> Kostas >>>>>> >>>>>> >>>>>> On Fri, Nov 13, 2015 at 12:26 PM, Mark Hamstra >>>>>> <m...@clearstorydata.com> wrote: >>>>>>> >>>>>>> Why does stabilization of those two features require a 1.7 release >>>>>>> instead of 1.6.1? >>>>>>> >>>>>>> On Fri, Nov 13, 2015 at 11:40 AM, Kostas Sakellis >>>>>>> <kos...@cloudera.com> wrote: >>>>>>>> >>>>>>>> We have veered off the topic of Spark 2.0 a little bit here - yes we >>>>>>>> can talk about RDD vs. DS/DF more but lets refocus on Spark 2.0. I'd >>>>>>>> like to >>>>>>>> propose we have one more 1.x release after Spark 1.6. This will allow >>>>>>>> us to >>>>>>>> stabilize a few of the new features that were added in 1.6: >>>>>>>> >>>>>>>> 1) the experimental Datasets API >>>>>>>> 2) the new unified memory manager. >>>>>>>> >>>>>>>> I understand our goal for Spark 2.0 is to offer an easy transition >>>>>>>> but there will be users that won't be able to seamlessly upgrade given >>>>>>>> what >>>>>>>> we have discussed as in scope for 2.0. For these users, having a 1.x >>>>>>>> release >>>>>>>> with these new features/APIs stabilized will be very beneficial. This >>>>>>>> might >>>>>>>> make Spark 1.7 a lighter release but that is not necessarily a bad >>>>>>>> thing. >>>>>>>> >>>>>>>> Any thoughts on this timeline? >>>>>>>> >>>>>>>> Kostas Sakellis >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Nov 12, 2015 at 8:39 PM, Cheng, Hao <hao.ch...@intel.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Agree, more features/apis/optimization need to be added in DF/DS. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I mean, we need to think about what kind of RDD APIs we have to >>>>>>>>> provide to developer, maybe the fundamental API is enough, like, the >>>>>>>>> ShuffledRDD etc.. But PairRDDFunctions probably not in this >>>>>>>>> category, as we >>>>>>>>> can do the same thing easily with DF/DS, even better performance. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Mark Hamstra [mailto:m...@clearstorydata.com] >>>>>>>>> Sent: Friday, November 13, 2015 11:23 AM >>>>>>>>> To: Stephen Boesch >>>>>>>>> >>>>>>>>> >>>>>>>>> Cc: dev@spark.apache.org >>>>>>>>> Subject: Re: A proposal for Spark 2.0 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hmmm... to me, that seems like precisely the kind of thing that >>>>>>>>> argues for retaining the RDD API but not as the first thing presented >>>>>>>>> to new >>>>>>>>> Spark developers: "Here's how to use groupBy with DataFrames.... >>>>>>>>> Until the >>>>>>>>> optimizer is more fully developed, that won't always get you the best >>>>>>>>> performance that can be obtained. In these particular circumstances, >>>>>>>>> ..., >>>>>>>>> you may want to use the low-level RDD API while setting >>>>>>>>> preservesPartitioning to true. Like this...." >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 12, 2015 at 7:05 PM, Stephen Boesch <java...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> My understanding is that the RDD's presently have more support for >>>>>>>>> complete control of partitioning which is a key consideration at >>>>>>>>> scale. >>>>>>>>> While partitioning control is still piecemeal in DF/DS it would seem >>>>>>>>> premature to make RDD's a second-tier approach to spark dev. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> An example is the use of groupBy when we know that the source >>>>>>>>> relation (/RDD) is already partitioned on the grouping expressions. >>>>>>>>> AFAIK >>>>>>>>> the spark sql still does not allow that knowledge to be applied to the >>>>>>>>> optimizer - so a full shuffle will be performed. However in the >>>>>>>>> native RDD >>>>>>>>> we can use preservesPartitioning=true. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2015-11-12 17:42 GMT-08:00 Mark Hamstra <m...@clearstorydata.com>: >>>>>>>>> >>>>>>>>> The place of the RDD API in 2.0 is also something I've been >>>>>>>>> wondering about. I think it may be going too far to deprecate it, but >>>>>>>>> changing emphasis is something that we might consider. The RDD API >>>>>>>>> came >>>>>>>>> well before DataFrames and DataSets, so programming guides, >>>>>>>>> introductory >>>>>>>>> how-to articles and the like have, to this point, also tended to >>>>>>>>> emphasize >>>>>>>>> RDDs -- or at least to deal with them early. What I'm thinking is >>>>>>>>> that with >>>>>>>>> 2.0 maybe we should overhaul all the documentation to de-emphasize and >>>>>>>>> reposition RDDs. In this scheme, DataFrames and DataSets would be >>>>>>>>> introduced and fully addressed before RDDs. They would be presented >>>>>>>>> as the >>>>>>>>> normal/default/standard way to do things in Spark. RDDs, in >>>>>>>>> contrast, would >>>>>>>>> be presented later as a kind of lower-level, closer-to-the-metal API >>>>>>>>> that >>>>>>>>> can be used in atypical, more specialized contexts where DataFrames or >>>>>>>>> DataSets don't fully fit. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 12, 2015 at 5:17 PM, Cheng, Hao <hao.ch...@intel.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I am not sure what the best practice for this specific problem, but >>>>>>>>> it’s really worth to think about it in 2.0, as it is a painful issue >>>>>>>>> for >>>>>>>>> lots of users. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> By the way, is it also an opportunity to deprecate the RDD API (or >>>>>>>>> internal API only?)? As lots of its functionality overlapping with >>>>>>>>> DataFrame >>>>>>>>> or DataSet. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hao >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Kostas Sakellis [mailto:kos...@cloudera.com] >>>>>>>>> Sent: Friday, November 13, 2015 5:27 AM >>>>>>>>> To: Nicholas Chammas >>>>>>>>> Cc: Ulanov, Alexander; Nan Zhu; wi...@qq.com; dev@spark.apache.org; >>>>>>>>> Reynold Xin >>>>>>>>> >>>>>>>>> >>>>>>>>> Subject: Re: A proposal for Spark 2.0 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I know we want to keep breaking changes to a minimum but I'm hoping >>>>>>>>> that with Spark 2.0 we can also look at better classpath isolation >>>>>>>>> with user >>>>>>>>> programs. I propose we build on >>>>>>>>> spark.{driver|executor}.userClassPathFirst, >>>>>>>>> setting it true by default, and not allow any spark transitive >>>>>>>>> dependencies >>>>>>>>> to leak into user code. For backwards compatibility we can have a >>>>>>>>> whitelist >>>>>>>>> if we want but I'd be good if we start requiring user apps to >>>>>>>>> explicitly >>>>>>>>> pull in all their dependencies. From what I can tell, Hadoop 3 is also >>>>>>>>> moving in this direction. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Kostas >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 12, 2015 at 9:56 AM, Nicholas Chammas >>>>>>>>> <nicholas.cham...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> With regards to Machine learning, it would be great to move useful >>>>>>>>> features from MLlib to ML and deprecate the former. Current structure >>>>>>>>> of two >>>>>>>>> separate machine learning packages seems to be somewhat confusing. >>>>>>>>> >>>>>>>>> With regards to GraphX, it would be great to deprecate the use of >>>>>>>>> RDD in GraphX and switch to Dataframe. This will allow GraphX evolve >>>>>>>>> with >>>>>>>>> Tungsten. >>>>>>>>> >>>>>>>>> On that note of deprecating stuff, it might be good to deprecate >>>>>>>>> some things in 2.0 without removing or replacing them immediately. >>>>>>>>> That way >>>>>>>>> 2.0 doesn’t have to wait for everything that we want to deprecate to >>>>>>>>> be >>>>>>>>> replaced all at once. >>>>>>>>> >>>>>>>>> Nick >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 12, 2015 at 12:45 PM Ulanov, Alexander >>>>>>>>> <alexander.ula...@hpe.com> wrote: >>>>>>>>> >>>>>>>>> Parameter Server is a new feature and thus does not match the goal >>>>>>>>> of 2.0 is “to fix things that are broken in the current API and remove >>>>>>>>> certain deprecated APIs”. At the same time I would be happy to have >>>>>>>>> that >>>>>>>>> feature. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> With regards to Machine learning, it would be great to move useful >>>>>>>>> features from MLlib to ML and deprecate the former. Current structure >>>>>>>>> of two >>>>>>>>> separate machine learning packages seems to be somewhat confusing. >>>>>>>>> >>>>>>>>> With regards to GraphX, it would be great to deprecate the use of >>>>>>>>> RDD in GraphX and switch to Dataframe. This will allow GraphX evolve >>>>>>>>> with >>>>>>>>> Tungsten. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Best regards, Alexander >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Nan Zhu [mailto:zhunanmcg...@gmail.com] >>>>>>>>> Sent: Thursday, November 12, 2015 7:28 AM >>>>>>>>> To: wi...@qq.com >>>>>>>>> Cc: dev@spark.apache.org >>>>>>>>> Subject: Re: A proposal for Spark 2.0 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Being specific to Parameter Server, I think the current agreement >>>>>>>>> is that PS shall exist as a third-party library instead of a >>>>>>>>> component of >>>>>>>>> the core code base, isn’t? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Nan Zhu >>>>>>>>> >>>>>>>>> http://codingcat.me >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thursday, November 12, 2015 at 9:49 AM, wi...@qq.com wrote: >>>>>>>>> >>>>>>>>> Who has the idea of machine learning? Spark missing some features >>>>>>>>> for machine learning, For example, the parameter server. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 在 2015年11月12日，05:32，Matei Zaharia <matei.zaha...@gmail.com> 写道： >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I like the idea of popping out Tachyon to an optional component too >>>>>>>>> to reduce the number of dependencies. In the future, it might even be >>>>>>>>> useful >>>>>>>>> to do this for Hadoop, but it requires too many API changes to be >>>>>>>>> worth >>>>>>>>> doing now. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regarding Scala 2.12, we should definitely support it eventually, >>>>>>>>> but I don't think we need to block 2.0 on that because it can be >>>>>>>>> added later >>>>>>>>> too. Has anyone investigated what it would take to run on there? I >>>>>>>>> imagine >>>>>>>>> we don't need many code changes, just maybe some REPL stuff. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Needless to say, but I'm all for the idea of making "major" >>>>>>>>> releases as undisruptive as possible in the model Reynold proposed. >>>>>>>>> Keeping >>>>>>>>> everyone working with the same set of releases is super important. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Matei >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Nov 11, 2015, at 4:58 AM, Sean Owen <so...@cloudera.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 11, 2015 at 12:10 AM, Reynold Xin <r...@databricks.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> to the Spark community. A major release should not be very >>>>>>>>> different from a >>>>>>>>> >>>>>>>>> minor release and should not be gated based on new features. The >>>>>>>>> main >>>>>>>>> >>>>>>>>> purpose of a major release is an opportunity to fix things that are >>>>>>>>> broken >>>>>>>>> >>>>>>>>> in the current API and remove certain deprecated APIs (examples >>>>>>>>> follow). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Agree with this stance. Generally, a major release might also be a >>>>>>>>> >>>>>>>>> time to replace some big old API or implementation with a new one, >>>>>>>>> but >>>>>>>>> >>>>>>>>> I don't see obvious candidates. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I wouldn't mind turning attention to 2.x sooner than later, unless >>>>>>>>> >>>>>>>>> there's a fairly good reason to continue adding features in 1.x to >>>>>>>>> a >>>>>>>>> >>>>>>>>> 1.7 release. The scope as of 1.6 is already pretty darned big. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. Scala 2.11 as the default build. We should still support Scala >>>>>>>>> 2.10, but >>>>>>>>> >>>>>>>>> it has been end-of-life. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> By the time 2.x rolls around, 2.12 will be the main version, 2.11 >>>>>>>>> will >>>>>>>>> >>>>>>>>> be quite stable, and 2.10 will have been EOL for a while. I'd >>>>>>>>> propose >>>>>>>>> >>>>>>>>> dropping 2.10. Otherwise it's supported for 2 more years. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2. Remove Hadoop 1 support. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'd go further to drop support for <2.2 for sure (2.0 and 2.1 were >>>>>>>>> >>>>>>>>> sort of 'alpha' and 'beta' releases) and even <2.6. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm sure we'll think of a number of other small things -- shading a >>>>>>>>> >>>>>>>>> bunch of stuff? reviewing and updating dependencies in light of >>>>>>>>> >>>>>>>>> simpler, more recent dependencies to support from Hadoop etc? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Farming out Tachyon to a module? (I felt like someone proposed >>>>>>>>> this?) >>>>>>>>> >>>>>>>>> Pop out any Docker stuff to another repo? >>>>>>>>> >>>>>>>>> Continue that same effort for EC2? >>>>>>>>> >>>>>>>>> Farming out some of the "external" integrations to another repo (? >>>>>>>>> >>>>>>>>> controversial) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> See also anything marked version "2+" in JIRA. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>> >>>>>>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>> >>>>>>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>> >>>>>>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
