well, I think such a release is reasonable given that we focused so far mostly on robustness features (parfor, OOMs, ultra-sparse) and various fixes due to the uncertainty of the upcoming release. The new code generator is also in reasonable shape to go out as an experimental feature. Similarly, I don't see major issues caused by our move to Java 8 as well as its exploitation via scalable counters, streams and lambda expressions.
With regard to backports of fixes into 0.14, however, let's decide on a case-by-case basis as a general policy like that would create considerable overhead. Regards, Matthias On Wed, Mar 22, 2017 at 11:47 PM, Arvind Surve <ac...@yahoo.com.invalid> wrote: > We have released SystemML 0.13, first code based on Spark 2.0.x, on March > 2nd 2017.Some users may have transformed to SystemML 0.13 with some > effort. Its not good idea to go out immediately with SystemML 1.0 as it has > high potential to break backward compatibility.In that case, SystemML 0.13 > users who need to get later version/fix, they have to go with additional > changes due to losing backward compatibility. > To avoid immediate burden on current users I am proposing following. > Lets cut SystemML 0.14 branch with current changes based on master > branch.Then onward master branch can be used to do changes specific to > SystemML 1.0 based code and any fixes based on current code.Whoever add any > fix to SystemML 1.0 (master branch then) will need to put fixes on SystemML > 0.14 branch as well as applicable to SystemML 0.13 code.Features added in > SystemML 1.0 (master branch then) should not go into SystemML 0.14 branch. > If there is no disagreement, then I will create new release SystemML 0.14 > on 03/30/2017. Please make sure to get your code in by EOD 03/29/2017. > -Arvind Arvind Surve | Spark Technology Center | http://www.spark.tc/ > > From: Berthold Reinwald <reinw...@us.ibm.com> > To: dev@systemml.incubator.apache.org > Sent: Sunday, March 12, 2017 5:54 PM > Subject: Re: Release cadence > > i am fine with 1.0 but let us stage it in a way that back porting of > possible bug fixes will not be too difficult in the next few weeks or > small nbr of month. > > Regards, > Berthold Reinwald > IBM Almaden Research Center > office: (408) 927 2208; T/L: 457 2208 > e-mail: reinw...@us.ibm.com > > > > From: Matthias Boehm <mboe...@googlemail.com> > To: dev@systemml.incubator.apache.org > Date: 03/12/2017 05:15 PM > Subject: Re: Release cadence > > > > ok great - as the majority is in favor of a 1.0 release and there is no > veto, I think we came to an agreement here: our next release will be > SystemML 1.0. > > Regards, > Matthias > > > On Tue, Mar 7, 2017 at 11:30 AM, <dusenberr...@gmail.com> wrote: > > > +1 for immediately starting work on SystemML 1.0 as our next release. > > > > At this point, the project and our users will benefit most from a > thorough > > cleanup, as it will make the project simpler to use and easier to > > maintain. Simplicity will allow users and maintainers to regain focus > on > > ML research and products, which is a win for the entire community. We > > should create a solid list of items that we, and the rest of the > community, > > want to address for the 1.0 release and make sure that they are indeed > > completed. At the same time, we should ensure that we don't drag out > the > > release process. > > > > -Mike > > > > -- > > > > Mike Dusenberry > > GitHub: github.com/dusenberrymw > > LinkedIn: linkedin.com/in/mikedusenberry > > > > Sent from my iPhone. > > > > > > > On Mar 6, 2017, at 10:14 AM, Luciano Resende <luckbr1...@gmail.com> > > wrote: > > > > > > +1 for SystemML 1.0 as the next release. > > > > > > On Sat, Mar 4, 2017 at 10:23 AM, Deron Eriksson > <deroneriks...@gmail.com > > > > > > wrote: > > > > > >> Personally I would like the next release to be 1.0. We have been an > > >> incubator project since November 2015 and I believe that after over > > 1,000 > > >> commits since then that the project is about ready for a 1.0 release. > > >> > > >> I agree with Matthias that we need to make a decision regarding this > > topic. > > >> For new issues and fixed issues in JIRA, we need to be able to assign > > the > > >> correct version, or else someone potentially needs to go through and > fix > > >> the version numbers, as Glenn has been doing. Additionally, it would > be > > >> nice to do some of the 1.0 code updates (such as removing the old > > >> MLContext) now rather than waiting additional months. Also I would > like > > to > > >> be able to correctly identify our next version in the online > > documentation. > > >> > > >> > > > How about just make SystemML Next and change the release name when we > do > > > the release ? > > > > > > > > > > > >> Deron > > >> > > >> > > >> On Sat, Mar 4, 2017 at 12:47 AM, Matthias Boehm > <mboe...@googlemail.com > > > > > >> wrote: > > >> > > >>> thanks Arvind for bringing some structure to the release process. I > > >> think a > > >>> fixed cadence of 2 months is useful as it makes upcoming releases > more > > >>> predictable for devs and users. > > >>> > > >>> However, we're discussing a major 1.0 release for a while now. I > think > > it > > >>> would be useful to come to an agreement if we go for 1.0 in April or > > not. > > >>> There are some pending changes such as removing the old MLContext, > > >> removing > > >>> the file-based transform, isolating the matrix block library, and > some > > >>> language changes that should only be addresses in a major release as > > they > > >>> break backwards compatibility. Right now, we can't touch these > changes > > >>> without knowing the target release. > > >>> > > >>> Personally, I don't see a good reason why we should wait. Postponing > > this > > >>> major release just creates unnecessary overhead in maintaining these > > old > > >>> components that will be removed eventually. Since we cut RC for 0.13 > on > > >> Feb > > >>> 20, I think having an RC around April 20 would be a good target for > > this > > >>> 1.0 release. > > >>> > > >>> > > >>> Regards, > > >>> Matthias > > >>> > > >>> > > >>> On Fri, Mar 3, 2017 at 5:44 PM, Arvind Surve > <ac...@yahoo.com.invalid> > > >>> wrote: > > >>> > > >>>> Based on last couple of release cycles, we will continue with 2 > months > > >>>> release cycles.We will do first RC build by end of first week of > > second > > >>>> month. > > >>>> We will plan on releasing next release by end of April 2017.We will > > >> have > > >>>> RC build on ~April 6th. -Arvind > > >>>> Arvind Surve | Spark Technology Center | http://www.spark.tc/ > > >>>> > > >>>> From: Acs S <ac...@yahoo.com.INVALID> > > >>>> To: "dev@systemml.incubator.apache.org" <dev@systemml.incubator. > > >>>> apache.org> > > >>>> Sent: Monday, January 9, 2017 11:41 AM > > >>>> Subject: Re: Release cadence > > >>>> > > >>>> We need to release SystemML on more frequent basis to get community > > >>>> engaged. It will provide us more feedback on functionality we > > add.While > > >>>> releasing SystemML on monthly basis is challenge due to longer > phase > > of > > >>>> validation process we need to find a way to be quicker. > > >>>> I can propose options to get closer to monthly release if > acceptable. > > >>>> Make every two releases available on monthly basis and third on two > > >>> months > > >>>> basis. This cycle will continue. > > >>>> 1. Do minimal testing on two releases (minor releases) and release > > them > > >>> on > > >>>> monthly basis. Performance testing is one of the major time > consuming > > >>>> activity especially for larger data size. We can limit testing only > > >> upto > > >>>> 80GB. We can do code freeze (other than fixes) at the end of third > > week > > >>> and > > >>>> do verification on last week. If we find any issues we can still > > >> release > > >>>> the code with limitation documented unless issue breaks major > > >>>> functionality.2. Do comprehensive testing on third release. This > > will > > >>>> include performance testing for all data size and every other > testing > > >> we > > >>>> do. We can do code freeze (other than fixes) at the end of third > week > > >> and > > >>>> start verification of code. All issues found will be addressed. > > >> Exception > > >>>> will be considered. > > >>>> > > >>>> Meantime we need to start automating testing pieces. > > >>>> > > >>>> Arvind SurveSpark Technology Centerhttp://www.spark.tc/ > > >>>> > > >>>> From: Berthold Reinwald <reinw...@us.ibm.com> > > >>>> To: dev@systemml.incubator.apache.org > > >>>> Sent: Saturday, January 7, 2017 1:35 AM > > >>>> Subject: Re: Release cadence > > >>>> > > >>>> I think that a 2 month cycle would be a good compromise for > > major/minor > > >>>> releases. Fixpack release could be at a 1 month cycle. > > >>>> > > >>>> > > >>>> Regards, > > >>>> Berthold Reinwald > > >>>> IBM Almaden Research Center > > >>>> office: (408) 927 2208; T/L: 457 2208 > > >>>> e-mail: reinw...@us.ibm.com > > >>>> > > >>>> > > >>>> > > >>>> From: Deron Eriksson <deroneriks...@gmail.com> > > >>>> To: dev@systemml.incubator.apache.org > > >>>> Date: 01/05/2017 02:14 PM > > >>>> Subject: Re: Release cadence > > >>>> > > >>>> > > >>>> > > >>>> +1 for trying out a 1 month release cycle. > > >>>> > > >>>> However, I highly agree with Matthias that there is a lot of > overhead > > >>> with > > >>>> releases, so it would be good if we can work to streamline/automate > > the > > >>>> process as much as possible. Also, it would be good to distribute > the > > >>>> tasks > > >>>> around as much as possible. This can result in cross-training and > help > > >>>> avoid overburdening the same contributors each month. > > >>>> > > >>>> If the overhead slows us down too much, then we can go to a slower > > >>> release > > >>>> cycle. > > >>>> > > >>>> Deron > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>> On Thu, Jan 5, 2017 at 1:50 PM, <dusenberr...@gmail.com> wrote: > > >>>>> > > >>>>> +1 for adopting a 1 month release cycle. > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Mike Dusenberry > > >>>>> GitHub: github.com/dusenberrymw > > >>>>> LinkedIn: linkedin.com/in/mikedusenberry > > >>>>> > > >>>>> Sent from my iPhone. > > >>>>> > > >>>>> > > >>>>>> On Jan 5, 2017, at 1:35 PM, Luciano Resende > <luckbr1...@gmail.com> > > >>>>> wrote: > > >>>>>> > > >>>>>> On Thu, Jan 5, 2017 at 6:05 AM, Matthias Boehm > > >>>> <mboe...@googlemail.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> In general, I like the idea of aiming for consistent release > > >> cycles. > > >>>>>>> However, every month is just too much, at least for me. There is > a > > >>>>>>> considerable overhead associated with each release for > end-to-end > > >>>>>>> performance tests, tests on different environments, code freeze > > >> for > > >>>> new > > >>>>>>> features, etc. Hence, a too short release cycle would not be > > >> "agile" > > >>>> but > > >>>>>>> would actually slow us down. From my perspective, a realistic > > >>> release > > >>>>>>> cadence would be 2-3 months, maybe a bit more for major > releases. > > >>>>>>> > > >>>>>>> > > >>>>>> 2-3 months of release cadence for an open source is probably a > long > > >>>>>> stretch, particular for a project that does not have very large > set > > >>> of > > >>>>> 3rd > > >>>>>> party dependencies. > > >>>>>> > > >>>>>> As for some of the overhead issues you mentioned, they are > probably > > >>>> easy > > >>>>> to > > >>>>>> workaround: > > >>>>>> > > >>>>>> - code-freeze timeframe can be resolved with branches > > >>>>>> - end-to-end performance regressions can be avoided by better > code > > >>>>> review, > > >>>>>> and if you were willing to go with 2-3 months without performing > > >>> these > > >>>>>> tests, we could perform them only for major releases, and > > >> proactively > > >>>>>> quickly build a minor release with the patch when a user report > any > > >>>>>> performance regression. > > >>>>>> > > >>>>>> > > >>>>>> Anyway, I would really like to see SystemML more agile with > regards > > >>> to > > >>>>> its > > >>>>>> release process because, as I mentioned before, the release > early, > > >>>>> release > > >>>>>> often mantra is good to increase community interest, generate > more > > >>>>> traffic > > >>>>>> to the list as developers discuss the roadmap and release > blockers, > > >>>> and > > >>>>>> also enable users to provide feedback sooner on the areas we are > > >>>>> developing. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Luciano Resende > > >>>>>> http://twitter.com/lresende1975 > > >>>>>> http://lresende.blogspot.com/ > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Deron Eriksson > > >>>> Spark Technology Center > > >>>> http://www.spark.tc/ > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> Deron Eriksson > > >> Spark Technology Center > > >> http://www.spark.tc/ > > >> > > > > > > > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > > > > > >
