Wait. Ack. Do we want everyone to do this? This sounds like fragmentation. :(
Russell Jurney twitter.com/rjurney On Dec 10, 2012, at 3:24 PM, Olga Natkovich <onatkov...@yahoo.com> wrote: > If everybody is using a private branch then > > (1) We are not serving a significant part of our community > (2) There is no motivation to contribute those patches to branches (only to > trunk). > > Yahoo has been trying hard to work of the Apache branches but if we increase > the scope of what is going into branches, we will go with private branch > approach as well. > > Olga > > > ________________________________ > From: Julien Le Dem <jul...@twitter.com> > To: Olga Natkovich <onatkov...@yahoo.com> > Cc: "dev@pig.apache.org" <dev@pig.apache.org>; Santhosh M S > <santhosh_mut...@yahoo.com>; "billgra...@gmail.com" <billgra...@gmail.com> > Sent: Friday, December 7, 2012 3:54 PM > Subject: Re: Our release process > > Here's my criteria for inclusion in a release branch: > - no new feature. Only bug fixes. > - The criteria is more about stability than priority. The person/group > asking for it has a good reason for wanting it in the branch. If commiters > think the patch is reasonable and won't make the branch unstable then we > should check it in. If it breaks something anyway, we revert it. > > For what it's worth we (at Twitter) maintain an internal branch where we > add patches we need and I would suggest anybody that wants to be able to > make emergency fixes to their own deployment to do the same. We do keep > that branch as close to apache as we can but it has a few patches that are > in trunk only and do not satisfy the no new feature criteria. > > What does the PMC think ? > > Julien > > > > > On Tue, Dec 4, 2012 at 12:46 PM, Olga Natkovich <onatkov...@yahoo.com>wrote: > >> I am ok with tests running nightly and reverting patches that cause >> failures. We used to have that. Does anybody know what happened? Is anybody >> volunteering to make it work again? >> >> I would like to see specific criteria for what goes into the branch been >> published (rather than case-by-case). This way each team can decided if the >> criteria stringent enough of if they need to run a private branch. >> >> Olga >> >> ------------------------------ >> *From:* Santhosh M S <santhosh_mut...@yahoo.com> >> *To:* Julien Le Dem <jul...@twitter.com>; "dev@pig.apache.org" < >> dev@pig.apache.org> >> *Cc:* "billgra...@gmail.com" <billgra...@gmail.com> >> *Sent:* Friday, November 30, 2012 11:46 PM >> >> *Subject:* Re: Our release process >> >> HI Julien, >> >> You are making most of the points that I did on this thread (CI for e2e, >> not burdening clean e2e prior to every commit for a release branch). The >> only point on which there is no clear agreement is the definition of a bug >> that can be included in a previously released branch. I am fine with a case >> by case inclusion. >> >> Hi Olga, >> >> Are you fine with Julien's proposal as it stands - bugs that are included >> will be determined at the time of inclusion instead of doing it now. >> >> Santhosh >> >> >> ________________________________ >> From: Julien Le Dem <jul...@twitter.com> >> To: dev@pig.apache.org; Santhosh M S <santhosh_mut...@yahoo.com> >> Cc: "billgra...@gmail.com" <billgra...@gmail.com> >> Sent: Friday, November 30, 2012 5:37 PM >> Subject: Re: Our release process >> >> Proposed criteria: >> - it makes the tests fail. targets test-commit + test + e2e tests >> - a critical bug is reported in a short time frame (definition of >> critical not needed as it is rare and can be decided on a case by case >> basis) >> >> That raises another question: what are the existing CI servers running >> the tests? >> - the Apache CI runs test-commit and test (is it more stable now?) >> and not e2e. It would be great if it did. >> - we have a Jenkins build at Twitter where we run test-commit and >> test, we could not run e2e easily in our environment. >> - I understand there's a Yahoo/Hortonworks build (test-commit + test + e2e >> ???) >> >> Whenever those builds fail we should open or reopen JIRAS and fix it. >> >> The time it takes to run the full >> test suite makes it impractical to >> run on a desktop/laptop. >> >> For the release Pig-0.11.0 we need to get this list of JIRAs down to 0 >> and publish the jar. >> >> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+PIG+AND+fixVersion+%3D+%220.11%22+AND+resolution+%3D+Unresolved+ORDER+BY+updated+DESC%2C+due+ASC%2C+priority+DESC >> >> Julien >> >> On Thu, Nov 29, 2012 at 11:16 PM, Santhosh M S >> <santhosh_mut...@yahoo.com> wrote: >>> Looks like everyone is interested in having frequent releases - I don't >> see anyone disagreeing with that. >>> >>> Regarding "If a patch >> makes the release branch unstable, we revert it" - what are the criteria? >> If we can't decide on the criteria on this thread (already pretty long) >> then lets get the release trains going. We can revisit the criteria for >> inclusion of bug fixes when that happens. >>> >>> Santhosh >>> >>> >>> ________________________________ >>> From: Julien Le Dem <jul...@twitter.com> >>> To: dev@pig.apache.org; Santhosh M S <santhosh_mut...@yahoo.com> >>> Cc: "billgra...@gmail.com" <billgra...@gmail.com> >>> Sent: >> Thursday, November 29, 2012 9:45 AM >>> Subject: Re: Our release process >>> >>> The release branch receives only bug fixes. Patch level releases (3rd >>> version number) are issued out of the release branch and introduce >>> only bug fixes and no new features. >>> Deciding whether a patch is applied to the release branch is based on >>> preserving stability (as Bill said). If a patch makes the release >>> branch unstable, we revert it. >>> New features are added to trunk where new major and minor releases will >> happen. >>> If we need a new feature out then we make a new minor release. >>> Doing frequent releases is the industry standard and will resolve >>> conflicts around what should go in a release branch. >>> >>> Making a new release is currently painful *because* we wait so long in >>> between two releases. Let's fix that. >>> >>> Julien >>> >>> On Wed, Nov 28, 2012 at >> 10:09 PM, Santhosh M S >>> <santhosh_mut...@yahoo.com> wrote: >>>> Since releasing a major version once a month is agressive and we have >> not released on a quarterly basis, we should allow commits to a released >> branch to facilitate dot releases. >>>> >>>> If we are allowing commits to a released branch, the criteria for >> inclusion can be created anew or we use the industry standards for severity >> (or priority). It could be painful for a few folks but I don't see better >> alternatives. >>>> >>>> Regarding reverting commits based on e2e tests breaking: >>>> 1. Who is running the tests? >>>> 2. How often are they run? >>>> If we have nightly e2e runs then its easier to catch these errors >> early. If not the barrier for inclusion is pretty high and time >> consuming making it harder to develop. >>>> >>>> Santhosh >>>> >>>> >>>> ________________________________ >>>> From: Bill Graham <billgra...@gmail.com> >>>> To: dev@pig.apache.org >>>> Sent: Wednesday, November 28, 2012 11:39 AM >>>> Subject: Re: Our release process >>>> >>>> I agree releasing often is ideal, but releasing major versions once a >> month >>>> would be a bit agressive. >>>> >>>> +1 to Olga's initial definition of how Yahoo! determines what goes into >> a >>>> released branch. Basically is something broken without a workaround or >> is >>>> there potential silent data loss. Trying to get a more granular >> definition >>>> than that (i.e. P1, P2, severity, etc) will be >> painful. The reality in that >>>> case is that for whomever is blocked by the bug will consider it a P1. >>>> >>>> Fixes need to be relatively low-risk though to keep stability, but this >> is >>>> also subjective. For this I'm in favor of relying on developer and >> reviewer >>>> judgement to make that call and I'm +1 to Alan's proposal of rolling >> back >>>> patches that break the e2e tests or anything else. >>>> >>>> I think our policy should avoid time-based consideration on how many >>>> quarters away are we from the next major release since that's also >>>> impossible to quantify. Plus, if the answer to the question is that >> we're >>>> more than 1-2 quarters from the next release is "yes" then we should be >>>> fixing that release problem. >>>> >>>> >>>> On Wed, Nov 28, 2012 at 10:22 AM, Julien Le Dem <jul...@twitter.com> >> wrote: >>>> >>>>> I would really like to see us doing frequent releases (at least once >>>>> per quarter if not once a month). >>>>> I think the whole notion of priority or being a "blocker" is >> subjective. >>>>> Releasing infrequently pressures us to push more changes than we would >>>>> want to the release branch. >>>>> We should focus on keeping TRUNK stable as well so that it is easier >>>>> to release and users can do more frequent and smaller upgrades. >>>>> >>>>> There should be a small enough number of patches going in the release >>>>> branch so that we can get agreement on whether we check them in or >>>>> not. >>>>> I like Alan's proposal of reverting quickly when there's a problem. >>>>> Again, this becomes less of a problem if we release more >> often. >>>>> >>>>> Which leads me to my next question: what are the next steps for >>>>> releasing pig 0.11 ? >>>>> >>>>> Julien >>>>> >>>>> On Tue, Nov 27, 2012 at 10:22 PM, Santhosh M S >>>>> <santhosh_mut...@yahoo.com> wrote: >>>>>> Hi Olga, >>>>>> >>>>>> For a moment, I will move away from P1 and P2 which are related to >>>>> priorities and use the Severity definitions. >>>>>> >>>>>> The standard bugzilla definitions for severity are: >>>>>> >>>>>> Blocker - Blocks development and/or testing work. >>>>>> Critical - Crashes, loss of data, severe memory leak. >>>>>> Major - Major loss of function. >>>>>> >>>>>> I am >> skipping the other levels (normal, minor and trivial) for this >>>>> discussion. >>>>>> >>>>>> Coming back to priorities, the proposed definitions map P1 to Blocker >>>>> and Critical. I am proposing mapping P2 to Major even when there are >> known >>>>> workarounds. We are doing this since JIRA does not have severity by >> default >>>>> (see: >> https://confluence.atlassian.com/pages/viewpage.action?pageId=192840 >>>>> ) >>>>>> >>>>>> I am proposing that P2s be included in the released branch only when >>>>> trunk or unreleased versions are known to be backward incompatible or >> if >>>>> the release is more than a quarter (or two) away. >>>>>> >>>>>> Thanks, >>>>>> Santhosh >>> >>>>>> ________________________________ >>>>>> From: Olga Natkovich <onatkov...@yahoo.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org>; Santhosh M S < >>>>> santhosh_mut...@yahoo.com> >>>>>> Sent: Tuesday, November 27, 2012 10:41 AM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> Hi Santhosh, >>>>>> >>>>>> What is your definition of P2s? >>>>>> >>>>>> Olga >>>>>> >>>>>> >>>>>> ----- Original >> Message ----- >>>>>> From: Santhosh M S <santhosh_mut...@yahoo.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org>; Olga Natkovich < >>>>> onatkov...@yahoo.com> >>>>>> Cc: >>>>>> Sent: Monday, November 26, 2012 11:49 PM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> Hi Olga, >>>>>> >>>>>> I agree that we cannot guarantee backward compatibility upfront. With >>>>> that knowledge, I am proposing a small modification to your proposal. >>> >>>>>> 1. If the trunk or unreleased version is known to be backwards >>>>> compatible then only P1 issues go into the released branch. >>>>>> 2. If the the trunk or unreleased version is known to be backwards >>>>> incompatible or the release is a long ways off (two quarters?) then we >>>>> should allow for dot releases on the branch, i.e., P1 and P2 issues. >>>>>> >>>>>> I am hoping that should provide an incentive for users to move to a >>>>> higher release and at the same time allow developers to fix issues of >>>>> significance without impacting stability. >>>>>> >>>>>> Thanks, >>>>>> Santhosh >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Olga Natkovich <onatkov...@yahoo.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org> >>>>>> Sent: Monday, November 26, 2012 9:38 AM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> Hi Santhosh, >>>>>> >>>>>> I understand the compatibility issue though I am not sure we can >>>>> guarantee it for all releases upfront but agree that we should make an >>>>> effort. >>>>>> >>>>>> On the e2e tests, part of the proposal is only do make P1 type of >>>>> changes to the branch after the initial release so they should be rare. >>>>>> >>>>>> Olga >>> >>>>>> ________________________________ >>>>>> From: Santhosh M S <santhosh_mut...@yahoo.com> >>>>>> To: Olga Natkovich <onatkov...@yahoo.com>; "dev@pig.apache.org" < >>>>> dev@pig.apache.org> >>>>>> Sent: Monday, November 26, 2012 12:00 AM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> >>>>>> It takes too long to run. If the e2e tests are run every night or a >>>>> reasonable timeframe then it will reduce the barrier for submitting >>>>> patches. The context for this: >> the reluctance of folks to move to a higher >>>>> version when the higher version is not backward compatible. >>>>>> >>>>>> Santhosh >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Olga Natkovich <onatkov...@yahoo.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org>; Santhosh M S < >>>>> santhosh_mut...@yahoo.com> >>>>>> Sent: Sunday, November 25, 2012 5:56 PM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> Hi >> Santhosh, >>>>>> >>>>>> Can you clarify why running e2e tests on every checking is a problem? >>>>>> >>>>>> Olga >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Santhosh M S <santhosh_mut...@yahoo.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org> >>>>>> Sent: Monday, November 19, 2012 3:48 PM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> The push for an upgrade will work only if the higher release is >> backward >>>>> compatible with the lower release. If not, folks will tend to use >> private >>>>> branches. Having a stable branch on a large deployment is a good >> indicator >>>>> of stability. However, please note that there have been instances where >>>>> some releases were never adopted. I will be extremely careful in >> applying >>>>> the rule of >>>>>> running e2e tests for every commit to a released branch. >>>>>> >>>>>> If we release every quarter (hopefully) and preserve backward >>>>> compatibility then I am +1 to the proposal. If the backward >> compatibility >>>>> is not preserved then I am -1 for having to run e2e for every commit >> to a >>>>> released branch. >>>>>> >>>>>> Santhosh >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Jonathan Coveney <jcove...@gmail.com> >>>>>> To: "dev@pig.apache.org" <dev@pig.apache.org> >>>>>> Sent: Tuesday, November 6, 2012 6:34 PM >>>>>> Subject: Re: Our release process >>>>>> >>>>>> I think it might be good to clarify (for me) a couple of cases: >>>>>> >>>>>> 1. we have branched a new release >>>>>> 2. an existing release >>>>>> >>>>>> The way I understand things, in the case of 1, we have >>>>>> a backlog of patches >>>>>> (not all of which are P1 bugs), and that's ok. If a new bad bug >> comes in >>>>>> (the subject of debate here), then it goes in anyway (and in some >> cases, >>>>>> would go into 0.9 etc). >>>>>> >>>>>> Olga is saying that for existing release (0.9, 0.10), we should only >>>>> commit >>>>>> P1 bug fixes there. This makes sense to me, as we're fixing the >> "official >>>>>> release" in place. >>>>>> >>>>>> IMHO, this would encourage people to use newer release (as this is >> where >>>>>> the latest and greatest stuff is, including non-critical bug fixes). >>>>> Olga's >>>>>> criteria is a pretty clear barrier for inclusion into these releases. >>>>> With >>>>>> old releases, I think the key is really that they keep doing what >> they >>>>> have >>>>>> always done. Most bugs are well understood by now, and the ones that >>>>> aren't >>>>>> will no doubt be P1. >>> I'm not decided (thus no formal +1 or whatnot), but Olga's point seems >>>>>> pretty reasonable to me, especially given that trunk has pretty >>>>>> liberal >>>>>> development. Once it gets tidied up, I can understand not wanting to >>>>> jostle >>>>>> it. >>>>>> >>>>>> >>>>>> 2012/11/5 Alan Gates <ga...@hortonworks.com> >>>>>> >>>>>>> Jonathan, for clarity, are you saying you agree that we should only >> put >>>>>>> bug fixes in branches or we should only put high priority bug fixes >> in >>>>>>> branches? I think we all agree on the former, but there appear to >> be >>>>>>> different views on the latter. >>>>>>> >>>>>>> Alan. >>>> >>>>>>> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote: >>>>>>> >>>>>>>> This seems to make sense to me. People can always back-port >> features, >>>>> and >>>>>>>> this encourages them to use the newer ones. It also means we will >> be >>>>> more >>>>>>>> rigorous about stability, which is good as it is a big plus for >> Pig. I >>>>>>>> think for older branches, stability trumps features in a big way. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> 2012/11/5 Gianmarco De Francisci Morales <g...@apache.org> >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On Mon, Nov 5, 2012 at 10:48 AM, Olga >> Natkovich < >>>>> onatkov...@yahoo.com> >>>>>>>>> wrote: >>>>>>>>>> Hi Gianmarco, >>>>>>>>>> >>>>>>>>>> Thanks for your comments. Here is a little more information. >>>>>>>>>> >>>>>>>>>> At Yahoo, we consider the following issues to be P1: >>>>>>>>>> >>>>>>>>>> (1) Bugs that cause wrong results being produced silently >>>>>>>>>> (2) Bugs that cause failures with no easy workaround >>>>>>>>> >>>>>>>>> Thanks Olga, now I get what you mean. >>>>>>>>> I don't have a strong opinion on >> this. >>>>>>>>> On one hand I see why you don't want to put too many patches in >> the >>>>>>>>> branches in order to keep things stable. >>>>>>>>> On the other hand when we do a 0.10.x release with x>0 the users >>>>> would >>>>>>>>> like to have as many bugs fixed as possible. >>>>>>>>> >>>>>>>>>> Regarding tests. I would suggest we have different rules for >> trunk >>>>> and >>>>>>>>> branches: >>>>>>>>>> >>>>>>>>>> (1) For branches, I think we should run the full regression >> suite >>>>>>>>> (including e2e) prior to commit. This way we can ensure branch >>>>> stability >>>>>>>>> and, as number of patches should be small, will not be a burden >>>>> >>>>>>> (2) For trunk, we can go with test-commit only and fix things >>>>> quickly >>>>>>>>> when things break. >>>>>>>>> >>>>>>>>> I think this makes sense. +1 >>>>>>>>> >>>>>>>>>> Olga >>>>>>> >>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> -- >>>>>>>>> Gianmarco >>>> >>>> >>>> >>>> -- >>>> *Note that I'm no longer using my Yahoo! email address. Please email me >> at >>>> billgra...@gmail.com going forward.* >>
