Thanks Brennan. I agree that cherry picking can be scary, but it doesn’t have to be. I hear you on the risks but I think we need to start somewhere. I feel like this won’t be as in practice as it may seem now and we may be dealing with some FUD. I could also be absolutely wrong and pushing a totally stupid idea :). We won’t know until we try.
Not to muddy the water but I think it’s worth noting that if we feel like master has what we want when we are ready to cut a release, we can always rebase it into the release branch. I think that’s reasonable and will basically allow us to build our release from master. On Wed, Oct 30, 2019 at 16:41 ocket 8888 <[email protected]> wrote: > Cherry picking is scary. It will always have some associated risk. I > understand that the point is to emulate pulling commits out rather than > picking ones in, so that incomplete features and risky changes can be left > out of a release that otherwise ought to be all-encompassing. However, that > means essentially asking people to develop against the commit at which the > last release branch was cut rather than master, because all changes need to > "stand on their own," since without being a part of the review process for > every PR that comes in no single person can be totally sure when they start > work how much of the work between then and now can be considered "stable" > enough to make the cut - even if that is a low bar. Which isn't inherently > bad, it's just that it can lead to duplication of effort, and it's just not > something we normally do as a community. Currently, nearly all development > and testing outside of internal patching is taking place on master, so for > what it's worth we already are somewhat concerned with it being stable all > the time (please ignore failing integration tests on master at the time of > this writing). > > So that's my opinion on cherry-picking. I do, though, think this represents > a step in the right direction, even with cherry-picking. More frequent > releases with better testing will lead to general improvements to the > automated testing infrastructure, the ability to kill off legacy code in > our lifetimes, and make the stability of new releases more > certain.Time-based release cycles get a big +1 from me, and if I have to > live with cherry-picking to get that then so be it. > > On Wed, Oct 30, 2019 at 3:47 PM Dave Neuman <[email protected]> wrote: > > > There are many things that are not working. First of all we do not have > > any semblance of a consistent release pattern. If I get my feature > merged > > into Traffic Control today, I have no idea when that'll actually be in a > > release. Secondly, we do not currently have any way to limit what goes > > into a release besides "everything in master since the last release". > This > > leads to a lot of uncertainty with new releases and lower confidence when > > deploying a new release. This also forces us to not be able to move fast > > when merging things into master since we are always trying to keep master > > "stable". Finally, since our releases are so big and infrequent it > takes > > a long time for the majority of the community to upgrade, which then > > requires us to support old releases for a longer period of time. Our > > current available release is 3.0 which means there are still folks that > are > > on 2.2 and we should technically support it since it's n-1. For > reference, > > 2.2 was released in June of 2018. > > I agree that not having clear objectives and definitions of good enough > > lead to problems. I don't think it's realistic to plan all of the > features > > that we want to put into each release, it's an open source project and > > that's not how it works. This is why I am suggesting a consistent time > > based release cycle. This will allow us to all have the same > understanding > > of when a release is going to be and when our features will be included > in > > a release. This also allows us to have smaller releases that are more > > easily digestible which allows us to hopefully move faster as a > community. > > > > BTW, 6 weeks was just a time period that felt right. Just like with the > > process if that doesn't work (too fast, too slow, etc) we can adjust that > > as well. > > > > On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan < > [email protected]> > > wrote: > > > > > Did we define why/how what we've been doing isnt' working? I see a > > > consensus that it's not, suggestions on alternatives, and an ambition > for > > > faster and more stable; but not a definition of the thing is the reason > > why > > > we must change? Not saying we shouldn't, but I'm just asking so we > try > > > not to repeat the same mistakes following new steps. Usually I've > found > > > that release blockage is derived from not defining clear objectives and > > > definitions of good enough (not necessarily perfect) consistently > > > throughout a release cycle. It's not enough to start a release cycle > > with > > > one scope of both, yet at the end continue to revise that definition. > > > > > > Jonathan G > > > > > > > > > On 10/30/19, 2:36 PM, "Dave Neuman" <[email protected]> wrote: > > > > > > Sorry, I have taken too long to respond. > > > > > > +1000000 on better CI and Testing, I think we all agree that needs > to > > > get > > > better and is a different beast than this topic. > > > > > > As for the release branches, I am trying to follow a process > similar > > to > > > ATS, which is usually the model we try to follow with our project. > > > While I agree that branching from master is the ideal solution, we > > > tried > > > that and it didn't work. I feel like we are trying to agree to do > > the > > > same > > > thing that we already tried and didn't work. I think we should > have > > a > > > little freedom to commit things into master and, if they break > > things, > > > work > > > on getting them fixed. We also might commit PRs to master that are > > > partial > > > implementations of a bigger feature. If we are using a time based > > > strategy to cut our releases from master then we are going to be > > > continually worrying about making sure Master stays not broken. I > > > think we > > > should try to keep master deployable, but it's fine if it's not for > > > short > > > periods of time. > > > > > > I am not trying to suggest that we are "picky" about what goes into > > our > > > release branch. I think we should try to cherry-pick as much as we > > can > > > into our release branch once it is ready to be cherry-picked. I > > think > > > we > > > should be able to take the tip of our release branch and deploy it > > > whenever > > > we want with some confidence since we know that features/bug-fixes > > that > > > have been cherry-picked have already been reviewed and tested > against > > > master. Yes, its more work, but it also give us the ability to > have > > > more > > > confidence in what is in our release at any given time. Again, we > > > should > > > be trying to cherry-pick in everything that makes sense to > > cherry-pick. > > > > > > I think the process that I would like to see looks like this: > > > - Cut a 4.x branch from Master > > > - Once we think we are ready for a RC cut a 4.0.x branch from the > 4.x > > > branch > > > - Tag the 4.0.0 release from the 4.0.x branch > > > - New non-breaking changes and bug fixes that go into master are > also > > > Cherry Picked to 4.x > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag > when > > > we > > > are ready for a RC. > > > - Rinse and repeat for all 4.x releases > > > - Once there's a need for a major release we make a 5.x branch from > > > master > > > > > > If we find a bug in 4.0.0 then we commit the fix to master, then > 4.x, > > > then > > > 4.0.x which we use to create the new 4.0.1 release (assuming we > don't > > > have > > > 4.1 yet). Yes, that is a more work than today, but I think A) it's > > not > > > really that bad and B) it gives us more control. > > > > > > This should allow us to deploy much quicker and with more > confidence > > > than > > > is possible today. We can also choose to deploy from any point in > > > time > > > from the 4.x branch with relatively high confidence. > > > > > > I'll wrap this up by saying that I don't think there's a perfect > > > solution, > > > but I think we can agree on something that is worth trying at least > > > with > > > the 4.x release life cycle. Once we have some experience, we can > > > figure > > > out where it is not meeting our needs and figure out how to adjust > as > > > necessary. I think we can all agree our current process is not > > > working. > > > > > > Thanks, > > > Dave > > > > > > > > > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> > > > wrote: > > > > > > > +1 on more frequent releases > > > > +1 on improving our CI. We need to get our CI to the point where > > > we're > > > > confident deploying changes to production if the CI passes. > > > > +1 on "trunk-based" development as opposed to GitFlow > > > > -1 on cherry-picking new features. This is risky, tedious, and > > > > represents unnecessary effort that would be better spent > elsewhere, > > > > like improving our CI or adding more automated tests. > > > > +1 on cutting minor release branches directly off master > > > > > > > > I think cutting minor release branches directly off master can > put > > us > > > > in a win-win scenario. Our releases become smaller and more > > frequent, > > > > and we avoid the unnecessary risk and effort of cherry-picking > new > > > > features onto stale release branches. > > > > > > > > This is the process that I would propose: > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > > > 2. Develop master for 4-6 weeks. > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > > > 5. Whenever a major (show-stopping) bug or security issue is > found, > > > > cherry-pick the fix from master into the latest release branch > > (e.g. > > > > 4.1.x), and create a 4.1.1 RC. > > > > > > > > As the 4.x release manager, I really don't want to get into the > > > > business of selecting what should and should not go into a > > particular > > > > release. Every 4-6 weeks, we should just cut a new minor release > > > > branch off master and take everything that was contributed during > > > that > > > > time, as long as everything builds fine and all the tests pass. > The > > > > master branch, while it doesn't always have to be perfect, should > > at > > > > least always be deployable. If a new feature is merged with bugs > > and > > > > becomes a show-stopper, it should be either immediately remedied > or > > > > reverted on master while we work on a fix. > > > > > > > > - Rawlin > > > > > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > > > <[email protected]> wrote: > > > > > > > > > > In my opinion, in order to get to a cadence we are discussing > we > > > need to > > > > put a lot more work into the CI system. It has been failing > > > consistently, > > > > doesn’t block PR approvals/merges and when it actually runs it > does > > > not > > > > test anything besides build and license headers. > > > > > > > > > > In the past couple of weeks we have had (off the top of my > head): > > > > > > > > > > * Unit tests broken > > > > > * TO API tests broken > > > > > * Golang vendor issues > > > > > * TO Go build issues > > > > > > > > > > All were merged and broke master. > > > > > > > > > > Yes, having release branches sounds great and more aggressive > > > cadence > > > > will minimize the amount of risk but I think that comes with the > > > need to > > > > improve on our automated validation and testing. > > > > > > > > > > We have proven that we have not kept master in a good state and > > > adding > > > > more releases (more branches) will make it even harder to keep > that > > > > stability we are already fighting. > > > > > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected] > > > > > wrote: > > > > > > > > > > Yeah, I get it. No one company should be driving release > > > > schedules/scope. > > > > > > > > > > What I was really getting at is if ANY company has a recent > > > version > > > > in a > > > > > test or prod-like enviro (meaning the version is being > > > exercised and > > > > > thoroughly tested), then maybe we consider simply cutting a > > > release > > > > from > > > > > that version. For example, imagine company X had this > version > > > in a > > > > > test/prod enviro: > > > > > > > > > > Master-10287.7e62d07 ( > > > > > > > > > > > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > > > ) > > > > > > > > > > Then I would be in favor of cutting the 4.0 release from > > > > > Master-10287.7e62d07 as we know it is proven to work based > on > > > > company X's > > > > > actual use of it. > > > > > > > > > > The truth is (imo), we don't have the bandwidth to manually > > > verify > > > > releases > > > > > that we are not using/have not used nor do we have the > > > necessary > > > > automated > > > > > test coverage to verify these releases. So most of our > > > releases are > > > > > significantly untested. > > > > > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 < > > > [email protected]> > > > > wrote: > > > > > > > > > > > I'm really not a fan of allowing Comcast to dictate the > > > release > > > > scope and > > > > > > schedule. If cherry-picking is too messy, then we can > just > > > cut new > > > > minor > > > > > > releases directly from master. > > > > > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell < > > > [email protected]> > > > > wrote: > > > > > > > > > > > > > I don't think it's as easy as cherry picking > > (backporting) > > > > certain > > > > > > features > > > > > > > into a release branch. I could be wrong but I really > > don't > > > think > > > > it is. > > > > > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master > and > > > we go > > > > through > > > > > > with > > > > > > > our normal process of testing, validating, etc. At the > > same > > > > time, 4.1 > > > > > > > branch is created from 4.0. Master moves on as normal. > > > Let's say > > > > 25 > > > > > > commits > > > > > > > come in to master during the next 6 weeks. During that > 6 > > > weeks, > > > > we cherry > > > > > > > pick only the features that look interesting for 4.1? > > (the > > > rest > > > > just stay > > > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are > > roughly 6 > > > > weeks apart > > > > > > and > > > > > > > have only "interesting" features so the releases are > > > > incrementally very > > > > > > > small and much easier to test/validate, however, those > > > features > > > > might > > > > > > > depend on other commits. At some point, cherry picking > > > becomes > > > > impossible > > > > > > > as the foundation of the code base has shifted so > > > drastically. > > > > Cherry > > > > > > > picking will become very messy and introduce a high > level > > > of > > > > risk imo. > > > > > > > > > > > > > > Also, you'd have to manage the scope of these minor > > > releases > > > > which we are > > > > > > > not very good at because the designated release manager > > > has a > > > > full-time > > > > > > job > > > > > > > to attend to and now we have even more releases to > > > > > > > scope/test/validate/approve. I like the idea of faster > > > releases > > > > but this > > > > > > > sounds like more work that nobody has the bandwidth > for. > > > > > > > > > > > > > > Ok, so rather than shooting down an idea with no > > > alternative > > > > proposal, I > > > > > > > propose this. At Comcast, we are trying to get to the > > point > > > > where we > > > > > > > release TC to prod from the head of master every few > > > weeks. What > > > > if every > > > > > > > time Comcast does an internal upgrade, they note the > > > commit hash > > > > and once > > > > > > > comcast is happy with the upgrade in their prod enviro, > > > they > > > > propose a > > > > > > > release pinned to that commit hash? With this approach > > > you'd end > > > > up with: > > > > > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > > > 2. well tested releases as it's in a real prod enviro > > > > > > > 3. little/no validation work for anyone > > > > > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else > is > > > out > > > > front at the > > > > > > > edge of master and running it in a trusted enviro, they > > > could > > > > propose a > > > > > > > release as well. > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > I have been thinking about how we can get better with > > our > > > > release > > > > > > > cadence. > > > > > > > > I feel like we have slowed to a crawl and not been as > > > good as > > > > we should > > > > > > > > about how we release. Our last Major release was in > > > March and > > > > we > > > > > > haven't > > > > > > > > had a real release since. Moving forward I would > like > > > to see > > > > us get > > > > > > on > > > > > > > a > > > > > > > > more consistent cadence and provide smaller releases > > > that can > > > > be more > > > > > > > > easily digested by the community. It's my hope that > we > > > can > > > > get to a > > > > > > > > cadence where we are doing releases every 4 to 6 > weeks, > > > we are > > > > > > releasing > > > > > > > > in such a way that not all releases all required > (e.g. > > > if you > > > > are on > > > > > > 4.0 > > > > > > > > you don't need 4.1, you can just install 4.2), and we > > > are more > > > > > > deliberate > > > > > > > > about what we put into a release. > > > > > > > > > > > > > > > > I think we can accomplish this by using our release > > > branches > > > > better. > > > > > > For > > > > > > > > each major release we will cut a release branch from > > > master > > > > (e.g. 4.x) > > > > > > > and > > > > > > > > then we will use cherry-picking to add new features > and > > > bug > > > > fixes to > > > > > > that > > > > > > > > release. This means that if 4.0 has been released > and > > > you > > > > want to get > > > > > > > your > > > > > > > > feature/bug fix in 4.1, you will first submit your PR > > to > > > > master and > > > > > > then > > > > > > > > cheery pick your squash merged commit to the 4.x > branch > > > which > > > > we will > > > > > > use > > > > > > > > to create the 4.1 release. I think we should allow > > > either the > > > > > > > contributor > > > > > > > > or the committer (who is merging the PR) to suggest > if > > a > > > > feature goes > > > > > > > into > > > > > > > > the release branch, and if we disagree we can take it > > to > > > the > > > > list. If > > > > > > we > > > > > > > > decide that every PR to master also goes into 4.1 > then > > > so be > > > > it, but at > > > > > > > > least we are making a conscious decision of what goes > > > into the > > > > next > > > > > > > > release. This will allow us to not be so worried > > about > > > what > > > > we are > > > > > > > > merging into master and how that will affect our next > > > release. > > > > > > According > > > > > > > > to our 6 week cadence we will cut a new release from > > the > > > > current > > > > > > feature > > > > > > > > branch and put that up for a vote. These releases > will > > > be > > > > small and > > > > > > > > testable enough that we can get feedback quickly and > > > move from > > > > RC to > > > > > > > > release in a short time. If a release has too many > > > issues > > > > then we may > > > > > > > end > > > > > > > > up deciding to skip that minor release in favor of > the > > > next > > > > one, > > > > > > > hopefully > > > > > > > > this does not happen very often if at all. > > > > > > > > > > > > > > > > Once a breaking change is introduced to master which > > will > > > > require a new > > > > > > > > major release, we will create a new major release > > branch > > > from > > > > master > > > > > > > (which > > > > > > > > will mean a new release manager). We will then > repeat > > > the > > > > same process > > > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > > > > > As for LTS we will provide support for the latest > major > > > > release plus > > > > > > the > > > > > > > > one previous. So, once we release 4.0 we will > support > > > 4.0 and > > > > 3.1. > > > > > > Any > > > > > > > > security issues that arise will -- if present -- be > > > applied to > > > > each of > > > > > > > > these versions. Once 4.1 is released we will support > > > 4.1 and > > > > 3.1, etc, > > > > > > > > etc. > > > > > > > > > > > > > > > > Please let me know if you have any thoughts on this > > > plan. If > > > > there are > > > > > > > no > > > > > > > > major objections I would like to try this with 4.x > with > > > the > > > > idea that > > > > > > we > > > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > > > > > We need to get better at releasing and something > needs > > to > > > > change, > > > > > > > hopefully > > > > > > > > this can get us going in the right direction. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > > > cherry-picks to > > > > > > our > > > > > > > > release branch to control what goes into a minor > > release. > > > > Major > > > > > > releases > > > > > > > > will be cut from master as necessary - such as if > there > > > is a > > > > breaking > > > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
