We are in violent agreement and that was sort of the point I was making. I’m wholly in favor of release process that encourages development that tracks master as close as possible and discourages long lived branches. Features imho should be written in a way that is always seamlessly (as much as possible) integrated into the existing code base and using feature flags is one way to accomplish that without risking (too much) the stability of that code base.
Geoff On 10/31/19, 3:19 PM, "Rawlin Peters" <[email protected]> wrote: I agree, but this release process wouldn't lead to long-lived feature branches since all development would still be done on master and merged into master. Unstable or incomplete features could potentially delay releases from going out, but I agree that feature flags could mitigate those problems. Anything we can do to keep the master branch deployable is a +1 in my book. - Rawlin On Thu, Oct 31, 2019 at 2:46 PM Genz, Geoffrey <[email protected]> wrote: > > Just throwing out that the judicious use of feature flags could reduce the pain from unstable or incomplete new features. Long lived feature branches are something I believe should be avoided. > > Geoff > > On 10/31/19, 2:32 PM, "Rawlin Peters" <[email protected]> wrote: > > Alright, I think that sounds fair. In general what I'm hearing is, > every 4-6 weeks: > 1. fast-forward 4.x to the most recent "stable" commit of master > (maybe we automate this and fast-forward every day if some set of > tests pass, unless we explicitly know master is unstable for other > reasons) > 2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into > 4.*.x that are necessary for the release (e.g. bugs that break the > data plane or prevent operators from being able to do their job or run > the control plane) > > Which means the release manager needs to know/track: > - unstable commits along with their corresponding fixes > - incomplete features > - features we'd like to hold until the next release (due to current > release having enough already) > - any cherry-picks that were deemed necessary for a certain release > (so that they can either be carried forward into the next minor > release or, ideally, included as part of fast-forwarding 4.x to the > most recent stable commit on master) > > I think this will work. We gain the ability to: > 1. let master be "unstable" periodically > 2. control the size of releases > 3. make releases mirror master as closely as possible (because they > will just be a snapshot of the most recent version of master that was > considered stable -- unless absolutely necessary bug/security fixes > were cherry-picked in) > > Does that all sound right, Dave? > > - Rawlin > > > > > > > > On Thu, Oct 31, 2019 at 8:37 AM David Neuman <[email protected]> wrote: > > > > I think a third reason would be that we are getting very close to our > > release branch date, it's a large/complex feature, and we feel like the > > release already has enough in it. In that case we could hold off on adding > > that change to the 4.x branch until after we cut the 4.x.x branch. The > > benefit is that we can still merge that feature into master and get all of > > our testing (both internally and externally) running against it. If you > > stick with just branching off master and this situation arises then we get > > in the spot where we just don't merge the PR at all until after the release > > branch is cut (we have two examples of this right now). I personally think > > it's better to get this into master and let our automation run against it > > than it is to leave it out in a PR even though it is ready. > > > > As for the process, I was thinking the same thing except that you don't > > even need to cherry-pick. In your scenario above you can just rebase > > master into 4.x once E was merged. This avoids the headaches of having to > > cherry pick each commit while still giving us the benefit of having all of > > the changes on the release branch. We can even decide to setup automation > > to rebase master into the release branch on an interval or on demand. If > > we rebase in something that turns out to be broken we just roll back to the > > previous rebase (or a certain commit) until that is fixed and then rebase > > again. If a feature gets merged to master that we aren't ready to put into > > the release, then we stop the automated rebase until we are ready to get it > > into the release -- at that point we should consider ONLY cherry-picking > > bug fixes for the remainder of that release cycle to make things easier to > > rebase again. > > > > I am sure there will be unintended consequences, and I am sure we will have > > challenges to work through, but I still think that it's worth trying. We > > should fail fast and be willing to change course if necessary. > > I think if we can be consistent enough with our releases then maybe we can > > discuss branching from master again. > > > > Thanks, > > Dave > > > > > > > > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <[email protected]> wrote: > > > > > I guess my concern is really about what possible reasons we might have > > > to not cherry-pick a specific PR that had been merged into master > > > already. The only reasons I can think of to not cherry-pick something > > > would be: > > > 1. the PR was completely broken (e.g. breaks the build, tests don't > > > pass, doesn't compile, etc). > > > 2. the PR one one of a larger multi-PR "feature", and *we know* > > > releasing some parts early could cause problems (that knowing part is > > > crucial) > > > > > > We should always strive to catch issue #1 in our CI before it merges > > > into master. When things like that merge it can block people from > > > doing development starting at that point until it's fixed. > > > I can see why issue #2 might arise due to multiple different people > > > developing one larger feature, but we should still strive to break > > > larger features up into individual pieces that can reasonably stand on > > > their own if possible. Due to time constraints or other reasons I can > > > see that might be a little difficult. > > > > > > That said, I would still strive to make the 4.x release branch a > > > snapshot of master at some "known good point" in time. Let's say we > > > cut 4.x at commit A, then a broken commit B gets merged into master. > > > If it's completely obvious that it's a broken commit, I'd wait until > > > at least a commit C gets merged to fix it. Then I would cherry-pick B > > > and C into 4.x at the same time. If the fix is really like a commit E > > > with unrelated commits C and D in between, ideally I would still wait > > > to cherry-pick C and D until a "fix" commit E was merged (to fix > > > commit B), at which point I'd cherrypick commits B through E at the > > > same time, in order, to avoid any possible merge conflicts. > > > > > > I think to make that process easier for myself I would just > > > cherry-pick everything into the 4.x branch as it gets merged into > > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable > > > commits were not fixed/completed in time, I would reset the 4.x branch > > > to the last stable commit. This would mean that unstable > > > commits/incomplete features could potentially block things from going > > > into the release. I might count that as a positive because it would > > > incentivize us to improve our CI and do our best to also keep the > > > master branch as stable as possible. > > > > > > The end result would be the same as just continually jumping up the > > > master branch from known good, stable points in time to the next, > > > creating a minor release branch at each of those points. > > > > > > - Rawlin > > > > > > > > > > > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <[email protected]> wrote: > > > > > > > > Sorry, I have taken too long to respond. > > > > > > > > +1000000 on better CI and Testing, I think we all agree that needs to get > > > > better and is a different beast than this topic. > > > > > > > > As for the release branches, I am trying to follow a process similar to > > > > ATS, which is usually the model we try to follow with our project. > > > > While I agree that branching from master is the ideal solution, we tried > > > > that and it didn't work. I feel like we are trying to agree to do the > > > same > > > > thing that we already tried and didn't work. I think we should have a > > > > little freedom to commit things into master and, if they break things, > > > work > > > > on getting them fixed. We also might commit PRs to master that are > > > partial > > > > implementations of a bigger feature. If we are using a time based > > > > strategy to cut our releases from master then we are going to be > > > > continually worrying about making sure Master stays not broken. I think > > > we > > > > should try to keep master deployable, but it's fine if it's not for short > > > > periods of time. > > > > > > > > I am not trying to suggest that we are "picky" about what goes into our > > > > release branch. I think we should try to cherry-pick as much as we can > > > > into our release branch once it is ready to be cherry-picked. I think we > > > > should be able to take the tip of our release branch and deploy it > > > whenever > > > > we want with some confidence since we know that features/bug-fixes that > > > > have been cherry-picked have already been reviewed and tested against > > > > master. Yes, its more work, but it also give us the ability to have more > > > > confidence in what is in our release at any given time. Again, we should > > > > be trying to cherry-pick in everything that makes sense to cherry-pick. > > > > > > > > I think the process that I would like to see looks like this: > > > > - Cut a 4.x branch from Master > > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x > > > branch > > > > - Tag the 4.0.0 release from the 4.0.x branch > > > > - New non-breaking changes and bug fixes that go into master are also > > > > Cherry Picked to 4.x > > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we > > > > are ready for a RC. > > > > - Rinse and repeat for all 4.x releases > > > > - Once there's a need for a major release we make a 5.x branch from > > > master > > > > > > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, > > > then > > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't > > > have > > > > 4.1 yet). Yes, that is a more work than today, but I think A) it's not > > > > really that bad and B) it gives us more control. > > > > > > > > This should allow us to deploy much quicker and with more confidence than > > > > is possible today. We can also choose to deploy from any point in time > > > > from the 4.x branch with relatively high confidence. > > > > > > > > I'll wrap this up by saying that I don't think there's a perfect > > > solution, > > > > but I think we can agree on something that is worth trying at least with > > > > the 4.x release life cycle. Once we have some experience, we can figure > > > > out where it is not meeting our needs and figure out how to adjust as > > > > necessary. I think we can all agree our current process is not working. > > > > > > > > Thanks, > > > > Dave > > > > > > > > > > > > > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> > > > wrote: > > > > > > > > > +1 on more frequent releases > > > > > +1 on improving our CI. We need to get our CI to the point where we're > > > > > confident deploying changes to production if the CI passes. > > > > > +1 on "trunk-based" development as opposed to GitFlow > > > > > -1 on cherry-picking new features. This is risky, tedious, and > > > > > represents unnecessary effort that would be better spent elsewhere, > > > > > like improving our CI or adding more automated tests. > > > > > +1 on cutting minor release branches directly off master > > > > > > > > > > I think cutting minor release branches directly off master can put us > > > > > in a win-win scenario. Our releases become smaller and more frequent, > > > > > and we avoid the unnecessary risk and effort of cherry-picking new > > > > > features onto stale release branches. > > > > > > > > > > This is the process that I would propose: > > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > > > > 2. Develop master for 4-6 weeks. > > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > > > > 5. Whenever a major (show-stopping) bug or security issue is found, > > > > > cherry-pick the fix from master into the latest release branch (e.g. > > > > > 4.1.x), and create a 4.1.1 RC. > > > > > > > > > > As the 4.x release manager, I really don't want to get into the > > > > > business of selecting what should and should not go into a particular > > > > > release. Every 4-6 weeks, we should just cut a new minor release > > > > > branch off master and take everything that was contributed during that > > > > > time, as long as everything builds fine and all the tests pass. The > > > > > master branch, while it doesn't always have to be perfect, should at > > > > > least always be deployable. If a new feature is merged with bugs and > > > > > becomes a show-stopper, it should be either immediately remedied or > > > > > reverted on master while we work on a fix. > > > > > > > > > > - Rawlin > > > > > > > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > > > > <[email protected]> wrote: > > > > > > > > > > > > In my opinion, in order to get to a cadence we are discussing we > > > need to > > > > > put a lot more work into the CI system. It has been failing > > > consistently, > > > > > doesn’t block PR approvals/merges and when it actually runs it does not > > > > > test anything besides build and license headers. > > > > > > > > > > > > In the past couple of weeks we have had (off the top of my head): > > > > > > > > > > > > * Unit tests broken > > > > > > * TO API tests broken > > > > > > * Golang vendor issues > > > > > > * TO Go build issues > > > > > > > > > > > > All were merged and broke master. > > > > > > > > > > > > Yes, having release branches sounds great and more aggressive cadence > > > > > will minimize the amount of risk but I think that comes with the need > > > to > > > > > improve on our automated validation and testing. > > > > > > > > > > > > We have proven that we have not kept master in a good state and > > > adding > > > > > more releases (more branches) will make it even harder to keep that > > > > > stability we are already fighting. > > > > > > > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> > > > wrote: > > > > > > > > > > > > Yeah, I get it. No one company should be driving release > > > > > schedules/scope. > > > > > > > > > > > > What I was really getting at is if ANY company has a recent > > > version > > > > > in a > > > > > > test or prod-like enviro (meaning the version is being exercised > > > and > > > > > > thoroughly tested), then maybe we consider simply cutting a > > > release > > > > > from > > > > > > that version. For example, imagine company X had this version in > > > a > > > > > > test/prod enviro: > > > > > > > > > > > > Master-10287.7e62d07 ( > > > > > > > > > > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > > > > ) > > > > > > > > > > > > Then I would be in favor of cutting the 4.0 release from > > > > > > Master-10287.7e62d07 as we know it is proven to work based on > > > > > company X's > > > > > > actual use of it. > > > > > > > > > > > > The truth is (imo), we don't have the bandwidth to manually > > > verify > > > > > releases > > > > > > that we are not using/have not used nor do we have the necessary > > > > > automated > > > > > > test coverage to verify these releases. So most of our releases > > > are > > > > > > significantly untested. > > > > > > > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <[email protected]> > > > > > wrote: > > > > > > > > > > > > > I'm really not a fan of allowing Comcast to dictate the release > > > > > scope and > > > > > > > schedule. If cherry-picking is too messy, then we can just cut > > > new > > > > > minor > > > > > > > releases directly from master. > > > > > > > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > I don't think it's as easy as cherry picking (backporting) > > > > > certain > > > > > > > features > > > > > > > > into a release branch. I could be wrong but I really don't > > > think > > > > > it is. > > > > > > > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master and we > > > go > > > > > through > > > > > > > with > > > > > > > > our normal process of testing, validating, etc. At the same > > > > > time, 4.1 > > > > > > > > branch is created from 4.0. Master moves on as normal. Let's > > > say > > > > > 25 > > > > > > > commits > > > > > > > > come in to master during the next 6 weeks. During that 6 > > > weeks, > > > > > we cherry > > > > > > > > pick only the features that look interesting for 4.1? (the > > > rest > > > > > just stay > > > > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 > > > > > weeks apart > > > > > > > and > > > > > > > > have only "interesting" features so the releases are > > > > > incrementally very > > > > > > > > small and much easier to test/validate, however, those > > > features > > > > > might > > > > > > > > depend on other commits. At some point, cherry picking > > > becomes > > > > > impossible > > > > > > > > as the foundation of the code base has shifted so > > > drastically. > > > > > Cherry > > > > > > > > picking will become very messy and introduce a high level of > > > > > risk imo. > > > > > > > > > > > > > > > > Also, you'd have to manage the scope of these minor releases > > > > > which we are > > > > > > > > not very good at because the designated release manager has a > > > > > full-time > > > > > > > job > > > > > > > > to attend to and now we have even more releases to > > > > > > > > scope/test/validate/approve. I like the idea of faster > > > releases > > > > > but this > > > > > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > > > > > > > > > Ok, so rather than shooting down an idea with no alternative > > > > > proposal, I > > > > > > > > propose this. At Comcast, we are trying to get to the point > > > > > where we > > > > > > > > release TC to prod from the head of master every few weeks. > > > What > > > > > if every > > > > > > > > time Comcast does an internal upgrade, they note the commit > > > hash > > > > > and once > > > > > > > > comcast is happy with the upgrade in their prod enviro, they > > > > > propose a > > > > > > > > release pinned to that commit hash? With this approach you'd > > > end > > > > > up with: > > > > > > > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > > > > 2. well tested releases as it's in a real prod enviro > > > > > > > > 3. little/no validation work for anyone > > > > > > > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else is out > > > > > front at the > > > > > > > > edge of master and running it in a trusted enviro, they could > > > > > propose a > > > > > > > > release as well. > > > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I have been thinking about how we can get better with our > > > > > release > > > > > > > > cadence. > > > > > > > > > I feel like we have slowed to a crawl and not been as good > > > as > > > > > we should > > > > > > > > > about how we release. Our last Major release was in March > > > and > > > > > we > > > > > > > haven't > > > > > > > > > had a real release since. Moving forward I would like to > > > see > > > > > us get > > > > > > > on > > > > > > > > a > > > > > > > > > more consistent cadence and provide smaller releases that > > > can > > > > > be more > > > > > > > > > easily digested by the community. It's my hope that we can > > > > > get to a > > > > > > > > > cadence where we are doing releases every 4 to 6 weeks, > > > we are > > > > > > > releasing > > > > > > > > > in such a way that not all releases all required (e.g. if > > > you > > > > > are on > > > > > > > 4.0 > > > > > > > > > you don't need 4.1, you can just install 4.2), and we are > > > more > > > > > > > deliberate > > > > > > > > > about what we put into a release. > > > > > > > > > > > > > > > > > > I think we can accomplish this by using our release > > > branches > > > > > better. > > > > > > > For > > > > > > > > > each major release we will cut a release branch from master > > > > > (e.g. 4.x) > > > > > > > > and > > > > > > > > > then we will use cherry-picking to add new features and bug > > > > > fixes to > > > > > > > that > > > > > > > > > release. This means that if 4.0 has been released and you > > > > > want to get > > > > > > > > your > > > > > > > > > feature/bug fix in 4.1, you will first submit your PR to > > > > > master and > > > > > > > then > > > > > > > > > cheery pick your squash merged commit to the 4.x branch > > > which > > > > > we will > > > > > > > use > > > > > > > > > to create the 4.1 release. I think we should allow either > > > the > > > > > > > > contributor > > > > > > > > > or the committer (who is merging the PR) to suggest if a > > > > > feature goes > > > > > > > > into > > > > > > > > > the release branch, and if we disagree we can take it to > > > the > > > > > list. If > > > > > > > we > > > > > > > > > decide that every PR to master also goes into 4.1 then so > > > be > > > > > it, but at > > > > > > > > > least we are making a conscious decision of what goes into > > > the > > > > > next > > > > > > > > > release. This will allow us to not be so worried about > > > what > > > > > we are > > > > > > > > > merging into master and how that will affect our next > > > release. > > > > > > > According > > > > > > > > > to our 6 week cadence we will cut a new release from the > > > > > current > > > > > > > feature > > > > > > > > > branch and put that up for a vote. These releases will be > > > > > small and > > > > > > > > > testable enough that we can get feedback quickly and move > > > from > > > > > RC to > > > > > > > > > release in a short time. If a release has too many issues > > > > > then we may > > > > > > > > end > > > > > > > > > up deciding to skip that minor release in favor of the next > > > > > one, > > > > > > > > hopefully > > > > > > > > > this does not happen very often if at all. > > > > > > > > > > > > > > > > > > Once a breaking change is introduced to master which will > > > > > require a new > > > > > > > > > major release, we will create a new major release branch > > > from > > > > > master > > > > > > > > (which > > > > > > > > > will mean a new release manager). We will then repeat the > > > > > same process > > > > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > > > > > > > As for LTS we will provide support for the latest major > > > > > release plus > > > > > > > the > > > > > > > > > one previous. So, once we release 4.0 we will support 4.0 > > > and > > > > > 3.1. > > > > > > > Any > > > > > > > > > security issues that arise will -- if present -- be > > > applied to > > > > > each of > > > > > > > > > these versions. Once 4.1 is released we will support 4.1 > > > and > > > > > 3.1, etc, > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > Please let me know if you have any thoughts on this plan. > > > If > > > > > there are > > > > > > > > no > > > > > > > > > major objections I would like to try this with 4.x with the > > > > > idea that > > > > > > > we > > > > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > > > > > > > We need to get better at releasing and something needs to > > > > > change, > > > > > > > > hopefully > > > > > > > > > this can get us going in the right direction. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > > > > cherry-picks to > > > > > > > our > > > > > > > > > release branch to control what goes into a minor release. > > > > > Major > > > > > > > releases > > > > > > > > > will be cut from master as necessary - such as if there is > > > a > > > > > breaking > > > > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
