+1 on more frequent releases +1 on improving our CI. We need to get our CI to the point where we're confident deploying changes to production if the CI passes. +1 on "trunk-based" development as opposed to GitFlow -1 on cherry-picking new features. This is risky, tedious, and represents unnecessary effort that would be better spent elsewhere, like improving our CI or adding more automated tests. +1 on cutting minor release branches directly off master
I think cutting minor release branches directly off master can put us in a win-win scenario. Our releases become smaller and more frequent, and we avoid the unnecessary risk and effort of cherry-picking new features onto stale release branches. This is the process that I would propose: 1. Cut a 4.0.x branch off master which becomes the 4.0 RC 2. Develop master for 4-6 weeks. 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. 4. Repeat steps 2 and 3 for new minor releases as necessary. 5. Whenever a major (show-stopping) bug or security issue is found, cherry-pick the fix from master into the latest release branch (e.g. 4.1.x), and create a 4.1.1 RC. As the 4.x release manager, I really don't want to get into the business of selecting what should and should not go into a particular release. Every 4-6 weeks, we should just cut a new minor release branch off master and take everything that was contributed during that time, as long as everything builds fine and all the tests pass. The master branch, while it doesn't always have to be perfect, should at least always be deployable. If a new feature is merged with bugs and becomes a show-stopper, it should be either immediately remedied or reverted on master while we work on a fix. - Rawlin On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael <[email protected]> wrote: > > In my opinion, in order to get to a cadence we are discussing we need to put > a lot more work into the CI system. It has been failing consistently, doesn’t > block PR approvals/merges and when it actually runs it does not test anything > besides build and license headers. > > In the past couple of weeks we have had (off the top of my head): > > * Unit tests broken > * TO API tests broken > * Golang vendor issues > * TO Go build issues > > All were merged and broke master. > > Yes, having release branches sounds great and more aggressive cadence will > minimize the amount of risk but I think that comes with the need to improve > on our automated validation and testing. > > We have proven that we have not kept master in a good state and adding more > releases (more branches) will make it even harder to keep that stability we > are already fighting. > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> wrote: > > Yeah, I get it. No one company should be driving release schedules/scope. > > What I was really getting at is if ANY company has a recent version in a > test or prod-like enviro (meaning the version is being exercised and > thoroughly tested), then maybe we consider simply cutting a release from > that version. For example, imagine company X had this version in a > test/prod enviro: > > Master-10287.7e62d07 ( > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master) > > Then I would be in favor of cutting the 4.0 release from > Master-10287.7e62d07 as we know it is proven to work based on company X's > actual use of it. > > The truth is (imo), we don't have the bandwidth to manually verify > releases > that we are not using/have not used nor do we have the necessary automated > test coverage to verify these releases. So most of our releases are > significantly untested. > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <[email protected]> wrote: > > > I'm really not a fan of allowing Comcast to dictate the release scope > and > > schedule. If cherry-picking is too messy, then we can just cut new minor > > releases directly from master. > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <[email protected]> > wrote: > > > > > I don't think it's as easy as cherry picking (backporting) certain > > features > > > into a release branch. I could be wrong but I really don't think it > is. > > > > > > So what I'm hearing is that 4.0 gets cut from master and we go through > > with > > > our normal process of testing, validating, etc. At the same time, 4.1 > > > branch is created from 4.0. Master moves on as normal. Let's say 25 > > commits > > > come in to master during the next 6 weeks. During that 6 weeks, we > cherry > > > pick only the features that look interesting for 4.1? (the rest just > stay > > > in master and will get picked up in 5.0?) > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks > apart > > and > > > have only "interesting" features so the releases are incrementally > very > > > small and much easier to test/validate, however, those features might > > > depend on other commits. At some point, cherry picking becomes > impossible > > > as the foundation of the code base has shifted so drastically. Cherry > > > picking will become very messy and introduce a high level of risk imo. > > > > > > Also, you'd have to manage the scope of these minor releases which we > are > > > not very good at because the designated release manager has a > full-time > > job > > > to attend to and now we have even more releases to > > > scope/test/validate/approve. I like the idea of faster releases but > this > > > sounds like more work that nobody has the bandwidth for. > > > > > > Ok, so rather than shooting down an idea with no alternative > proposal, I > > > propose this. At Comcast, we are trying to get to the point where we > > > release TC to prod from the head of master every few weeks. What if > every > > > time Comcast does an internal upgrade, they note the commit hash and > once > > > comcast is happy with the upgrade in their prod enviro, they propose a > > > release pinned to that commit hash? With this approach you'd end up > with: > > > > > > 1. fast releases (hopefully every few weeks) > > > 2. well tested releases as it's in a real prod enviro > > > 3. little/no validation work for anyone > > > > > > Also, this doesn't have to be Comcast. If anyone else is out front at > the > > > edge of master and running it in a trusted enviro, they could propose > a > > > release as well. > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman > <[email protected]> > > > wrote: > > > > > > > I have been thinking about how we can get better with our release > > > cadence. > > > > I feel like we have slowed to a crawl and not been as good as we > should > > > > about how we release. Our last Major release was in March and we > > haven't > > > > had a real release since. Moving forward I would like to see us > get > > on > > > a > > > > more consistent cadence and provide smaller releases that can be > more > > > > easily digested by the community. It's my hope that we can get to a > > > > cadence where we are doing releases every 4 to 6 weeks, we are > > releasing > > > > in such a way that not all releases all required (e.g. if you are on > > 4.0 > > > > you don't need 4.1, you can just install 4.2), and we are more > > deliberate > > > > about what we put into a release. > > > > > > > > I think we can accomplish this by using our release branches better. > > For > > > > each major release we will cut a release branch from master (e.g. > 4.x) > > > and > > > > then we will use cherry-picking to add new features and bug fixes to > > that > > > > release. This means that if 4.0 has been released and you want to > get > > > your > > > > feature/bug fix in 4.1, you will first submit your PR to master and > > then > > > > cheery pick your squash merged commit to the 4.x branch which we > will > > use > > > > to create the 4.1 release. I think we should allow either the > > > contributor > > > > or the committer (who is merging the PR) to suggest if a feature > goes > > > into > > > > the release branch, and if we disagree we can take it to the list. > If > > we > > > > decide that every PR to master also goes into 4.1 then so be it, > but at > > > > least we are making a conscious decision of what goes into the next > > > > release. This will allow us to not be so worried about what we are > > > > merging into master and how that will affect our next release. > > According > > > > to our 6 week cadence we will cut a new release from the current > > feature > > > > branch and put that up for a vote. These releases will be small and > > > > testable enough that we can get feedback quickly and move from RC to > > > > release in a short time. If a release has too many issues then we > may > > > end > > > > up deciding to skip that minor release in favor of the next one, > > > hopefully > > > > this does not happen very often if at all. > > > > > > > > Once a breaking change is introduced to master which will require a > new > > > > major release, we will create a new major release branch from master > > > (which > > > > will mean a new release manager). We will then repeat the same > process > > > > with the new release branch, etc, etc. > > > > > > > > As for LTS we will provide support for the latest major release plus > > the > > > > one previous. So, once we release 4.0 we will support 4.0 and 3.1. > > Any > > > > security issues that arise will -- if present -- be applied to each > of > > > > these versions. Once 4.1 is released we will support 4.1 and 3.1, > etc, > > > > etc. > > > > > > > > Please let me know if you have any thoughts on this plan. If there > are > > > no > > > > major objections I would like to try this with 4.x with the idea > that > > we > > > > will adjust how we do things as necessary. > > > > > > > > We need to get better at releasing and something needs to change, > > > hopefully > > > > this can get us going in the right direction. > > > > > > > > Thanks, > > > > Dave > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using cherry-picks to > > our > > > > release branch to control what goes into a minor release. Major > > releases > > > > will be cut from master as necessary - such as if there is a > breaking > > > > change that is introduced. > > > > > > > > > > >
