I guess my concern is really about what possible reasons we might have to not cherry-pick a specific PR that had been merged into master already. The only reasons I can think of to not cherry-pick something would be: 1. the PR was completely broken (e.g. breaks the build, tests don't pass, doesn't compile, etc). 2. the PR one one of a larger multi-PR "feature", and *we know* releasing some parts early could cause problems (that knowing part is crucial)
We should always strive to catch issue #1 in our CI before it merges into master. When things like that merge it can block people from doing development starting at that point until it's fixed. I can see why issue #2 might arise due to multiple different people developing one larger feature, but we should still strive to break larger features up into individual pieces that can reasonably stand on their own if possible. Due to time constraints or other reasons I can see that might be a little difficult. That said, I would still strive to make the 4.x release branch a snapshot of master at some "known good point" in time. Let's say we cut 4.x at commit A, then a broken commit B gets merged into master. If it's completely obvious that it's a broken commit, I'd wait until at least a commit C gets merged to fix it. Then I would cherry-pick B and C into 4.x at the same time. If the fix is really like a commit E with unrelated commits C and D in between, ideally I would still wait to cherry-pick C and D until a "fix" commit E was merged (to fix commit B), at which point I'd cherrypick commits B through E at the same time, in order, to avoid any possible merge conflicts. I think to make that process easier for myself I would just cherry-pick everything into the 4.x branch as it gets merged into master. Then, after 4-6 weeks of cherry-picking commits, if unstable commits were not fixed/completed in time, I would reset the 4.x branch to the last stable commit. This would mean that unstable commits/incomplete features could potentially block things from going into the release. I might count that as a positive because it would incentivize us to improve our CI and do our best to also keep the master branch as stable as possible. The end result would be the same as just continually jumping up the master branch from known good, stable points in time to the next, creating a minor release branch at each of those points. - Rawlin On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <[email protected]> wrote: > > Sorry, I have taken too long to respond. > > +1000000 on better CI and Testing, I think we all agree that needs to get > better and is a different beast than this topic. > > As for the release branches, I am trying to follow a process similar to > ATS, which is usually the model we try to follow with our project. > While I agree that branching from master is the ideal solution, we tried > that and it didn't work. I feel like we are trying to agree to do the same > thing that we already tried and didn't work. I think we should have a > little freedom to commit things into master and, if they break things, work > on getting them fixed. We also might commit PRs to master that are partial > implementations of a bigger feature. If we are using a time based > strategy to cut our releases from master then we are going to be > continually worrying about making sure Master stays not broken. I think we > should try to keep master deployable, but it's fine if it's not for short > periods of time. > > I am not trying to suggest that we are "picky" about what goes into our > release branch. I think we should try to cherry-pick as much as we can > into our release branch once it is ready to be cherry-picked. I think we > should be able to take the tip of our release branch and deploy it whenever > we want with some confidence since we know that features/bug-fixes that > have been cherry-picked have already been reviewed and tested against > master. Yes, its more work, but it also give us the ability to have more > confidence in what is in our release at any given time. Again, we should > be trying to cherry-pick in everything that makes sense to cherry-pick. > > I think the process that I would like to see looks like this: > - Cut a 4.x branch from Master > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x branch > - Tag the 4.0.0 release from the 4.0.x branch > - New non-breaking changes and bug fixes that go into master are also > Cherry Picked to 4.x > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we > are ready for a RC. > - Rinse and repeat for all 4.x releases > - Once there's a need for a major release we make a 5.x branch from master > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, then > 4.0.x which we use to create the new 4.0.1 release (assuming we don't have > 4.1 yet). Yes, that is a more work than today, but I think A) it's not > really that bad and B) it gives us more control. > > This should allow us to deploy much quicker and with more confidence than > is possible today. We can also choose to deploy from any point in time > from the 4.x branch with relatively high confidence. > > I'll wrap this up by saying that I don't think there's a perfect solution, > but I think we can agree on something that is worth trying at least with > the 4.x release life cycle. Once we have some experience, we can figure > out where it is not meeting our needs and figure out how to adjust as > necessary. I think we can all agree our current process is not working. > > Thanks, > Dave > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> wrote: > > > +1 on more frequent releases > > +1 on improving our CI. We need to get our CI to the point where we're > > confident deploying changes to production if the CI passes. > > +1 on "trunk-based" development as opposed to GitFlow > > -1 on cherry-picking new features. This is risky, tedious, and > > represents unnecessary effort that would be better spent elsewhere, > > like improving our CI or adding more automated tests. > > +1 on cutting minor release branches directly off master > > > > I think cutting minor release branches directly off master can put us > > in a win-win scenario. Our releases become smaller and more frequent, > > and we avoid the unnecessary risk and effort of cherry-picking new > > features onto stale release branches. > > > > This is the process that I would propose: > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > 2. Develop master for 4-6 weeks. > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > 5. Whenever a major (show-stopping) bug or security issue is found, > > cherry-pick the fix from master into the latest release branch (e.g. > > 4.1.x), and create a 4.1.1 RC. > > > > As the 4.x release manager, I really don't want to get into the > > business of selecting what should and should not go into a particular > > release. Every 4-6 weeks, we should just cut a new minor release > > branch off master and take everything that was contributed during that > > time, as long as everything builds fine and all the tests pass. The > > master branch, while it doesn't always have to be perfect, should at > > least always be deployable. If a new feature is merged with bugs and > > becomes a show-stopper, it should be either immediately remedied or > > reverted on master while we work on a fix. > > > > - Rawlin > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > <[email protected]> wrote: > > > > > > In my opinion, in order to get to a cadence we are discussing we need to > > put a lot more work into the CI system. It has been failing consistently, > > doesn’t block PR approvals/merges and when it actually runs it does not > > test anything besides build and license headers. > > > > > > In the past couple of weeks we have had (off the top of my head): > > > > > > * Unit tests broken > > > * TO API tests broken > > > * Golang vendor issues > > > * TO Go build issues > > > > > > All were merged and broke master. > > > > > > Yes, having release branches sounds great and more aggressive cadence > > will minimize the amount of risk but I think that comes with the need to > > improve on our automated validation and testing. > > > > > > We have proven that we have not kept master in a good state and adding > > more releases (more branches) will make it even harder to keep that > > stability we are already fighting. > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> wrote: > > > > > > Yeah, I get it. No one company should be driving release > > schedules/scope. > > > > > > What I was really getting at is if ANY company has a recent version > > in a > > > test or prod-like enviro (meaning the version is being exercised and > > > thoroughly tested), then maybe we consider simply cutting a release > > from > > > that version. For example, imagine company X had this version in a > > > test/prod enviro: > > > > > > Master-10287.7e62d07 ( > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > ) > > > > > > Then I would be in favor of cutting the 4.0 release from > > > Master-10287.7e62d07 as we know it is proven to work based on > > company X's > > > actual use of it. > > > > > > The truth is (imo), we don't have the bandwidth to manually verify > > releases > > > that we are not using/have not used nor do we have the necessary > > automated > > > test coverage to verify these releases. So most of our releases are > > > significantly untested. > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <[email protected]> > > wrote: > > > > > > > I'm really not a fan of allowing Comcast to dictate the release > > scope and > > > > schedule. If cherry-picking is too messy, then we can just cut new > > minor > > > > releases directly from master. > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <[email protected]> > > wrote: > > > > > > > > > I don't think it's as easy as cherry picking (backporting) > > certain > > > > features > > > > > into a release branch. I could be wrong but I really don't think > > it is. > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master and we go > > through > > > > with > > > > > our normal process of testing, validating, etc. At the same > > time, 4.1 > > > > > branch is created from 4.0. Master moves on as normal. Let's say > > 25 > > > > commits > > > > > come in to master during the next 6 weeks. During that 6 weeks, > > we cherry > > > > > pick only the features that look interesting for 4.1? (the rest > > just stay > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 > > weeks apart > > > > and > > > > > have only "interesting" features so the releases are > > incrementally very > > > > > small and much easier to test/validate, however, those features > > might > > > > > depend on other commits. At some point, cherry picking becomes > > impossible > > > > > as the foundation of the code base has shifted so drastically. > > Cherry > > > > > picking will become very messy and introduce a high level of > > risk imo. > > > > > > > > > > Also, you'd have to manage the scope of these minor releases > > which we are > > > > > not very good at because the designated release manager has a > > full-time > > > > job > > > > > to attend to and now we have even more releases to > > > > > scope/test/validate/approve. I like the idea of faster releases > > but this > > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > > > Ok, so rather than shooting down an idea with no alternative > > proposal, I > > > > > propose this. At Comcast, we are trying to get to the point > > where we > > > > > release TC to prod from the head of master every few weeks. What > > if every > > > > > time Comcast does an internal upgrade, they note the commit hash > > and once > > > > > comcast is happy with the upgrade in their prod enviro, they > > propose a > > > > > release pinned to that commit hash? With this approach you'd end > > up with: > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > 2. well tested releases as it's in a real prod enviro > > > > > 3. little/no validation work for anyone > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else is out > > front at the > > > > > edge of master and running it in a trusted enviro, they could > > propose a > > > > > release as well. > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > [email protected]> > > > > > wrote: > > > > > > > > > > > I have been thinking about how we can get better with our > > release > > > > > cadence. > > > > > > I feel like we have slowed to a crawl and not been as good as > > we should > > > > > > about how we release. Our last Major release was in March and > > we > > > > haven't > > > > > > had a real release since. Moving forward I would like to see > > us get > > > > on > > > > > a > > > > > > more consistent cadence and provide smaller releases that can > > be more > > > > > > easily digested by the community. It's my hope that we can > > get to a > > > > > > cadence where we are doing releases every 4 to 6 weeks, we are > > > > releasing > > > > > > in such a way that not all releases all required (e.g. if you > > are on > > > > 4.0 > > > > > > you don't need 4.1, you can just install 4.2), and we are more > > > > deliberate > > > > > > about what we put into a release. > > > > > > > > > > > > I think we can accomplish this by using our release branches > > better. > > > > For > > > > > > each major release we will cut a release branch from master > > (e.g. 4.x) > > > > > and > > > > > > then we will use cherry-picking to add new features and bug > > fixes to > > > > that > > > > > > release. This means that if 4.0 has been released and you > > want to get > > > > > your > > > > > > feature/bug fix in 4.1, you will first submit your PR to > > master and > > > > then > > > > > > cheery pick your squash merged commit to the 4.x branch which > > we will > > > > use > > > > > > to create the 4.1 release. I think we should allow either the > > > > > contributor > > > > > > or the committer (who is merging the PR) to suggest if a > > feature goes > > > > > into > > > > > > the release branch, and if we disagree we can take it to the > > list. If > > > > we > > > > > > decide that every PR to master also goes into 4.1 then so be > > it, but at > > > > > > least we are making a conscious decision of what goes into the > > next > > > > > > release. This will allow us to not be so worried about what > > we are > > > > > > merging into master and how that will affect our next release. > > > > According > > > > > > to our 6 week cadence we will cut a new release from the > > current > > > > feature > > > > > > branch and put that up for a vote. These releases will be > > small and > > > > > > testable enough that we can get feedback quickly and move from > > RC to > > > > > > release in a short time. If a release has too many issues > > then we may > > > > > end > > > > > > up deciding to skip that minor release in favor of the next > > one, > > > > > hopefully > > > > > > this does not happen very often if at all. > > > > > > > > > > > > Once a breaking change is introduced to master which will > > require a new > > > > > > major release, we will create a new major release branch from > > master > > > > > (which > > > > > > will mean a new release manager). We will then repeat the > > same process > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > As for LTS we will provide support for the latest major > > release plus > > > > the > > > > > > one previous. So, once we release 4.0 we will support 4.0 and > > 3.1. > > > > Any > > > > > > security issues that arise will -- if present -- be applied to > > each of > > > > > > these versions. Once 4.1 is released we will support 4.1 and > > 3.1, etc, > > > > > > etc. > > > > > > > > > > > > Please let me know if you have any thoughts on this plan. If > > there are > > > > > no > > > > > > major objections I would like to try this with 4.x with the > > idea that > > > > we > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > We need to get better at releasing and something needs to > > change, > > > > > hopefully > > > > > > this can get us going in the right direction. > > > > > > > > > > > > Thanks, > > > > > > Dave > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > cherry-picks to > > > > our > > > > > > release branch to control what goes into a minor release. > > Major > > > > releases > > > > > > will be cut from master as necessary - such as if there is a > > breaking > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > >
