Ok, I'm starting to see the light. If we moved faster, 4.1, 4.2, 4.3 for example 6 weeks apart then maybe where I work (Comcast) would get on board with actually using those releases (rather than some arbitrary point on master because the open source releases are moving too slow for us).
But I still think cherry-picking select features into a release could be problematic. I would suggest: 4.0 (from master) 4.1 (from master) 4.2 (from master) ... 5.0 (from master) and the only reason for the major bump is something pretty big went into master. also +100 on CI that: 1. actually runs...consistently (I think we should look more into these Github Actions) 2. tests a good deal of things (build, unit tests, api tests, UI tests, maybe some integration tests?) jeremy On Wed, Oct 30, 2019 at 9:56 AM ocket 8888 <[email protected]> wrote: > > What I was really getting at is if ANY company has a recent version in > a > > test or prod-like enviro (meaning the version is being exercised and > > thoroughly tested), then maybe we consider simply cutting a release > from > > that version > > I think the idea s that companies should stop doing their own versioning - > which was totally necessary because it's unreasonable to wait half a year > for an Apache release - and start deploying official releases. That way the > releases ARE thoroughly exercised and tested, just that the commit hashes > aren't dictated by the companies. Companies could refuse, of course, but > that only makes things harder for them IMO. At Comcast it's taken > notoriously long to validate releases, and making them smaller will help > that. > > > In my opinion, in order to get to a cadence we are discussing we need to > put a lot more work into the CI system. > > I agree, it's all but worthless at the moment and testing is vital to > providing stable releases. I know GitHub Actions have been discussed > before, but they are coming out of beta in a couple of weeks and if the > problem with CI is capacity then maybe we could disable the Jenkins jobs > and set up Actions in their stead. > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael < > [email protected]> > wrote: > > > In my opinion, in order to get to a cadence we are discussing we need to > > put a lot more work into the CI system. It has been failing consistently, > > doesn’t block PR approvals/merges and when it actually runs it does not > > test anything besides build and license headers. > > > > In the past couple of weeks we have had (off the top of my head): > > > > * Unit tests broken > > * TO API tests broken > > * Golang vendor issues > > * TO Go build issues > > > > All were merged and broke master. > > > > Yes, having release branches sounds great and more aggressive cadence > will > > minimize the amount of risk but I think that comes with the need to > improve > > on our automated validation and testing. > > > > We have proven that we have not kept master in a good state and adding > > more releases (more branches) will make it even harder to keep that > > stability we are already fighting. > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> wrote: > > > > Yeah, I get it. No one company should be driving release > > schedules/scope. > > > > What I was really getting at is if ANY company has a recent version > in > > a > > test or prod-like enviro (meaning the version is being exercised and > > thoroughly tested), then maybe we consider simply cutting a release > > from > > that version. For example, imagine company X had this version in a > > test/prod enviro: > > > > Master-10287.7e62d07 ( > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > ) > > > > Then I would be in favor of cutting the 4.0 release from > > Master-10287.7e62d07 as we know it is proven to work based on company > > X's > > actual use of it. > > > > The truth is (imo), we don't have the bandwidth to manually verify > > releases > > that we are not using/have not used nor do we have the necessary > > automated > > test coverage to verify these releases. So most of our releases are > > significantly untested. > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <[email protected]> > > wrote: > > > > > I'm really not a fan of allowing Comcast to dictate the release > > scope and > > > schedule. If cherry-picking is too messy, then we can just cut new > > minor > > > releases directly from master. > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <[email protected] > > > > wrote: > > > > > > > I don't think it's as easy as cherry picking (backporting) > certain > > > features > > > > into a release branch. I could be wrong but I really don't think > > it is. > > > > > > > > So what I'm hearing is that 4.0 gets cut from master and we go > > through > > > with > > > > our normal process of testing, validating, etc. At the same time, > > 4.1 > > > > branch is created from 4.0. Master moves on as normal. Let's say > 25 > > > commits > > > > come in to master during the next 6 weeks. During that 6 weeks, > we > > cherry > > > > pick only the features that look interesting for 4.1? (the rest > > just stay > > > > in master and will get picked up in 5.0?) > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks > > apart > > > and > > > > have only "interesting" features so the releases are > incrementally > > very > > > > small and much easier to test/validate, however, those features > > might > > > > depend on other commits. At some point, cherry picking becomes > > impossible > > > > as the foundation of the code base has shifted so drastically. > > Cherry > > > > picking will become very messy and introduce a high level of risk > > imo. > > > > > > > > Also, you'd have to manage the scope of these minor releases > which > > we are > > > > not very good at because the designated release manager has a > > full-time > > > job > > > > to attend to and now we have even more releases to > > > > scope/test/validate/approve. I like the idea of faster releases > > but this > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > Ok, so rather than shooting down an idea with no alternative > > proposal, I > > > > propose this. At Comcast, we are trying to get to the point where > > we > > > > release TC to prod from the head of master every few weeks. What > > if every > > > > time Comcast does an internal upgrade, they note the commit hash > > and once > > > > comcast is happy with the upgrade in their prod enviro, they > > propose a > > > > release pinned to that commit hash? With this approach you'd end > > up with: > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > 2. well tested releases as it's in a real prod enviro > > > > 3. little/no validation work for anyone > > > > > > > > Also, this doesn't have to be Comcast. If anyone else is out > front > > at the > > > > edge of master and running it in a trusted enviro, they could > > propose a > > > > release as well. > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > [email protected]> > > > > wrote: > > > > > > > > > I have been thinking about how we can get better with our > release > > > > cadence. > > > > > I feel like we have slowed to a crawl and not been as good as > we > > should > > > > > about how we release. Our last Major release was in March and > we > > > haven't > > > > > had a real release since. Moving forward I would like to see > > us get > > > on > > > > a > > > > > more consistent cadence and provide smaller releases that can > be > > more > > > > > easily digested by the community. It's my hope that we can get > > to a > > > > > cadence where we are doing releases every 4 to 6 weeks, we are > > > releasing > > > > > in such a way that not all releases all required (e.g. if you > > are on > > > 4.0 > > > > > you don't need 4.1, you can just install 4.2), and we are more > > > deliberate > > > > > about what we put into a release. > > > > > > > > > > I think we can accomplish this by using our release branches > > better. > > > For > > > > > each major release we will cut a release branch from master > > (e.g. 4.x) > > > > and > > > > > then we will use cherry-picking to add new features and bug > > fixes to > > > that > > > > > release. This means that if 4.0 has been released and you want > > to get > > > > your > > > > > feature/bug fix in 4.1, you will first submit your PR to master > > and > > > then > > > > > cheery pick your squash merged commit to the 4.x branch which > we > > will > > > use > > > > > to create the 4.1 release. I think we should allow either the > > > > contributor > > > > > or the committer (who is merging the PR) to suggest if a > feature > > goes > > > > into > > > > > the release branch, and if we disagree we can take it to the > > list. If > > > we > > > > > decide that every PR to master also goes into 4.1 then so be > it, > > but at > > > > > least we are making a conscious decision of what goes into the > > next > > > > > release. This will allow us to not be so worried about what > we > > are > > > > > merging into master and how that will affect our next release. > > > According > > > > > to our 6 week cadence we will cut a new release from the > current > > > feature > > > > > branch and put that up for a vote. These releases will be > small > > and > > > > > testable enough that we can get feedback quickly and move from > > RC to > > > > > release in a short time. If a release has too many issues then > > we may > > > > end > > > > > up deciding to skip that minor release in favor of the next > one, > > > > hopefully > > > > > this does not happen very often if at all. > > > > > > > > > > Once a breaking change is introduced to master which will > > require a new > > > > > major release, we will create a new major release branch from > > master > > > > (which > > > > > will mean a new release manager). We will then repeat the same > > process > > > > > with the new release branch, etc, etc. > > > > > > > > > > As for LTS we will provide support for the latest major release > > plus > > > the > > > > > one previous. So, once we release 4.0 we will support 4.0 and > > 3.1. > > > Any > > > > > security issues that arise will -- if present -- be applied to > > each of > > > > > these versions. Once 4.1 is released we will support 4.1 and > > 3.1, etc, > > > > > etc. > > > > > > > > > > Please let me know if you have any thoughts on this plan. If > > there are > > > > no > > > > > major objections I would like to try this with 4.x with the > idea > > that > > > we > > > > > will adjust how we do things as necessary. > > > > > > > > > > We need to get better at releasing and something needs to > change, > > > > hopefully > > > > > this can get us going in the right direction. > > > > > > > > > > Thanks, > > > > > Dave > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > cherry-picks to > > > our > > > > > release branch to control what goes into a minor release. > Major > > > releases > > > > > will be cut from master as necessary - such as if there is a > > breaking > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > >
