Re: [DISCUSS] Hotfix release procedure
Took the liberty to update the confluence wiki to reflect the "create branch off last released tag with only delta required" for hotfixes. https://cwiki.apache.org/confluence/display/CASSANDRA/Patching%2C+versioning%2C+and+LTS+releases If anyone disagrees please let me know. On Tue, Feb 15, 2022, at 3:00 PM, Brandon Williams wrote: > Any committer can submit a devbranch build... > > Kind Regards, > Brandon > > On Tue, Feb 15, 2022 at 1:58 PM Mick Semb Wever wrote: > > > > > > > >> We've done concurrent releases without security before, and you follow > >> much closer than others. I feel most people, if they saw all of the > >> changes reverted and a release of a single fix, would either instantly > >> know it's security (high confidence pointer to exactly which patch) OR > >> assume someone botched the release prep and draw attention to it. So we're > >> trading "someone who's very involved has a high confidence it's security > >> but has to dig through 30 patches to find it" vs "everyone knows exactly > >> what's going on", the former seems better > > > > > > > > My initial thoughts are aligned with what Jeff writes here. Furthermore > > when you apply our new-found practice of stable trunk and focus on QA, > > which I hope is continuously improving, this point only becomes more valid. > > > > And how to do CI (we need a green CI for a release remember ;) on a private > > commit is something i really am unsure how we would do… >
Re: [DISCUSS] Hotfix release procedure
Any committer can submit a devbranch build... Kind Regards, Brandon On Tue, Feb 15, 2022 at 1:58 PM Mick Semb Wever wrote: > > > >> We've done concurrent releases without security before, and you follow much >> closer than others. I feel most people, if they saw all of the changes >> reverted and a release of a single fix, would either instantly know it's >> security (high confidence pointer to exactly which patch) OR assume someone >> botched the release prep and draw attention to it. So we're trading "someone >> who's very involved has a high confidence it's security but has to dig >> through 30 patches to find it" vs "everyone knows exactly what's going on", >> the former seems better > > > > My initial thoughts are aligned with what Jeff writes here. Furthermore when > you apply our new-found practice of stable trunk and focus on QA, which I > hope is continuously improving, this point only becomes more valid. > > And how to do CI (we need a green CI for a release remember ;) on a private > commit is something i really am unsure how we would do…
Re: [DISCUSS] Hotfix release procedure
We've done concurrent releases without security before, and you follow much > closer than others. I feel most people, if they saw all of the > changes reverted and a release of a single fix, would either instantly know > it's security (high confidence pointer to exactly which patch) OR assume > someone botched the release prep and draw attention to it. So we're trading > "someone who's very involved has a high confidence it's security but has to > dig through 30 patches to find it" vs "everyone knows exactly what's going > on", the former seems better > My initial thoughts are aligned with what Jeff writes here. Furthermore when you apply our new-found practice of stable trunk and focus on QA, which I hope is continuously improving, this point only becomes more valid. And how to do CI (we need a green CI for a release remember ;) on a private commit is something i really am unsure how we would do…
Re: [DISCUSS] Hotfix release procedure
Correct. No need to revert anything or keep extra branches around. You just checkout the tag and then make a branch with the single fix on it. > On Feb 15, 2022, at 10:08 AM, Josh McKenzie wrote: > > > Was thinking that too after I wrote this. Means we'd only need to change our > process for future hotfixes and keep everything else as-is. > >> On Tue, Feb 15, 2022, at 10:55 AM, Brandon Williams wrote: >> On Tue, Feb 15, 2022 at 9:53 AM Josh McKenzie wrote: >> > >> > The only way I'd be in favor of a release that removes all other committed >> > patches >> > >> > Couldn't we just have a snapshot branch for each supported major/minor >> > release branch that we patch for hotfixes and we bump up whenever we have >> > a GA on a parent branch? >> >> I think you could just checkout the tag of the previous release into a >> new branch and apply the fix to it. >> >> Kind Regards, >> Brandon >> >
Re: [DISCUSS] Hotfix release procedure
Was thinking that too after I wrote this. Means we'd only need to change our process for future hotfixes and keep everything else as-is. On Tue, Feb 15, 2022, at 10:55 AM, Brandon Williams wrote: > On Tue, Feb 15, 2022 at 9:53 AM Josh McKenzie wrote: > > > > The only way I'd be in favor of a release that removes all other committed > > patches > > > > Couldn't we just have a snapshot branch for each supported major/minor > > release branch that we patch for hotfixes and we bump up whenever we have a > > GA on a parent branch? > > I think you could just checkout the tag of the previous release into a > new branch and apply the fix to it. > > Kind Regards, > Brandon >
Re: [DISCUSS] Hotfix release procedure
On Tue, Feb 15, 2022 at 9:53 AM Josh McKenzie wrote: > > The only way I'd be in favor of a release that removes all other committed > patches > > Couldn't we just have a snapshot branch for each supported major/minor > release branch that we patch for hotfixes and we bump up whenever we have a > GA on a parent branch? I think you could just checkout the tag of the previous release into a new branch and apply the fix to it. Kind Regards, Brandon
Re: [DISCUSS] Hotfix release procedure
+1 (hotfix with only security fix, private vote, private branch & private ci) On Tue, Feb 15, 2022 at 02:18:42PM +, bened...@apache.org wrote: > One issue with this approach is that we are advertising that we are preparing > a security release by preparing such a release candidate. > > I wonder if we need to find a way to produce binaries without leaving an > obvious public mark (i.e. private CI, private branch) > > > From: Josh McKenzie > Date: Tuesday, 15 February 2022 at 14:09 > To: dev@cassandra.apache.org > Subject: [DISCUSS] Hotfix release procedure > On the release thread for 4.0.2 Jeremiah brought up a point about hotfix > releases and CI: > https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr > > If we are making this release for a security incident/data loss/hot fix > reason, then I would expect to see the related change set only containing > those patches. But the change set in the tag here the latest 4.0-dev commits. > > I'd like to propose that in the future, regardless of the state of CI, if we > need to cut a hotfix release we do so from the previous released SHA + only > the changes required to address the hotfix to minimally impact our end users > and provide them with as minimally disruptive a fix as possible.
Re: [DISCUSS] Hotfix release procedure
> The only way I'd be in favor of a release that removes all other committed > patches Couldn't we just have a snapshot branch for each supported major/minor release branch that we patch for hotfixes and we bump up whenever we have a GA on a parent branch? Shouldn't be any extra burden other than having three extra branches to remember exist; wouldn't need to merge to them or patch to them other than a fast-forward on a given GA. On Tue, Feb 15, 2022, at 10:48 AM, Jeff Jirsa wrote: > We've done concurrent releases without security before, and you follow much > closer than others. I feel most people, if they saw all of the changes > reverted and a release of a single fix, would either instantly know it's > security (high confidence pointer to exactly which patch) OR assume someone > botched the release prep and draw attention to it. So we're trading "someone > who's very involved has a high confidence it's security but has to dig > through 30 patches to find it" vs "everyone knows exactly what's going on", > the former seems better > > The only way I'd be in favor of a release that removes all other committed > patches is if the vote happened in private@ . I don't see any prohibition on > that in https://www.apache.org/foundation/voting.html#ReleaseVotes , so that > seems like an alternate, easy workaround to privacy. > > > > On Tue, Feb 15, 2022 at 7:42 AM J. D. Jordan > wrote: >> >> We already advertise that we are preparing a security release when ever we >> release all of our patch versions at the same time. So I don’t think there >> is an issue there. >> I was not involved in any PMC discussions and had no knowledge of the CVE, >> but when three branches got release votes at the same moment I knew one of >> the final couple patches that was on all three must be an un-announced CVE. >> It is especially more obvious when said patches mention JIRA ticket numbers >> with no information in the ticket. Nobody is being sneaky here as long as >> the vote and code are in the open. >> >>> On Feb 15, 2022, at 9:15 AM, bened...@apache.org wrote: >>> >>> One issue with this approach is that we are advertising that we are >>> preparing a security release by preparing such a release candidate. >>> __ __ >>> I wonder if we need to find a way to produce binaries without leaving an >>> obvious public mark (i.e. private CI, private branch) >>> __ __ >>> __ __ >>> *From: *Josh McKenzie >>> *Date: *Tuesday, 15 February 2022 at 14:09 >>> *To: *dev@cassandra.apache.org >>> *Subject: *[DISCUSS] Hotfix release procedure >>> On the release thread for 4.0.2 Jeremiah brought up a point about hotfix >>> releases and CI: >>> https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr >>> __ __ If we are making this release for a security incident/data loss/hot fix reason, then I would expect to see the related change set only containing those patches. But the change set in the tag here the latest 4.0-dev commits. >>> __ __ >>> I'd like to propose that in the future, regardless of the state of CI, if >>> we need to cut a hotfix release we do so from the previous released SHA + >>> only the changes required to address the hotfix to minimally impact our end >>> users and provide them with as minimally disruptive a fix as possible.
Re: [DISCUSS] Hotfix release procedure
We've done concurrent releases without security before, and you follow much closer than others. I feel most people, if they saw all of the changes reverted and a release of a single fix, would either instantly know it's security (high confidence pointer to exactly which patch) OR assume someone botched the release prep and draw attention to it. So we're trading "someone who's very involved has a high confidence it's security but has to dig through 30 patches to find it" vs "everyone knows exactly what's going on", the former seems better The only way I'd be in favor of a release that removes all other committed patches is if the vote happened in private@ . I don't see any prohibition on that in https://www.apache.org/foundation/voting.html#ReleaseVotes , so that seems like an alternate, easy workaround to privacy. On Tue, Feb 15, 2022 at 7:42 AM J. D. Jordan wrote: > We already advertise that we are preparing a security release when ever we > release all of our patch versions at the same time. So I don’t think there > is an issue there. > I was not involved in any PMC discussions and had no knowledge of the CVE, > but when three branches got release votes at the same moment I knew one of > the final couple patches that was on all three must be an un-announced CVE. > It is especially more obvious when said patches mention JIRA ticket numbers > with no information in the ticket. Nobody is being sneaky here as long as > the vote and code are in the open. > > On Feb 15, 2022, at 9:15 AM, bened...@apache.org wrote: > > > > One issue with this approach is that we are advertising that we are > preparing a security release by preparing such a release candidate. > > > > I wonder if we need to find a way to produce binaries without leaving an > obvious public mark (i.e. private CI, private branch) > > > > > > *From: *Josh McKenzie > *Date: *Tuesday, 15 February 2022 at 14:09 > *To: *dev@cassandra.apache.org > *Subject: *[DISCUSS] Hotfix release procedure > > On the release thread for 4.0.2 Jeremiah brought up a point about hotfix > releases and CI: > > https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr > > > > If we are making this release for a security incident/data loss/hot fix > reason, then I would expect to see the related change set only containing > those patches. But the change set in the tag here the latest 4.0-dev > commits. > > > > I'd like to propose that in the future, regardless of the state of CI, if > we need to cut a hotfix release we do so from the previous released SHA + > only the changes required to address the hotfix to minimally impact our end > users and provide them with as minimally disruptive a fix as possible. > >
Re: [DISCUSS] Hotfix release procedure
We already advertise that we are preparing a security release when ever we release all of our patch versions at the same time. So I don’t think there is an issue there. I was not involved in any PMC discussions and had no knowledge of the CVE, but when three branches got release votes at the same moment I knew one of the final couple patches that was on all three must be an un-announced CVE. It is especially more obvious when said patches mention JIRA ticket numbers with no information in the ticket. Nobody is being sneaky here as long as the vote and code are in the open. > On Feb 15, 2022, at 9:15 AM, bened...@apache.org wrote: > > > One issue with this approach is that we are advertising that we are preparing > a security release by preparing such a release candidate. > > I wonder if we need to find a way to produce binaries without leaving an > obvious public mark (i.e. private CI, private branch) > > > From: Josh McKenzie > Date: Tuesday, 15 February 2022 at 14:09 > To: dev@cassandra.apache.org > Subject: [DISCUSS] Hotfix release procedure > > On the release thread for 4.0.2 Jeremiah brought up a point about hotfix > releases and CI: > https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr > > If we are making this release for a security incident/data loss/hot fix > reason, then I would expect to see the related change set only containing > those patches. But the change set in the tag here the latest 4.0-dev commits. > > I'd like to propose that in the future, regardless of the state of CI, if we > need to cut a hotfix release we do so from the previous released SHA + only > the changes required to address the hotfix to minimally impact our end users > and provide them with as minimally disruptive a fix as possible.
Re: [DISCUSS] Hotfix release procedure
One issue with this approach is that we are advertising that we are preparing a security release by preparing such a release candidate. I wonder if we need to find a way to produce binaries without leaving an obvious public mark (i.e. private CI, private branch) From: Josh McKenzie Date: Tuesday, 15 February 2022 at 14:09 To: dev@cassandra.apache.org Subject: [DISCUSS] Hotfix release procedure On the release thread for 4.0.2 Jeremiah brought up a point about hotfix releases and CI: https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr If we are making this release for a security incident/data loss/hot fix reason, then I would expect to see the related change set only containing those patches. But the change set in the tag here the latest 4.0-dev commits. I'd like to propose that in the future, regardless of the state of CI, if we need to cut a hotfix release we do so from the previous released SHA + only the changes required to address the hotfix to minimally impact our end users and provide them with as minimally disruptive a fix as possible.
Re: [DISCUSS] Hotfix release procedure
+1 On Tue, Feb 15, 2022 at 8:09 AM Josh McKenzie wrote: > > On the release thread for 4.0.2 Jeremiah brought up a point about hotfix > releases and CI: > https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr > > If we are making this release for a security incident/data loss/hot fix > reason, then I would expect to see the related change set only containing > those patches. But the change set in the tag here the latest 4.0-dev commits. > > > I'd like to propose that in the future, regardless of the state of CI, if we > need to cut a hotfix release we do so from the previous released SHA + only > the changes required to address the hotfix to minimally impact our end users > and provide them with as minimally disruptive a fix as possible.
Re: [DISCUSS] Hotfix release procedure
+1. If we want to take our release quality seriously then I think this would be a great policy to have. > On Feb 15, 2022, at 8:09 AM, Josh McKenzie wrote: > > > On the release thread for 4.0.2 Jeremiah brought up a point about hotfix > releases and CI: > https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr > >> If we are making this release for a security incident/data loss/hot fix >> reason, then I would expect to see the related change set only containing >> those patches. But the change set in the tag here the latest 4.0-dev commits. > > I'd like to propose that in the future, regardless of the state of CI, if we > need to cut a hotfix release we do so from the previous released SHA + only > the changes required to address the hotfix to minimally impact our end users > and provide them with as minimally disruptive a fix as possible.
