Hi Steve, On Mon, 10 Apr 2023 at 06:54, Steve Langasek <steve.langa...@ubuntu.com> wrote:
> On Thu, Apr 06, 2023 at 12:31:18PM +0100, Phil Roche wrote: > > Hi all, > > > Bumping this for visibility again. > > > Are there any objections to formally considering Ubuntu cloud images for > > no/no go decisions on release day? > > > It is very unlikely even to be an issue, but should we be unfortunate > > enough to have another kernel CVE land on > > release day, then at least we have it agreed that the cloud image use > cases > > should be considered when deciding > > to release or not. > > I welcome the idea that the cloud images would be always released in > lockstep with the desktop and server images. > > The decision not to release the cloud images the day scheduled for release > was one taken by the CPC team. That's also fine IMHO, but if the goal is > to > ensure they all release at the same time, and that means delaying all the > images, that has a significant impact across the organization. You say > that > it's unlikely to be an issue again, but what process improvements could we > be putting in place to guard against a repeat, and ensure that all the > images are actually ready to be released on the scheduled day? > I have searched through the conversation history today, and the summary of what happened on 22.10 release WRT CVE-2022-2602 <https://ubuntu.com/security/CVE-2022-2602> is as follows: * https://ubuntu.com/security/CVE-2022-2602 was announced and made public on 18th October * There was confusion around whether publishing a kernel to -updates|-security for an unreleased release during freeze was allowed and the decision was made not to and to wait until the archive was open again for uploads * October 20 - release day. Late evening UTC, the release team give +1 on publishing the kernels * As kernels appear to be forthcoming CPC decide to wait for new kernels as we did not and do not wish to publish any cloud images with known and public high or critical priority vulnerabilities * Due to kernel publication infrastructure issues the kernels did not complete publication until October 22nd. So to avoid this specific issue happening again, we could state that in circumstances like these kernels that are addressing a CVE can be published before release regardless of freeze state. That would mean that for CVE-2022-2602 all kernels would have been published and server, desktop and cloud could release at the same time. You mention a kernel CVE; I don't remember the details, but it evidently > wasn't considered a reason to hold back and respin all of the installer > images. Why was it necessary to hold the cloud images back? For cloud > images in particular, the next image is not far away. For cloud, CPC's stance, and one which the cloud partners will hold us to, is that we do not knowingly publish any release, as opposed to a daily, cloud image with a high or critical priority CVE present. CPC spoke about this again today in Google Meet and even if there were no assurances that the updated kernel was forthcoming, we would have made the same decision. Another factor for such a decision is that for our public cloud marketplace offering images with publicly known CVEs present will fail security scanning and publication will be blocked by the cloud provider. The key difference between CPC's stance on CVEs and server and desktop is that CPC do not wish to publish any release cloud image with a high or critical priority CVE present. We have learned that the use case for cloud images is different as users are more likely grab the latest image and redeploy rather than update and reboot for a kernel CVE, so if there is no updated image then they are left with a VM vulnerable to a known public CVE. Also note that there are some clouds where the publication can take over a day leaving all those deployments vulnerable until publication is complete. If that means that the story changes from server, desktop and cloud will release in lockstep on release day to one where server, desktop and cloud will release in lockstep on release day *. * exceptions to this are where a high or critical priority CVE becomes > public before or on release day. If this were to occur then the server, > desktop and cloud images might release independently while waiting for > updated builds to address the CVE. Phil > > > On Thu, 23 Mar 2023 at 14:50, Phil Roche <phil.ro...@canonical.com> > wrote: > > > > > Hi all, > > > > > > I work on the Canonical Public Cloud (CPC) team responsible for the > build > > > and publication of all the Ubuntu cloud images > > > <http://cloud-images.ubuntu.com/> and all their supported derivatives > in > > > the major public and private clouds. > > > > > > As 23.04 release day fast approaches, I would like to start a new > thread > > > on CPC's involvement in release day decisions. > > > > > > Reflecting on the last Ubuntu 22.10 release, from a cloud image > > > perspective, it did not go very well and we were a few days behind the > main > > > desktop/server release, finally releasing on October 22nd instead of > > > October 20th. This was due to the decision by CPC to wait for the high > > > priority CVE https://ubuntu.com/security/CVE-2022-2602 changes to > land in > > > the Kinetic kernel. > > > > > > The use cases for cloud images are not the same as for server and > desktop > > > and releasing with a vulnerable kernel did not make sense even if we > knew > > > an updated kernel that people could upgrade to was forthcoming. > > > > > > The current release process is centered on ISOs with cloud images being > > > downstream but I feel that given Ubuntu cloud images’ usage a situation > > > like the above with CVE-2022-2602 should have warranted a no-go > decision. > > > > > > What are the release teams' thoughts on CPC team being more involved in > > > the no/go decision process on release day? I recognise that release > team > > > member Utkarsh Gupta is an engineer on the CPC team but his > involvement in > > > the release team is not with cloud images specifically. > > > > > > Thank you for all that you do, > > > > > > Phil > > > > > > > > > -- > > > Phil Roche > > > Staff Software Engineer > > > Canonical Public Cloud > > > > > > > > > -- > > Phil Roche > > Staff Software Engineer > > Canonical Public Cloud > > > -- > > Ubuntu-release mailing list > > Ubuntu-release@lists.ubuntu.com > > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release > > > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developer https://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org > -- Phil Roche Staff Software Engineer Canonical Public Cloud
-- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
