Re: [Xen-devel] Design session report: Xen on Distros
On 17/07/2019 13:37, Jan Beulich wrote: > On 17.07.2019 12:33, George Dunlap wrote: >>> On Jul 16, 2019, at 11:03 PM, Andrew Cooper >>> >>> We could trivially throw the fixes into the branch, tag it, sign it and >>> throw it out into the open, but doing that on the embargo date itself >>> would result in an official release of Xen which has had 0 testing in >>> the incumbent test system. >> The point is that anyone who ships / deploys a fix on the disclosure date >> is pretty much shipping exactly that. If it’s not good enough to sign, >> why is it good enough to deploy? > I think the security fixes themselves are good enough to deploy, perhaps > with the assumption that consumers of our pre-disclosure list have done > some testing on their own. The stable trees, however, contain more than > just security fixes, and can have regressions (most likely due to > backporting mistakes). Right, but e.g. proposed changing the commit/push model whereby all changes must pass CI before they get merged would reduce the likelyhood of bad backports getting into staging-* in the first place. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 17.07.2019 12:33, George Dunlap wrote: > >> On Jul 16, 2019, at 11:03 PM, Andrew Cooper >> >> We could trivially throw the fixes into the branch, tag it, sign it and >> throw it out into the open, but doing that on the embargo date itself >> would result in an official release of Xen which has had 0 testing in >> the incumbent test system. > > The point is that anyone who ships / deploys a fix on the disclosure date > is pretty much shipping exactly that. If it’s not good enough to sign, > why is it good enough to deploy? I think the security fixes themselves are good enough to deploy, perhaps with the assumption that consumers of our pre-disclosure list have done some testing on their own. The stable trees, however, contain more than just security fixes, and can have regressions (most likely due to backporting mistakes). Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
> On Jul 16, 2019, at 11:03 PM, Andrew Cooper > > We could trivially throw the fixes into the branch, tag it, sign it and > throw it out into the open, but doing that on the embargo date itself > would result in an official release of Xen which has had 0 testing in > the incumbent test system. The point is that anyone who ships / deploys a fix on the disclosure date is pretty much shipping exactly that. If it’s not good enough to sign, why is it good enough to deploy? Probably the best answer is that it’s _not_ really good enough to deploy; and so it’s worth thinking about how we can improve that, perhaps by having embargoed osstest runs or something. > What a number of people want is for the patches in an XSA advisory to > apply cleanly to whatever random thing they have in their local/distro > patch queue. This is entirely impossible for the security to arrange, > and furthermore, we have exactly one location where the patches we > produce will be committed. I’m not sure people want to apply “wherever”; it’s more that there wasn’t a clear place that they *could* apply. Without any prior knowledge, I think the most natural expectation would be that you could take the most recent point release and just stack on all the outstanding XSAs since then. There are reasons why we don’t do that, but I wouldn’t expect users to guess that. This is the sort of area where maybe a document in your sphinx system would be helpful, just to lay out the issues. > As a personal view here, I don't think blindly taking the latest > staging-$X.$Y is a viable substitute for at least understanding the > patches well enough to work around trivial offsets/fuzz due to minor > variations. I think this is fine for downstreams that have full-fledged hypervisor development teams (like XenServer or Amazon), but is a bit too high a barrier for "mom-and-pop" cloud providers or community-maintained distributions. -George ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
Hi, On 7/15/19 4:42 PM, George Dunlap wrote: > On 7/15/19 3:23 PM, Jan Beulich wrote: >> On 15.07.2019 16:11, George Dunlap wrote: >>> There was a long discussion about security patches, with the general >>> proposal being that we should cut a point release for every security issue. >> >> Interesting. Looks like in politics that until a decision fits people >> they keep re-raising the point. Iirc on a prior meeting (Budapest?) >> we had settled on continuing with the current scheme. Were there any >> new arguments towards this alternative model? > > Well I don't know if there were any new arguments because I don't > immediately remember the old discussion. Do we have a summary of the > discussion in Budapest, with its conclusions, anywhere? > > The basic idea was that: > > 1. Most distros / packagers are going to want to do an immediate release > anyway. > > 2. Distros generally seemed to be rebasing on top of staging as soon as > the XSA went out anyway (and ISTR this being the recommeneded course of > action) FYI, In Debian, we only ship the stable branch, not the staging branch. Better safe than sorry. And yes, this means there's a delay, and it's not immediate, etc. Well, at least since I have been involved. In the past security update packages were made by applying patches manually on top of older (like, 4.4.1 instead of 4.4.x) frozen versions, but we decided that "trying to assemble our own subset of the patches is riskier than taking upstream's collection". The Debian Release Team allows us to upload newer upstream versions during a security upload for Xen, which is officially not allowed in Debian stable. One of the things that obviously help with being able to keep doing this is the "track record" of the quality (expecially regression testing) of the stable-4.x branches. Currently, our package versions mostly look like e.g. 4.11.1+92-g6c33308a8d-1, instead of a nice simple version number. For me this is fine, but I can understand that for an end user it would just look nicer (and feel better perception-wise) to get a 4.11.x package. So just for that reason I would be all +1 for more tagged releases. > So for all intents and purposes, we have something which is, in fact, a > release; all it's missing is a signed tag and a tarball. > > Obviously there are testing implications that would need to be sorted > out before this could become a reality. > > In any case, the ball is in the court of "VOLUNTEER" to write up a > concrete proposal which could be discussed. You'll be able to raise all > your concerns at that point if you want (although having a sketch would > of course be helpful for whoever is writing such a proposal). Hans ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 15/07/2019 18:52, Amit Shah wrote: > On Mon, 2019-07-15 at 14:52 +, Jan Beulich wrote: >> On 15.07.2019 16:42, George Dunlap wrote: >>> On 7/15/19 3:23 PM, Jan Beulich wrote: On 15.07.2019 16:11, George Dunlap wrote: > There was a long discussion about security patches, with the > general > proposal being that we should cut a point release for every > security issue. Interesting. Looks like in politics that until a decision fits people they keep re-raising the point. Iirc on a prior meeting (Budapest?) we had settled on continuing with the current scheme. Were there any new arguments towards this alternative model? >>> Well I don't know if there were any new arguments because I don't >>> immediately remember the old discussion. Do we have a summary of >>> the >>> discussion in Budapest, with its conclusions, anywhere? >> I don't recall if suitable notes were taken back then; as indicated >> I'm not even sure which meeting it was at. >> >>> The basic idea was that: >>> >>> 1. Most distros / packagers are going to want to do an immediate >>> release >>> anyway. >>> >>> 2. Distros generally seemed to be rebasing on top of staging as >>> soon as >>> the XSA went out anyway (and ISTR this being the recommeneded >>> course of >>> action) >>> >>> So for all intents and purposes, we have something which is, in >>> fact, a >>> release; all it's missing is a signed tag and a tarball. >>> >>> Obviously there are testing implications that would need to be >>> sorted >>> out before this could become a reality. >>> >>> In any case, the ball is in the court of "VOLUNTEER" to write up a >>> concrete proposal which could be discussed. You'll be able to >>> raise all >>> your concerns at that point if you want (although having a sketch >>> would >>> of course be helpful for whoever is writing such a proposal). >> Sure - I realized soon after having sent the initial reply that >> perhaps >> this was the wrong context in the first place to raise my question. > In any case, I'd like to know why it doesn't make sense for Xen to have > a point release frequently, and not have a point release after an XSA > above some severity level (pick one - high/critical/important). We specifically do not rate XSAs like that. One persons apple is a different persons orange, considering how varied the environments which Xen runs in are. If in doubt, people should take all the XSA fixes. > As George mentioned, distros have to do it anyway, and the upstream > project not doing it only makes it more difficult for all distros > involved. > > Not sure of the politics involved though, and what can of worms this > opens. Its a perfectly valid discussion to have, and comes down (in large part) to the overhead of doing releases, which comes largely from other areas of our infrastructure which are currently under question. A full point release includes tagging a load of repos (2x qemu, seabios, ovmf iirc), then waiting for generally 2-3 weeks for OSSTest to stabilise enough to declare the patches good, even for ones which were tested (elsewhere) to various downstream's satisfaction during the embargo period. We could trivially throw the fixes into the branch, tag it, sign it and throw it out into the open, but doing that on the embargo date itself would result in an official release of Xen which has had 0 testing in the incumbent test system. One aspect which would ease things is to not include 3rd party packages in releases. This was discussed as part of the build system gripes session iirc, and would reduce the number of in-flight moving parts to just xen.git. Another aspect is that of the test system. At the moment, nothing can be released on the older security branches, because they still haven't recovered from the carnage of the Debian upgrade. Irrespective of that particular issue, the general delay between patches appearing and OSSTest saying yes is a concerning factor in how much effort it takes to get a release out. What a number of people want is for the patches in an XSA advisory to apply cleanly to whatever random thing they have in their local/distro patch queue. This is entirely impossible for the security to arrange, and furthermore, we have exactly one location where the patches we produce will be committed. As a personal view here, I don't think blindly taking the latest staging-$X.$Y is a viable substitute for at least understanding the patches well enough to work around trivial offsets/fuzz due to minor variations. However, if the overhead of doing a micro release became substantially less, and producing a micro release after every XSA would help a decent chunk of our downstreams consume security fixes more easily, then it would be worth considering. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On Mon, 2019-07-15 at 14:52 +, Jan Beulich wrote: > On 15.07.2019 16:42, George Dunlap wrote: > > On 7/15/19 3:23 PM, Jan Beulich wrote: > > > On 15.07.2019 16:11, George Dunlap wrote: > > > > There was a long discussion about security patches, with the > > > > general > > > > proposal being that we should cut a point release for every > > > > security issue. > > > > > > Interesting. Looks like in politics that until a decision fits > > > people > > > they keep re-raising the point. Iirc on a prior meeting > > > (Budapest?) > > > we had settled on continuing with the current scheme. Were there > > > any > > > new arguments towards this alternative model? > > > > Well I don't know if there were any new arguments because I don't > > immediately remember the old discussion. Do we have a summary of > > the > > discussion in Budapest, with its conclusions, anywhere? > > I don't recall if suitable notes were taken back then; as indicated > I'm not even sure which meeting it was at. > > > The basic idea was that: > > > > 1. Most distros / packagers are going to want to do an immediate > > release > > anyway. > > > > 2. Distros generally seemed to be rebasing on top of staging as > > soon as > > the XSA went out anyway (and ISTR this being the recommeneded > > course of > > action) > > > > So for all intents and purposes, we have something which is, in > > fact, a > > release; all it's missing is a signed tag and a tarball. > > > > Obviously there are testing implications that would need to be > > sorted > > out before this could become a reality. > > > > In any case, the ball is in the court of "VOLUNTEER" to write up a > > concrete proposal which could be discussed. You'll be able to > > raise all > > your concerns at that point if you want (although having a sketch > > would > > of course be helpful for whoever is writing such a proposal). > > Sure - I realized soon after having sent the initial reply that > perhaps > this was the wrong context in the first place to raise my question. In any case, I'd like to know why it doesn't make sense for Xen to have a point release frequently, and not have a point release after an XSA above some severity level (pick one - high/critical/important). As George mentioned, distros have to do it anyway, and the upstream project not doing it only makes it more difficult for all distros involved. Not sure of the politics involved though, and what can of worms this opens. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 15.07.2019 16:42, George Dunlap wrote: > On 7/15/19 3:23 PM, Jan Beulich wrote: >> On 15.07.2019 16:11, George Dunlap wrote: >>> There was a long discussion about security patches, with the general >>> proposal being that we should cut a point release for every security issue. >> >> Interesting. Looks like in politics that until a decision fits people >> they keep re-raising the point. Iirc on a prior meeting (Budapest?) >> we had settled on continuing with the current scheme. Were there any >> new arguments towards this alternative model? > > Well I don't know if there were any new arguments because I don't > immediately remember the old discussion. Do we have a summary of the > discussion in Budapest, with its conclusions, anywhere? I don't recall if suitable notes were taken back then; as indicated I'm not even sure which meeting it was at. > The basic idea was that: > > 1. Most distros / packagers are going to want to do an immediate release > anyway. > > 2. Distros generally seemed to be rebasing on top of staging as soon as > the XSA went out anyway (and ISTR this being the recommeneded course of > action) > > So for all intents and purposes, we have something which is, in fact, a > release; all it's missing is a signed tag and a tarball. > > Obviously there are testing implications that would need to be sorted > out before this could become a reality. > > In any case, the ball is in the court of "VOLUNTEER" to write up a > concrete proposal which could be discussed. You'll be able to raise all > your concerns at that point if you want (although having a sketch would > of course be helpful for whoever is writing such a proposal). Sure - I realized soon after having sent the initial reply that perhaps this was the wrong context in the first place to raise my question. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 7/15/19 3:42 PM, George Dunlap wrote: > On 7/15/19 3:23 PM, Jan Beulich wrote: >> On 15.07.2019 16:11, George Dunlap wrote: >>> There was a long discussion about security patches, with the general >>> proposal being that we should cut a point release for every security issue. >> >> Interesting. Looks like in politics that until a decision fits people >> they keep re-raising the point. Iirc on a prior meeting (Budapest?) >> we had settled on continuing with the current scheme. Were there any >> new arguments towards this alternative model? If we end up deciding against doing this again, it might be worth creating a ticket somewhere to try to capture the arguments on each side, so that we don't end up re-hashing the same issues in another few years. -George ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 7/15/19 3:23 PM, Jan Beulich wrote: > On 15.07.2019 16:11, George Dunlap wrote: >> There was a long discussion about security patches, with the general >> proposal being that we should cut a point release for every security issue. > > Interesting. Looks like in politics that until a decision fits people > they keep re-raising the point. Iirc on a prior meeting (Budapest?) > we had settled on continuing with the current scheme. Were there any > new arguments towards this alternative model? Well I don't know if there were any new arguments because I don't immediately remember the old discussion. Do we have a summary of the discussion in Budapest, with its conclusions, anywhere? The basic idea was that: 1. Most distros / packagers are going to want to do an immediate release anyway. 2. Distros generally seemed to be rebasing on top of staging as soon as the XSA went out anyway (and ISTR this being the recommeneded course of action) So for all intents and purposes, we have something which is, in fact, a release; all it's missing is a signed tag and a tarball. Obviously there are testing implications that would need to be sorted out before this could become a reality. In any case, the ball is in the court of "VOLUNTEER" to write up a concrete proposal which could be discussed. You'll be able to raise all your concerns at that point if you want (although having a sketch would of course be helpful for whoever is writing such a proposal). -George ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Design session report: Xen on Distros
On 15.07.2019 16:11, George Dunlap wrote: > There was a long discussion about security patches, with the general > proposal being that we should cut a point release for every security issue. Interesting. Looks like in politics that until a decision fits people they keep re-raising the point. Iirc on a prior meeting (Budapest?) we had settled on continuing with the current scheme. Were there any new arguments towards this alternative model? Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
[Xen-devel] Design session report: Xen on Distros
Much of this covered things discussed elsewhere: * Allowing multiple versions of the tools to be installed at the same time * Getting rid of external builds There was a long discussion about security patches, with the general proposal being that we should cut a point release for every security issue. One random thing was that xenstored apparently has an 'in-memory-only' option. Since xenstored can't actually be restarted ATM, and most distros seemed to put xenstored in a tmpfs for performance reasons, this should probably be the default. https://hackmd.io/vmacVBYbQiORJ9H4_a9Ivw # Xen on Distros design session * qemu / libxc dependency loop * build system needs "extras" turned off * xenstored / tmpfs / memory-only option? * Disabling auto-download in build system * WGET=/bin/false * Multiple version of Xen / tools? * Debian has co-install * Change some installation paths * /usr/lib/xen/4.11/... * /usr/bin/xl is a shell script * libfsimage is special * Don't need to downgrade to older tools * Gentoo has a ~~dumpster fire~~ something * A hack which stops the package manager to allow you to reboot the box halfway through * Security issues * Building from stable branch / staging branch * Doing a "point release" every XSA? * "Release from staging" is effectively a low-quality release * Idea: Always immediately release from staging? # Actions * [ ] Ian: Post a git branch of Debian co-install to xen-devel * [ ] George: Post systemd / selinux / xenstored patch * [ ] George, Ian: private osstest runs * [ ] VOLUNTEER: Propose / argue for a point release per XSA * [ ] VOLUNTEER: Improve release automation ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel