Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On 08/29/2017 03:44 PM, Konrad Rzeszutek Wilk wrote: > .giant snip.. >> Indeed; and as I think I said before, I think we need to move forward >> with getting a statement on livepatching in, and since most of the >> voices involved in this conversation seem to be in favor of saying >> livepatch-tools are *not* supported, I won't object. I'm only still > > Thank you. > > As such, here is the patch. Would folks like me to repost it, or > OK with Acking/Reviewing it as such? > > I think the point 3) succinctly explains the position that has been so hotly > debated. I can of course expand it, but not sure if it makes sense? I'd prefer to remove a justification I believe to be unreasonable, and just say what the support status is: > +3) Bugs in livepatch-build-tools creating an incorrect live patch that > + results in an insecure host: > + If livepatch-build-tools creates an incorrect live patch that > + results in an insecure host, this shall not be considered a security > + issue. A live patch should be checked to verify that it is valid > + before loading. Is that OK with everyone? -George ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On 29/08/17 15:44, Konrad Rzeszutek Wilk wrote: > .giant snip.. >> Indeed; and as I think I said before, I think we need to move forward >> with getting a statement on livepatching in, and since most of the >> voices involved in this conversation seem to be in favor of saying >> livepatch-tools are *not* supported, I won't object. I'm only still > Thank you. > > As such, here is the patch. Would folks like me to repost it, or > OK with Acking/Reviewing it as such? > > I think the point 3) succinctly explains the position that has been so hotly > debated. I can of course expand it, but not sure if it makes sense? > > > > From f968f003d36ac2b9d1b670b34bbe2a1222830138 Mon Sep 17 00:00:00 2001 > From: Ross Lagerwall > Date: Wed, 28 Jun 2017 17:13:44 +0100 > Subject: [PATCH v3] livepatch: Declare live patching as a supported feature > > See docs/features/livepatch.pandoc for the details. > > Signed-off-by: Ross Lagerwall > Signed-off-by: Konrad Rzeszutek Wilk > > -- > v2: > - Moved it into a feature document. > - Clarified a few bits and pieces based on feedback. > v3: > - default X86 > --- > docs/features/livepatch.pandoc | 103 > + > xen/common/Kconfig | 4 +- > 2 files changed, 105 insertions(+), 2 deletions(-) > create mode 100644 docs/features/livepatch.pandoc > > diff --git a/docs/features/livepatch.pandoc b/docs/features/livepatch.pandoc > new file mode 100644 > index 00..faaf2d1e77 > --- /dev/null > +++ b/docs/features/livepatch.pandoc > @@ -0,0 +1,103 @@ > +% Live Patching > +% Revision 1 > + > +\clearpage > + > +# Basics > + > + > + Status: **Supported** > + > + Architecture: x86 Sorry to only be noticing this now. This should read Supported for x86, Tech Preview/Experimental (as appropriate) for ARM. ~Andrew > + > + Component: Hypervisor, toolstack > + ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
.giant snip.. > Indeed; and as I think I said before, I think we need to move forward > with getting a statement on livepatching in, and since most of the > voices involved in this conversation seem to be in favor of saying > livepatch-tools are *not* supported, I won't object. I'm only still Thank you. As such, here is the patch. Would folks like me to repost it, or OK with Acking/Reviewing it as such? I think the point 3) succinctly explains the position that has been so hotly debated. I can of course expand it, but not sure if it makes sense? From f968f003d36ac2b9d1b670b34bbe2a1222830138 Mon Sep 17 00:00:00 2001 From: Ross Lagerwall Date: Wed, 28 Jun 2017 17:13:44 +0100 Subject: [PATCH v3] livepatch: Declare live patching as a supported feature See docs/features/livepatch.pandoc for the details. Signed-off-by: Ross Lagerwall Signed-off-by: Konrad Rzeszutek Wilk -- v2: - Moved it into a feature document. - Clarified a few bits and pieces based on feedback. v3: - default X86 --- docs/features/livepatch.pandoc | 103 + xen/common/Kconfig | 4 +- 2 files changed, 105 insertions(+), 2 deletions(-) create mode 100644 docs/features/livepatch.pandoc diff --git a/docs/features/livepatch.pandoc b/docs/features/livepatch.pandoc new file mode 100644 index 00..faaf2d1e77 --- /dev/null +++ b/docs/features/livepatch.pandoc @@ -0,0 +1,103 @@ +% Live Patching +% Revision 1 + +\clearpage + +# Basics + + + Status: **Supported** + + Architecture: x86 + + Component: Hypervisor, toolstack + + + +# Details + +Xen Live Patching has been available as tech preview feature since Xen +4.7 and has now had a couple of releases to stabilize. Xen Live patching +has been used by multiple vendors to fix several real-world security +issues without any severe bugs encountered. Additionally, there are now +tests in OSSTest that test live patching to ensure that no regressions +are introduced. + +Based on the amount of testing and usage it has had, we are ready to +declare live patching as a 'Supported' feature on x86. + +Live patching is slightly peculiar when it comes to support because it +allows the host administrator to break their system rather easily +depending on the content of the live patch. Because of this, it is +worth detailing the scope of security support: + +1) Unprivileged access to live patching operations: + Live patching operations should only be accessible to privileged + guests and it shall be treated as a security issue if this is not + the case. + +2) Bugs in the patch-application code such that vulnerabilities exist + after application: + If a correct live patch is loaded but it is not applied correctly + such that it might result in an insecure system (e.g. not all + functions are patched), it shall be treated as a security issue. + +3) Bugs in livepatch-build-tools creating an incorrect live patch that + results in an insecure host: + If livepatch-build-tools creates an incorrect live patch that + results in an insecure host, this shall not be considered a security + issue. There are too many OSes and toolchains to consider supporting + this. A live patch should be checked to verify that it is valid + before loading. + +4) Loading an incorrect live patch that results in an insecure host or + host crash: + If a live patch (whether created using livepatch-build-tools or some + alternative) is loaded and it results in an insecure host or host + crash due to the content of the live patch being incorrect or the + issue being inappropriate to live patch, this is not considered as a + security issue. + +5) Bugs in the live patch parsing code (the ELF loader): + Bugs in the live patch parsing code such as out-of-bounds reads + caused by invalid ELF files are not considered to be security issues + because the it can only be triggered by a privileged domain. + +6) Bugs which allow a guest to prevent the application of a livepatch: + A guest should not be able to prevent the application of a live + patch. If an unprivileged guest can somehow prevent the application + of a live patch despite pausing it (xl pause ...), it shall be + treated as a security issue. + +Note: It is expected that live patches are tested in a test environment +before being used in production to avoid unexpected issues. In +particular, to avoid the issues described by (3), (4), & (5). + +There are also some generic security questions which are worth asking: + +1) Is guest->host privilege escalation possible? + +The new live patching sysctl subops are only accessible to privileged +domains and this is tested by OSSTest with an XTF test. +There is a caveat -- an incorrect live patch can introduce a guest->host +privilege escalation. + +2) Is guest user->guest kernel escalati
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On Tue, Aug 22, 2017 at 11:58:57AM +0100, George Dunlap wrote: > I think guest OS support is actually a pretty good analog. I can't > imagine not issuing XSAs for bugs in Linux, just as I can't imagine > not issuing XSAs for actual security issues that get found in the > livepatch tools. If you think we shouldn't give security support for > Linux, it makes sense that you would feel the same way for > livepatch-tools (although I don't really understand why you think that > way about either). > > We issue more XSAs for Linux than for other guests, in part because of > the complexity of the code inside Linux compared to other OSes; but > also in part due to the fact that that is the most tested and > looked-at. There probably *are* more bugs in Linux than in NetBSD or > FreeBSD; but also more of them are found because more people are > testing and looking. IMHO, we issue XSA for Linux because Linux lacks a security process. If a bug was found in the BSDs, it should be handled using the normal security process that each BSD has, and a SA would be issued by the security officer: https://www.freebsd.org/security/advisories.html For example NetBSD has recently released a SA for a Xen-specific PV vulnerability in their implementation: ftp://ftp.nl.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2017-003.txt.asc Roger. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On Tue, Aug 22, 2017 at 7:37 AM, Jan Beulich wrote:
On 21.08.17 at 17:28, wrote:
>> So your argument seems to be:
>>
>> 1. We can only provide security support in situations where we can test
>> all possible combinations in the support matrix.
>>
>> 2. We cannot test the entire matrix of combinations for Xen x livepatch
>> tools x compilers
>>
>> 3. Therefore, we cannot provide security support for livepatching tools.
>>
>> Put this way, I hope you can see what the flaw in the argument is: #1 is
>> false. Xen has {Xen version} x {Linux version} x {Compiler} x
>> {Hardware}. Hardware of course includes not only the chip itself, but
>> the BIOS / firmware, and the particular devices (and device firmware).
>> If we wanted we could add in {Python version} for people using pygrub,
>> and {Ocaml compiler version} for people running Ocaml, versions of
>> systemd -- I'm sure with effort I could find more dimensions to add to
>> the matrix.
>>
>> We do not, and never have, *tested* the entire matrix of possible
>> combinations considered "security supported" to make sure they work.
>> Such a matrix is completely impossible to even consider, and even if we
>> did some sort of testing, that could not guarantee that they are bug free.
>>
>> What we do for security support is:
>>
>> 1. Test a *representative sample* of combinations (via osstest, product
>> testing, user testing, &c)
>>
>> 2. Promise to issue XSAs if anyone *happens to discover* a combination
>> in the rest of the support matrix that has a security issue
>>
>> That is the requirment for normal Xen, and it would be the same
>> requirement for livepatch-tools: That between osstest, product, and the
>> community, we get regular testing of *a representative sample* of {Xen,
>> livepatch-tools, compiler}, and (what primarily concerns me) issue an
>> XSA if anyone discovers a security issue somewhere in that matrix.
>>
>> I'm not frustrated, but I am baffled by the fact that this "support
>> matrix" objection is so persistent. Nearly everyone has brought it up,
>> as though "test every combination" was a necessary requirement, in spite
>> of the fact that 1) there is *no* piece of software for which we test
>> the entire matrix of possible combinations 2) I have said over and over
>> again (in fact, I specifically said a few replies ago -- it's there at
>> the top of this email) that we do not test all possible combinations.
>
> Well, part of it may be that the other components involved in the
> test matrix you suggest are external, i.e. we're just their consumers.
But our response to me, you mentioned distros having patched compilers
as a reason that the matrix is untenably large.
> If we consider just our own portions, the matrix is - as said - one
> dimensional. With the livepatching tools, a dimension is being added.
> Even us issuing Linux XSAs is, with the current upstream status of
> the Xen pieces in there, questionable imo. This is also considering
> the fact that iirc we've never issued an XSA for another Xen guest
> OS, yet it is hard to believe that only Linux would ever had any
> vulnerability.
Well, no -- we have at very least {Linux} x {Xen}, for which we end up
testing *many* different configurations, but certainly not all.
I think guest OS support is actually a pretty good analog. I can't
imagine not issuing XSAs for bugs in Linux, just as I can't imagine
not issuing XSAs for actual security issues that get found in the
livepatch tools. If you think we shouldn't give security support for
Linux, it makes sense that you would feel the same way for
livepatch-tools (although I don't really understand why you think that
way about either).
We issue more XSAs for Linux than for other guests, in part because of
the complexity of the code inside Linux compared to other OSes; but
also in part due to the fact that that is the most tested and
looked-at. There probably *are* more bugs in Linux than in NetBSD or
FreeBSD; but also more of them are found because more people are
testing and looking.
But in any case, in my mind, the promise never was "we test all
versions of Linux with all versions of Xen", much less "We test all
versions of all operating systems with all versions of Xen". The
promise was, "We test a representative sample of Linux and Xen
combinations, and we promise to report issues if they are found."
That's what I thing is the right thing to do for livepatch-tools as
well.
-George
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
>>> On 21.08.17 at 17:28, wrote:
> So your argument seems to be:
>
> 1. We can only provide security support in situations where we can test
> all possible combinations in the support matrix.
>
> 2. We cannot test the entire matrix of combinations for Xen x livepatch
> tools x compilers
>
> 3. Therefore, we cannot provide security support for livepatching tools.
>
> Put this way, I hope you can see what the flaw in the argument is: #1 is
> false. Xen has {Xen version} x {Linux version} x {Compiler} x
> {Hardware}. Hardware of course includes not only the chip itself, but
> the BIOS / firmware, and the particular devices (and device firmware).
> If we wanted we could add in {Python version} for people using pygrub,
> and {Ocaml compiler version} for people running Ocaml, versions of
> systemd -- I'm sure with effort I could find more dimensions to add to
> the matrix.
>
> We do not, and never have, *tested* the entire matrix of possible
> combinations considered "security supported" to make sure they work.
> Such a matrix is completely impossible to even consider, and even if we
> did some sort of testing, that could not guarantee that they are bug free.
>
> What we do for security support is:
>
> 1. Test a *representative sample* of combinations (via osstest, product
> testing, user testing, &c)
>
> 2. Promise to issue XSAs if anyone *happens to discover* a combination
> in the rest of the support matrix that has a security issue
>
> That is the requirment for normal Xen, and it would be the same
> requirement for livepatch-tools: That between osstest, product, and the
> community, we get regular testing of *a representative sample* of {Xen,
> livepatch-tools, compiler}, and (what primarily concerns me) issue an
> XSA if anyone discovers a security issue somewhere in that matrix.
>
> I'm not frustrated, but I am baffled by the fact that this "support
> matrix" objection is so persistent. Nearly everyone has brought it up,
> as though "test every combination" was a necessary requirement, in spite
> of the fact that 1) there is *no* piece of software for which we test
> the entire matrix of possible combinations 2) I have said over and over
> again (in fact, I specifically said a few replies ago -- it's there at
> the top of this email) that we do not test all possible combinations.
Well, part of it may be that the other components involved in the
test matrix you suggest are external, i.e. we're just their consumers.
If we consider just our own portions, the matrix is - as said - one
dimensional. With the livepatching tools, a dimension is being added.
Even us issuing Linux XSAs is, with the current upstream status of
the Xen pieces in there, questionable imo. This is also considering
the fact that iirc we've never issued an XSA for another Xen guest
OS, yet it is hard to believe that only Linux would ever had any
vulnerability.
Jan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On 08/21/2017 01:07 PM, Jan Beulich wrote:
And remember, this is not "We have tested all compiler versions and
promise you there are no bugs." It's, "If someone finds a bug for this
set of compilers, we will tell you about it so you can do something
about it."
>>>
>>> I can see and understand all of what you say; my argument,
>>> however was more towards the matrix of what needs supporting
>>> possibly becoming unreasonably large (no matter whether we
>>> specify a range of compilers, as once again distros tend to not
>>> ship plain unpatched upstream compiler versions).
>>
>> What do you mean, "The matrix of what needs supporting [may possibly
>> become] increasingly large"? What is the problem with having a large
>> (implicit) "supported" matrix? How is supporting a "large matrix" for
>> livepatch tools different than the current "large matrix" we support
>> for just building Xen at all?
>
> The matrix of Xen only has just a single dimension. Since livepatch
> tools and Xen are independent, any pair of them would need
> building/testing in order to be sure things work in all supported
> combinations.
So your argument seems to be:
1. We can only provide security support in situations where we can test
all possible combinations in the support matrix.
2. We cannot test the entire matrix of combinations for Xen x livepatch
tools x compilers
3. Therefore, we cannot provide security support for livepatching tools.
Put this way, I hope you can see what the flaw in the argument is: #1 is
false. Xen has {Xen version} x {Linux version} x {Compiler} x
{Hardware}. Hardware of course includes not only the chip itself, but
the BIOS / firmware, and the particular devices (and device firmware).
If we wanted we could add in {Python version} for people using pygrub,
and {Ocaml compiler version} for people running Ocaml, versions of
systemd -- I'm sure with effort I could find more dimensions to add to
the matrix.
We do not, and never have, *tested* the entire matrix of possible
combinations considered "security supported" to make sure they work.
Such a matrix is completely impossible to even consider, and even if we
did some sort of testing, that could not guarantee that they are bug free.
What we do for security support is:
1. Test a *representative sample* of combinations (via osstest, product
testing, user testing, &c)
2. Promise to issue XSAs if anyone *happens to discover* a combination
in the rest of the support matrix that has a security issue
That is the requirment for normal Xen, and it would be the same
requirement for livepatch-tools: That between osstest, product, and the
community, we get regular testing of *a representative sample* of {Xen,
livepatch-tools, compiler}, and (what primarily concerns me) issue an
XSA if anyone discovers a security issue somewhere in that matrix.
I'm not frustrated, but I am baffled by the fact that this "support
matrix" objection is so persistent. Nearly everyone has brought it up,
as though "test every combination" was a necessary requirement, in spite
of the fact that 1) there is *no* piece of software for which we test
the entire matrix of possible combinations 2) I have said over and over
again (in fact, I specifically said a few replies ago -- it's there at
the top of this email) that we do not test all possible combinations.
>> I have elsewhere described a hypothetical scenario where I think we
>> should issue an XSA for livepatch-tools. Are you really seriously
>> suggesting that in that scenario we should simply publish the
>> vulnerability onto xen-devel with no predisclosure?
>
> Well, at least I'm not 100% convinced issuing an XSA in this case
> would be appropriate.
>
> Anyway - since it feels like we're moving in circles (which in part
> may be because I can't express well enough the reasons for my
> hesitation to go to the full XSA extent with the livepatch tools)
> I'd like to just conclude my part here with saying that I'm not
> going to stand in the way whichever decision is taken. I've
> voiced my reservations, and that will have to do. I'd therefore
> prefer to leave the discussion to those more familiar with those
> tools (and their possible limitations and issues).
Indeed; and as I think I said before, I think we need to move forward
with getting a statement on livepatching in, and since most of the
voices involved in this conversation seem to be in favor of saying
livepatch-tools are *not* supported, I won't object. I'm only still
continuing this thread because people seem to be confused about what I
am asking people to do.
I think the likelihood of an XSA-worthy bug being found in the livepatch
tools is very low. I'm happy to defer the argument about whether we
should issue an XSA for such a bug until such time as one becomes known.
-George
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
>>> On 21.08.17 at 12:59, wrote:
> On Wed, Aug 9, 2017 at 8:36 AM, Jan Beulich wrote:
> On 08.08.17 at 13:16, wrote:
>>> On 08/07/2017 04:59 PM, Jan Beulich wrote:
>>> George Dunlap 08/07/17 12:27 PM >>>
> So it seems that people are still not quite clear about what I'm
> proposing.
And indeed your examples helped me understand better what you mean
(or at least I hope they did).
> Suppose someone builds a livepatch with the correct compiler, with a
> correct patch (that would fix the bug if rebooted into a new
> hypervisor), with correct fix-up code. Suppose that the bug passes all
> reasonable testing; but that, *due to a bug in the tools*, the patch
> also gives PV guests access to hypervisor memory. Is this a security
> issue? Yes -- the human told it to do safe thing X ("build a livepatch
> based on correct inputs to fix this bug") and it did unsafe thing Y
> ("build a livepatch that opens up a new security hole").
There's one more factor here: The livepatch tools may behave properly
with one version of the compiler, and improperly with another.
>>>
>>> I don't really understand the reasoning here. Is this your argument:
>>> "One can imagine a security-critical livepatch bug that only affects
>>> say, gcc 6.x and not gcc 5.x or 7.x. Therefore, we should never issue
>>> XSAs for any security-critical livepatch bugs."
>>>
>>> If we found that livepatching tools make an incorrect patch only when
>>> using gcc 5.x, and we have reason to believe that some people may be
>>> using gcc 5.x, then I think we should issue an XSA and say that it only
>>> affects people compiling xen with gcc 5.x.
>>>
>>> It probably would make sense to specify some range of compiler versions
>>> for which we will issue XSAs for the livepatch tools. A good baseline
>>> would be what versions of gcc Xen uses, and then we can restrict it
>>> further if we need to (for instance, if some versions of gcc are missing
>>> requisite features, or if they are just known to be buggy).
>>>
>>> And remember, this is not "We have tested all compiler versions and
>>> promise you there are no bugs." It's, "If someone finds a bug for this
>>> set of compilers, we will tell you about it so you can do something
>>> about it."
>>
>> I can see and understand all of what you say; my argument,
>> however was more towards the matrix of what needs supporting
>> possibly becoming unreasonably large (no matter whether we
>> specify a range of compilers, as once again distros tend to not
>> ship plain unpatched upstream compiler versions).
>
> What do you mean, "The matrix of what needs supporting [may possibly
> become] increasingly large"? What is the problem with having a large
> (implicit) "supported" matrix? How is supporting a "large matrix" for
> livepatch tools different than the current "large matrix" we support
> for just building Xen at all?
The matrix of Xen only has just a single dimension. Since livepatch
tools and Xen are independent, any pair of them would need
building/testing in order to be sure things work in all supported
combinations.
>>> Suppose instead we issued an XSA with a patch, and that it was later
>>> discovered that the patch opened up a different security hole when
>>> applied on top of XenServer's patchqueue, but not on the baseline
>>> XenProject. Would we issue another XSA and/or an update to an existing XSA?
>>>
>>> The obvious *default* answer to that is "No; it's not practical for us
>>> to deal with software that is not inside the XenProject's control." One
>>> could imagine circumstances in which we issue statements or an XSA
>>> anyway, but that would the exception and not the rule.
>>>
>>> I think the same kind of thing would apply to the livepatch tools: *by
>>> default*, we only issue XSAs for the livepatch tools if they create
>>> security issues when generating blobs based on security patches issued
>>> by the XenProject, and on top of XenProject-released software. As
>>> always, if there's some unforeseen circumstance then someone could argue
>>> for an exception.
>>
>> Not sure here - if analysis showed that the same issue could happen
>> elsewhere, and others were just lucky so far, I think we'd have to
>> alter the default (and I'm hesitant to call this an exception). Plus
>> analysis may, the more different components are involved
>> (specifically the compiler, which perhaps none of us has deep enough
>> knowledge about), become more and more difficult.
>>
>> Bottom line - while technically I agree it would be good for the tools
>> to be security supported, from a practical perspective I see too
>> much complexity for this to be reasonably manageable.
>
> But I still don't understand why you think so. Every single objection
> or question about what would or would not be supported that has been
> raised so far has analogs in what we already support. It is no more
> complex to support livepatch-tools than
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On Wed, Aug 9, 2017 at 8:36 AM, Jan Beulich wrote:
On 08.08.17 at 13:16, wrote:
>> On 08/07/2017 04:59 PM, Jan Beulich wrote:
>> George Dunlap 08/07/17 12:27 PM >>>
So it seems that people are still not quite clear about what I'm proposing.
>>>
>>> And indeed your examples helped me understand better what you mean
>>> (or at least I hope they did).
>>>
Suppose someone builds a livepatch with the correct compiler, with a
correct patch (that would fix the bug if rebooted into a new
hypervisor), with correct fix-up code. Suppose that the bug passes all
reasonable testing; but that, *due to a bug in the tools*, the patch
also gives PV guests access to hypervisor memory. Is this a security
issue? Yes -- the human told it to do safe thing X ("build a livepatch
based on correct inputs to fix this bug") and it did unsafe thing Y
("build a livepatch that opens up a new security hole").
>>>
>>> There's one more factor here: The livepatch tools may behave properly
>>> with one version of the compiler, and improperly with another.
>>
>> I don't really understand the reasoning here. Is this your argument:
>> "One can imagine a security-critical livepatch bug that only affects
>> say, gcc 6.x and not gcc 5.x or 7.x. Therefore, we should never issue
>> XSAs for any security-critical livepatch bugs."
>>
>> If we found that livepatching tools make an incorrect patch only when
>> using gcc 5.x, and we have reason to believe that some people may be
>> using gcc 5.x, then I think we should issue an XSA and say that it only
>> affects people compiling xen with gcc 5.x.
>>
>> It probably would make sense to specify some range of compiler versions
>> for which we will issue XSAs for the livepatch tools. A good baseline
>> would be what versions of gcc Xen uses, and then we can restrict it
>> further if we need to (for instance, if some versions of gcc are missing
>> requisite features, or if they are just known to be buggy).
>>
>> And remember, this is not "We have tested all compiler versions and
>> promise you there are no bugs." It's, "If someone finds a bug for this
>> set of compilers, we will tell you about it so you can do something
>> about it."
>
> I can see and understand all of what you say; my argument,
> however was more towards the matrix of what needs supporting
> possibly becoming unreasonably large (no matter whether we
> specify a range of compilers, as once again distros tend to not
> ship plain unpatched upstream compiler versions).
What do you mean, "The matrix of what needs supporting [may possibly
become] increasingly large"? What is the problem with having a large
(implicit) "supported" matrix? How is supporting a "large matrix" for
livepatch tools different than the current "large matrix" we support
for just building Xen at all?
>> Suppose instead we issued an XSA with a patch, and that it was later
>> discovered that the patch opened up a different security hole when
>> applied on top of XenServer's patchqueue, but not on the baseline
>> XenProject. Would we issue another XSA and/or an update to an existing XSA?
>>
>> The obvious *default* answer to that is "No; it's not practical for us
>> to deal with software that is not inside the XenProject's control." One
>> could imagine circumstances in which we issue statements or an XSA
>> anyway, but that would the exception and not the rule.
>>
>> I think the same kind of thing would apply to the livepatch tools: *by
>> default*, we only issue XSAs for the livepatch tools if they create
>> security issues when generating blobs based on security patches issued
>> by the XenProject, and on top of XenProject-released software. As
>> always, if there's some unforeseen circumstance then someone could argue
>> for an exception.
>
> Not sure here - if analysis showed that the same issue could happen
> elsewhere, and others were just lucky so far, I think we'd have to
> alter the default (and I'm hesitant to call this an exception). Plus
> analysis may, the more different components are involved
> (specifically the compiler, which perhaps none of us has deep enough
> knowledge about), become more and more difficult.
>
> Bottom line - while technically I agree it would be good for the tools
> to be security supported, from a practical perspective I see too
> much complexity for this to be reasonably manageable.
But I still don't understand why you think so. Every single objection
or question about what would or would not be supported that has been
raised so far has analogs in what we already support. It is no more
complex to support livepatch-tools than to support HVM USB
passthrough, or credit2.
I have elsewhere described a hypothetical scenario where I think we
should issue an XSA for livepatch-tools. Are you really seriously
suggesting that in that scenario we should simply publish the
vulnerability onto xen-devel with no predisclosure?
-George
___
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
>>> On 08.08.17 at 13:16, wrote:
> On 08/07/2017 04:59 PM, Jan Beulich wrote:
> George Dunlap 08/07/17 12:27 PM >>>
>>> So it seems that people are still not quite clear about what I'm proposing.
>>
>> And indeed your examples helped me understand better what you mean
>> (or at least I hope they did).
>>
>>> Suppose someone builds a livepatch with the correct compiler, with a
>>> correct patch (that would fix the bug if rebooted into a new
>>> hypervisor), with correct fix-up code. Suppose that the bug passes all
>>> reasonable testing; but that, *due to a bug in the tools*, the patch
>>> also gives PV guests access to hypervisor memory. Is this a security
>>> issue? Yes -- the human told it to do safe thing X ("build a livepatch
>>> based on correct inputs to fix this bug") and it did unsafe thing Y
>>> ("build a livepatch that opens up a new security hole").
>>
>> There's one more factor here: The livepatch tools may behave properly
>> with one version of the compiler, and improperly with another.
>
> I don't really understand the reasoning here. Is this your argument:
> "One can imagine a security-critical livepatch bug that only affects
> say, gcc 6.x and not gcc 5.x or 7.x. Therefore, we should never issue
> XSAs for any security-critical livepatch bugs."
>
> If we found that livepatching tools make an incorrect patch only when
> using gcc 5.x, and we have reason to believe that some people may be
> using gcc 5.x, then I think we should issue an XSA and say that it only
> affects people compiling xen with gcc 5.x.
>
> It probably would make sense to specify some range of compiler versions
> for which we will issue XSAs for the livepatch tools. A good baseline
> would be what versions of gcc Xen uses, and then we can restrict it
> further if we need to (for instance, if some versions of gcc are missing
> requisite features, or if they are just known to be buggy).
>
> And remember, this is not "We have tested all compiler versions and
> promise you there are no bugs." It's, "If someone finds a bug for this
> set of compilers, we will tell you about it so you can do something
> about it."
I can see and understand all of what you say; my argument,
however was more towards the matrix of what needs supporting
possibly becoming unreasonably large (no matter whether we
specify a range of compilers, as once again distros tend to not
ship plain unpatched upstream compiler versions).
>>> We could even place more restrictions on the scope if we wanted to. We
>>> could say that we only support the livepatch tools generating patches
>>> for XSAs.
>>
>> For me, much depends on how tight such restrictions would be. I.e.
>> with the examples given above, how would we determine a canonical
>> livepatch-tools / hypervisor pair (or set of pairs)? After all tools
>> mis-behavior may be a result of some custom patch in someone's
>> derived tree.
>
> Well, suppose that we issued an XSA with a patch, and suppose it was
> later discovered that the patch opened up a different security hole when
> applied on the upstream tree. Would we issue another XSA and/or an
> update to the existing XSA? I think obviously yes we would.
Yes (this has happened in the past already).
> Suppose instead we issued an XSA with a patch, and that it was later
> discovered that the patch opened up a different security hole when
> applied on top of XenServer's patchqueue, but not on the baseline
> XenProject. Would we issue another XSA and/or an update to an existing XSA?
>
> The obvious *default* answer to that is "No; it's not practical for us
> to deal with software that is not inside the XenProject's control." One
> could imagine circumstances in which we issue statements or an XSA
> anyway, but that would the exception and not the rule.
>
> I think the same kind of thing would apply to the livepatch tools: *by
> default*, we only issue XSAs for the livepatch tools if they create
> security issues when generating blobs based on security patches issued
> by the XenProject, and on top of XenProject-released software. As
> always, if there's some unforeseen circumstance then someone could argue
> for an exception.
Not sure here - if analysis showed that the same issue could happen
elsewhere, and others were just lucky so far, I think we'd have to
alter the default (and I'm hesitant to call this an exception). Plus
analysis may, the more different components are involved
(specifically the compiler, which perhaps none of us has deep enough
knowledge about), become more and more difficult.
Bottom line - while technically I agree it would be good for the tools
to be security supported, from a practical perspective I see too
much complexity for this to be reasonably manageable.
Jan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On 08/07/2017 04:59 PM, Jan Beulich wrote:
George Dunlap 08/07/17 12:27 PM >>>
>> So it seems that people are still not quite clear about what I'm proposing.
>
> And indeed your examples helped me understand better what you mean
> (or at least I hope they did).
>
>> Suppose someone builds a livepatch with the correct compiler, with a
>> correct patch (that would fix the bug if rebooted into a new
>> hypervisor), with correct fix-up code. Suppose that the bug passes all
>> reasonable testing; but that, *due to a bug in the tools*, the patch
>> also gives PV guests access to hypervisor memory. Is this a security
>> issue? Yes -- the human told it to do safe thing X ("build a livepatch
>> based on correct inputs to fix this bug") and it did unsafe thing Y
>> ("build a livepatch that opens up a new security hole").
>
> There's one more factor here: The livepatch tools may behave properly
> with one version of the compiler, and improperly with another.
I don't really understand the reasoning here. Is this your argument:
"One can imagine a security-critical livepatch bug that only affects
say, gcc 6.x and not gcc 5.x or 7.x. Therefore, we should never issue
XSAs for any security-critical livepatch bugs."
If we found that livepatching tools make an incorrect patch only when
using gcc 5.x, and we have reason to believe that some people may be
using gcc 5.x, then I think we should issue an XSA and say that it only
affects people compiling xen with gcc 5.x.
It probably would make sense to specify some range of compiler versions
for which we will issue XSAs for the livepatch tools. A good baseline
would be what versions of gcc Xen uses, and then we can restrict it
further if we need to (for instance, if some versions of gcc are missing
requisite features, or if they are just known to be buggy).
And remember, this is not "We have tested all compiler versions and
promise you there are no bugs." It's, "If someone finds a bug for this
set of compilers, we will tell you about it so you can do something
about it."
>> We could even place more restrictions on the scope if we wanted to. We
>> could say that we only support the livepatch tools generating patches
>> for XSAs.
>
> For me, much depends on how tight such restrictions would be. I.e.
> with the examples given above, how would we determine a canonical
> livepatch-tools / hypervisor pair (or set of pairs)? After all tools
> mis-behavior may be a result of some custom patch in someone's
> derived tree.
Well, suppose that we issued an XSA with a patch, and suppose it was
later discovered that the patch opened up a different security hole when
applied on the upstream tree. Would we issue another XSA and/or an
update to the existing XSA? I think obviously yes we would.
Suppose instead we issued an XSA with a patch, and that it was later
discovered that the patch opened up a different security hole when
applied on top of XenServer's patchqueue, but not on the baseline
XenProject. Would we issue another XSA and/or an update to an existing XSA?
The obvious *default* answer to that is "No; it's not practical for us
to deal with software that is not inside the XenProject's control." One
could imagine circumstances in which we issue statements or an XSA
anyway, but that would the exception and not the rule.
I think the same kind of thing would apply to the livepatch tools: *by
default*, we only issue XSAs for the livepatch tools if they create
security issues when generating blobs based on security patches issued
by the XenProject, and on top of XenProject-released software. As
always, if there's some unforeseen circumstance then someone could argue
for an exception.
>>> This is very similar to what XSA-155 was - the GCC compiler optimizations
>>> added a nice jump table that was accessed twice. And the offset was
>>> retrieved from the shared ring.
>>>
>>> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
>>> file a ticket with GCC saying 'Hey, your compiler can create an race
>>> on shared memory. Could you make your compiler be smarter in these cases'
>>> We instead wrote code with this optimization in mind with more
>>> barriers.
>>
>> Right -- so the gcc compiler guys are using a specification that allows
>> that behavior. So from their perspective, we told the compiler to do
>> unsafe thing Y (or at least, said that we were OK with it doing unsafe
>> thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
>> for gcc. If gcc had *violated* the spec when causing the security
>> issue, then we certainly would have called that a security issue in gcc.
>
> But would we have issued an XSA? Wouldn't that rather be a CVE
> against gcc then?
This is changing the question slightly, from "Should X have security
support", to "If X is to be security supported, what organization and
process should be used to support them?"
Obviously in the case of gcc, we would primarily handle the security
issue the way the gcc
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
>>> George Dunlap 08/07/17 12:27 PM >>>
>So it seems that people are still not quite clear about what I'm proposing.
And indeed your examples helped me understand better what you mean
(or at least I hope they did).
>Suppose someone builds a livepatch with the correct compiler, with a
>correct patch (that would fix the bug if rebooted into a new
>hypervisor), with correct fix-up code. Suppose that the bug passes all
>reasonable testing; but that, *due to a bug in the tools*, the patch
>also gives PV guests access to hypervisor memory. Is this a security
>issue? Yes -- the human told it to do safe thing X ("build a livepatch
>based on correct inputs to fix this bug") and it did unsafe thing Y
>("build a livepatch that opens up a new security hole").
There's one more factor here: The livepatch tools may behave properly
with one version of the compiler, and improperly with another. Or they
may behave properly with the Oracle patched hypervisor sources, but
improperly with the Citrix ones (assuming every distro carries a different
set of patches on top of an upstream stable release or branch).
>We could even place more restrictions on the scope if we wanted to. We
>could say that we only support the livepatch tools generating patches
>for XSAs.
For me, much depends on how tight such restrictions would be. I.e.
with the examples given above, how would we determine a canonical
livepatch-tools / hypervisor pair (or set of pairs)? After all tools
mis-behavior may be a result of some custom patch in someone's
derived tree.
>> This is very similar to what XSA-155 was - the GCC compiler optimizations
>> added a nice jump table that was accessed twice. And the offset was
>> retrieved from the shared ring.
>>
>> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
>> file a ticket with GCC saying 'Hey, your compiler can create an race
>> on shared memory. Could you make your compiler be smarter in these cases'
>> We instead wrote code with this optimization in mind with more
>> barriers.
>
>Right -- so the gcc compiler guys are using a specification that allows
>that behavior. So from their perspective, we told the compiler to do
>unsafe thing Y (or at least, said that we were OK with it doing unsafe
>thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
>for gcc. If gcc had *violated* the spec when causing the security
>issue, then we certainly would have called that a security issue in gcc.
But would we have issued an XSA? Wouldn't that rather be a CVE
against gcc then?
Jan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature
On 08/06/2017 01:07 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Aug 03, 2017 at 06:21:30PM +0100, George Dunlap wrote:
>> On 08/03/2017 06:20 PM, George Dunlap wrote:
>>> On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
On 06/30/2017 02:42 PM, George Dunlap wrote:
> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>> On 26/06/17 18:30, Andrew Cooper wrote:
On 26/06/17 18:00, George Dunlap wrote:
> On 26/06/17 16:36, Ross Lagerwall wrote:
>> ...
>>> You seem to be simply refusing to use your imagination. Step back.
>>> Imagine yourself in one year. You come to the office and find an
>>> e-mail
>>> on security@ which says, "Livepatch tools open a security hole when
>>> compiling with gcc x.yy". You realize that XenVerson ${LATEST-2} uses
>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>> that the livepatches generated actually do contain the bug, but you
>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>> newer versions of gcc), the difference was subtle, and it passed all
>>> the
>>> functional tests.
>>>
>>> Now all of the customers that have applied those patches are
>>> vulnerable.
>>>
>>> Do you:
>>>
>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
>>> since
>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>> your customers (as well as anyone else who happens to have used x.yy to
>>> build a hypervisor)?
>>>
>>> 2. Secretly take advantage of Citrix' privileged position on the
>>> security list, and try to get an update out to your customers before it
>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>> experience a zero-day)?
>>>
>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>> before making a public announcement, and so that anyone not on the
>>> embargo list gets an alert, so they know to either update their own
>>> livepatches, or look for updates from their software provider?
>>>
>>> I think #3 is the only possible choice.
>>>
>>>-George
>>>
>>
>> The issue here is that any bug in livepatch-build-tools which still
>> results in output being generated would be a security issue, because
>> someone might have used it to patch a security issue.
>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>> treated in this fashion.
>
> You didn't answer my question. If the situation described happens, what
> position do you want Andrew to be put in? (If I missed a potential
> action, let me know.)
>
I would choose #3 as it is the obvious choice. But I still don't think
it is a sensible idea to have security support for the build tools, at
least at this point. The same scenario could be posed for a nasty bug
that affects Xen 4.4 only, but it is now just out of security support.
IMO something being not supported doesn't preclude it from having an XSA
released if there is a particularly nasty vulnerability found.
>>>
>>> Well basically I think we agree, but we're using different terms. You
>>> want to say, "This isn't security supported, but if important bug is
>>> actually found then we'll issue an XSA". I want to say, "This is
>>> security supported, because if an important bug is actually found we'll
>>> issue an XSA."
>>>
>>> So it seems to me there are likely two things that make you resistant to
>>> calling it "security supported":
>>>
>>> 1. The fear that we'll be issuing XSAs over trivial things that don't matter
>>>
>>> 2. The fear that people will not do due diligence when creating patches
>>> with the tools.
>>>
>>> I think #1 is just a misconception. *Every* bug reported to us about
>>> any part of the code we go through the process of trying to determine
>>> its impact and whether we need to issue an XSA or not. All of the
>>> examples put forward of things we don't want to issue an XSA for are
>>> things that I'm sure we would not issue an XSA for.
>>>
>>> For #2, that is a reasonable fear, but we can deal with that in a
>>> different way than calling the tools "unsupported". We can, for
>>> instance, mention that in the documents. We can add a warning message
>>> that the build tools output saying that the result should be manually
>>> inspected for correctness.
>>
>> We need to get a resolution on this. Anyone else (particarly
>> committers) want to give their opinion?
>
> Changing title as this is all about now livepatch-build-tools.
>
> The livepatch-build-tools get a lot of usage around XSA times.
> And that is when the corner cases are being found. The three of them:
> 0c10457 Remove section alignment requirement
> b30d34c Ignore .discard sections
> 6327ab9 create-diff-object: Update fixup offsets in .rela.
