Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
On Fri, 2009-10-30 at 16:15 -0500, Dustin Kirkland wrote: On Thu, Oct 29, 2009 at 10:34 AM, Dustin Kirkland kirkl...@canonical.com wrote: whitelist host virtio networking features This patch is a followup to 8eca6b1bc770982595db2f7207c65051572436cb, fixing crashes when guests with 2.6.25 virtio drivers have saturated virtio network connections. https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/458521 That patch should have been whitelisting *_HOST_* rather than the the *_GUEST_* features. I tested this by running an Ubuntu 8.04 Hardy guest (2.6.24 kernel + 2.6.25-virtio driver). I saturated both the incoming, and outgoing network connection with nc, seeing sustained 6MB/s up and 6MB/s down bitrates for ~20 minutes. Previously, this crashed immediately. Now, the guest does not crash and maintains network connectivity throughout the test. snip FYI... Thanks for the notice Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. IMHO, the CVE should be against the 2.6.25 virtio drivers - the bug is in the guest and the issue we're discussing here is just a hacky workaround for the guest bug. Cheers, Mark. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
Mark McLoughlin wrote: Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. IMHO, the CVE should be against the 2.6.25 virtio drivers - the bug is in the guest and the issue we're discussing here is just a hacky workaround for the guest bug. Yeah, I'm inclined to agree. The guest generates bad data and we exit. exit()ing is probably not wonderful but it's a well understood behavior. The fundamental bug here is in the guest, not in qemu. Regards, Anthony Liguori Cheers, Mark. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
On Mon, Nov 2, 2009 at 8:38 AM, Mark McLoughlin mar...@redhat.com wrote: On Fri, 2009-10-30 at 16:15 -0500, Dustin Kirkland wrote: Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. IMHO, the CVE should be against the 2.6.25 virtio drivers - the bug is in the guest and the issue we're discussing here is just a hacky workaround for the guest bug. Kees/Jamie/Marc- I think Mark has a good point. This bug has two parts. Ultimately, it's triggered by a buggy virtio-net implementation in the Ubuntu 8.04 kernel (as well as any others using the circa 2.6.25 virtio net code). The CVE should probably mention (or focus on) this too. The qemu-kvm patch is still a good thing to do, as it shouldn't just exit and terminate the VM, so that's needed as well. :-Dustin -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
Jamie Lokier wrote: Anthony Liguori wrote: Mark McLoughlin wrote: Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. IMHO, the CVE should be against the 2.6.25 virtio drivers - the bug is in the guest and the issue we're discussing here is just a hacky workaround for the guest bug. Yeah, I'm inclined to agree. The guest generates bad data and we exit. exit()ing is probably not wonderful but it's a well understood behavior. The fundamental bug here is in the guest, not in qemu. Guests should never be able to crash or terminate qemu, unless they call something that is intentionally an exit qemu hook for the guest. And even that should be possible to disable. Well, if your buggy NIC driver does something wrong programming the hardware (like the famous r8169 did - it allocated less buffer space than telling to the card, so the card were happily overwriting unrelated kernel memory with content received from network), you will most likely get a machine which does not respond to external events, a stuck machine, until you hit reset button (provided there is one) or toggle power. Or just a reboot, depending on what exactly you've hit. If you want kvm to behave like this, wrap it into a trivial shell script that restarts the guest. /mjt -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
Jamie Lokier wrote: Anthony Liguori wrote: Mark McLoughlin wrote: Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. IMHO, the CVE should be against the 2.6.25 virtio drivers - the bug is in the guest and the issue we're discussing here is just a hacky workaround for the guest bug. Yeah, I'm inclined to agree. The guest generates bad data and we exit. exit()ing is probably not wonderful but it's a well understood behavior. The fundamental bug here is in the guest, not in qemu. Guests should never be able to crash or terminate qemu, unless they call something that is intentionally an exit qemu hook for the guest. And even that should be possible to disable. They can exit qemu via an ACPI shutdown. I don't see the difference. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
On Mon, 2009-11-02 at 12:55 -0600, Anthony Liguori wrote: They can exit qemu via an ACPI shutdown. I don't see the difference. An ACPI shutdown is triggered by an authenticated user inside of the guest. The present exit is triggered by any other anonymous user on the network, with the ability to send a lot of packets very quickly to the VM guest. The guest isn't able to handle this properly (and rightly that guest's kernel should be fixed). But I do see a difference. :-Dustin signature.asc Description: This is a digitally signed message part
Re: [Qemu-devel] Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
Michael Tokarev wrote: If you want kvm to behave like this, wrap it into a trivial shell script that restarts the guest. True, kvm has enough crash-bugs elsewhere that I already have to deal with that. It'd be nice to distinguish kvm/qemu bugs from guest bugs, though :-) kvm/qemu also has lock-up bugs, where it's spinning at 100% and the guest seems to be stuck (even though the VNC server continues to work). That's a bit harder to fix with a wrapper script. -- Jamie -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
Dustin Kirkland wrote: On Mon, 2009-11-02 at 12:55 -0600, Anthony Liguori wrote: They can exit qemu via an ACPI shutdown. I don't see the difference. An ACPI shutdown is triggered by an authenticated user inside of the guest. The present exit is triggered by any other anonymous user on the network, with the ability to send a lot of packets very quickly to the VM guest. The guest isn't able to handle this properly (and rightly that guest's kernel should be fixed). But I do see a difference. Well the problem is triggered by the guest kernel writing garbage to virtio-net's backend. That's why we're suggesting it's really a guest kernel issue. If the guest kernel writes something bad to qemu, we're may kill the guest. That's not a qemu bug, it's the designed behavior. Regards, Anthony Liguori :-Dustin -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
On Thu, Oct 29, 2009 at 10:34 AM, Dustin Kirkland kirkl...@canonical.com wrote: whitelist host virtio networking features This patch is a followup to 8eca6b1bc770982595db2f7207c65051572436cb, fixing crashes when guests with 2.6.25 virtio drivers have saturated virtio network connections. https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/458521 That patch should have been whitelisting *_HOST_* rather than the the *_GUEST_* features. I tested this by running an Ubuntu 8.04 Hardy guest (2.6.24 kernel + 2.6.25-virtio driver). I saturated both the incoming, and outgoing network connection with nc, seeing sustained 6MB/s up and 6MB/s down bitrates for ~20 minutes. Previously, this crashed immediately. Now, the guest does not crash and maintains network connectivity throughout the test. snip FYI... Canonical's Ubuntu Security Team will be filing a CVE on this issue, since there is a bit of an attack vector here, and since qemu-kvm-0.11.0 is generally available as an official release (and now part of Ubuntu 9.10). Guests running linux = 2.6.25 virtio-net (e.g Ubuntu 8.04 hardy) on top of qemu-kvm-0.11.0 can be remotely crashed by a non-privileged network user flooding an open port on the guest. The crash happens in a manner that abruptly terminates the guest's execution (ie, without shutting down cleanly). This may affect the guest filesystem's general happiness. :-Dustin -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] whitelist host virtio networking features [was Re: qemu-kvm-0.11 regression, crashes on older ...]
whitelist host virtio networking features This patch is a followup to 8eca6b1bc770982595db2f7207c65051572436cb, fixing crashes when guests with 2.6.25 virtio drivers have saturated virtio network connections. https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/458521 That patch should have been whitelisting *_HOST_* rather than the the *_GUEST_* features. I tested this by running an Ubuntu 8.04 Hardy guest (2.6.24 kernel + 2.6.25-virtio driver). I saturated both the incoming, and outgoing network connection with nc, seeing sustained 6MB/s up and 6MB/s down bitrates for ~20 minutes. Previously, this crashed immediately. Now, the guest does not crash and maintains network connectivity throughout the test. Signed-off-by: Dustin Kirkland kirkl...@canonical.com Signed-off-by: Mark McLoughlin mar...@redhat.com Tested-by: Dustin Kirkland kirkl...@canonical.com diff --git a/hw/virtio-net.c b/hw/virtio-net.c index ce8e6cb..27834fa 100644 --- a/hw/virtio-net.c +++ b/hw/virtio-net.c @@ -164,10 +164,10 @@ static uint32_t virtio_net_bad_features(VirtIODevice *vdev) /* Linux kernel 2.6.25. It understood MAC (as everyone must), * but also these: */ features |= (1 VIRTIO_NET_F_MAC); -features |= (1 VIRTIO_NET_F_GUEST_CSUM); -features |= (1 VIRTIO_NET_F_GUEST_TSO4); -features |= (1 VIRTIO_NET_F_GUEST_TSO6); -features |= (1 VIRTIO_NET_F_GUEST_ECN); +features |= (1 VIRTIO_NET_F_CSUM); +features |= (1 VIRTIO_NET_F_HOST_TSO4); +features |= (1 VIRTIO_NET_F_HOST_TSO6); +features |= (1 VIRTIO_NET_F_HOST_ECN); return features virtio_net_get_features(vdev); } signature.asc Description: This is a digitally signed message part
