Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-19 Thread Rusty Russell
On Mon, 19 Dec 2011 10:50:13 +0800, Amos Kong ak...@redhat.com wrote: The format is broken in webpage, attached the result file. it's also available here: http://amosk.info/download/rusty-fix-perf.txt Thanks Amos. Linus, please apply. Fixes virtio-mmio without breaking virtio_pci performance.

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-18 Thread Amos Kong
On 12/12/11 13:12, Rusty Russell wrote: On Mon, 12 Dec 2011 11:06:53 +0800, Amos Kongak...@redhat.com wrote: On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who run multiple netperf streams in

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-18 Thread Benjamin Herrenschmidt
On Mon, 2011-12-19 at 10:19 +0800, Amos Kong wrote: I tested with the same environment and scenarios. tested one scenarios for three times and compute the average for more precision. Thanks, Amos - compare results --- Mon Dec 19 09:51:09 2011 1 -

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-18 Thread Amos Kong
On 19/12/11 10:19, Amos Kong wrote: On 12/12/11 13:12, Rusty Russell wrote: On Mon, 12 Dec 2011 11:06:53 +0800, Amos Kongak...@redhat.com wrote: On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-18 Thread Rusty Russell
On Tue, 13 Dec 2011 07:56:36 +0800, Amos Kong ak...@redhat.com wrote: On 12/12/2011 01:12 PM, Rusty Russell wrote: On Mon, 12 Dec 2011 11:06:53 +0800, Amos Kong ak...@redhat.com wrote: On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-18 Thread Amos Kong
On 19/12/11 10:41, Benjamin Herrenschmidt wrote: On Mon, 2011-12-19 at 10:19 +0800, Amos Kong wrote: I tested with the same environment and scenarios. tested one scenarios for three times and compute the average for more precision. Thanks, Amos - compare results --- Mon Dec

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-12 Thread Amos Kong
On 12/12/2011 01:12 PM, Rusty Russell wrote: On Mon, 12 Dec 2011 11:06:53 +0800, Amos Kong ak...@redhat.com wrote: On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who run multiple netperf streams in

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-11 Thread Michael S. Tsirkin
On Sat, Dec 03, 2011 at 03:44:36PM +1030, Rusty Russell wrote: On Sat, 03 Dec 2011 10:09:44 +1100, Benjamin Herrenschmidt b...@kernel.crashing.org wrote: On Tue, 2011-11-29 at 14:31 +0200, Ohad Ben-Cohen wrote: A trivial, albeit sub-optimal, solution would be to simply revert commit

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-11 Thread Benjamin Herrenschmidt
On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who run multiple netperf streams in parallel, from an external box to the guest. TCP_STREAM results were noisy. This could be due to buffering done by TCP, where packet size varies even as message

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-11 Thread Amos Kong
On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who run multiple netperf streams in parallel, from an external box to the guest. TCP_STREAM results were noisy. This could be due to buffering done

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-11 Thread Rusty Russell
On Mon, 12 Dec 2011 11:06:53 +0800, Amos Kong ak...@redhat.com wrote: On 12/12/11 06:27, Benjamin Herrenschmidt wrote: On Sun, 2011-12-11 at 14:25 +0200, Michael S. Tsirkin wrote: Forwarding some results by Amos, who run multiple netperf streams in parallel, from an external box to the

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-04 Thread Rusty Russell
On Sat, 03 Dec 2011 10:09:44 +1100, Benjamin Herrenschmidt b...@kernel.crashing.org wrote: On Tue, 2011-11-29 at 14:31 +0200, Ohad Ben-Cohen wrote: A trivial, albeit sub-optimal, solution would be to simply revert commit d57ed95 virtio: use smp_XX barriers on SMP. Obviously, though, that's

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-02 Thread Benjamin Herrenschmidt
On Tue, 2011-11-29 at 14:31 +0200, Ohad Ben-Cohen wrote: Virtio is using memory barriers to control the ordering of references to the vrings on SMP systems. When the guest is compiled with SMP support, virtio is only using SMP barriers in order to avoid incurring the overhead involved with

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-02 Thread Ohad Ben-Cohen
On Sat, Dec 3, 2011 at 1:09 AM, Benjamin Herrenschmidt b...@kernel.crashing.org wrote: Have you measured the impact of using normal barriers (non-SMP ones) like we use on normal HW drivers unconditionally ? IE. If the difference is small enough I'd say just go for it and avoid the bloat. I

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-01 Thread Michael S. Tsirkin
On Thu, Dec 01, 2011 at 12:58:59PM +1030, Rusty Russell wrote: On Thu, 1 Dec 2011 01:13:07 +0200, Michael S. Tsirkin m...@redhat.com wrote: For x86, stores into memory are ordered. So I think that yes, smp_XXX can be selected at compile time. So let's forget the virtio strangeness for

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-01 Thread Michael S. Tsirkin
On Thu, Dec 01, 2011 at 08:14:26AM +0200, Ohad Ben-Cohen wrote: On Thu, Dec 1, 2011 at 1:13 AM, Michael S. Tsirkin m...@redhat.com wrote: For x86, stores into memory are ordered. So I think that yes, smp_XXX can be selected at compile time. But then you can't use the same kernel image for

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-12-01 Thread Rusty Russell
On Thu, 1 Dec 2011 10:12:37 +0200, Michael S. Tsirkin m...@redhat.com wrote: On Thu, Dec 01, 2011 at 12:58:59PM +1030, Rusty Russell wrote: On Thu, 1 Dec 2011 01:13:07 +0200, Michael S. Tsirkin m...@redhat.com wrote: For x86, stores into memory are ordered. So I think that yes, smp_XXX

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Tue, Nov 29, 2011 at 5:16 PM, Michael S. Tsirkin m...@redhat.com wrote: This mentions iommu - is there a need to use dma api to let the firmware acess the rings? Or does it have access to all of memory? IOMMU may or may not be used, it really depends on the hardware (my personal SoC does

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Tue, Nov 29, 2011 at 5:19 PM, Michael S. Tsirkin m...@redhat.com wrote: On Tue, Nov 29, 2011 at 03:57:19PM +0200, Ohad Ben-Cohen wrote: Is an extra branch faster or slower than reverting d57ed95? Sorry, unfortunately I have no way to measure this, as I don't have any virtualization/x86

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Michael S. Tsirkin
On Wed, Nov 30, 2011 at 01:55:53PM +0200, Ohad Ben-Cohen wrote: On Tue, Nov 29, 2011 at 5:19 PM, Michael S. Tsirkin m...@redhat.com wrote: On Tue, Nov 29, 2011 at 03:57:19PM +0200, Ohad Ben-Cohen wrote: Is an extra branch faster or slower than reverting d57ed95? Sorry, unfortunately I

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Michael S. Tsirkin
On Wed, Nov 30, 2011 at 01:45:05PM +0200, Ohad Ben-Cohen wrote: So you put virtio rings in MMIO memory? I'll be precise: the vrings are created in non-cacheable memory, which both processors have access to. Could you please give a couple of examples of breakage? Sure. Basically, the

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Wed, Nov 30, 2011 at 4:59 PM, Michael S. Tsirkin m...@redhat.com wrote: I see. And this happens because the ARM processor reorders memory writes Yes. And in an SMP configuration, writes are somehow not reordered? They are, but then the smp memory barriers are enough to control these

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Michael S. Tsirkin
On Wed, Nov 30, 2011 at 06:04:56PM +0200, Ohad Ben-Cohen wrote: On Wed, Nov 30, 2011 at 4:59 PM, Michael S. Tsirkin m...@redhat.com wrote: I see. And this happens because the ARM processor reorders memory writes Yes. And in an SMP configuration, writes are somehow not reordered?

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Wed, Nov 30, 2011 at 6:15 PM, Michael S. Tsirkin m...@redhat.com wrote: How are the rings mapped? normal memory, right? No, device memory. We allocate them with plan alloc_pages_exact in virtio_pci.c ... I'm not using virtio_pci.c; remoteproc is allocating the rings using the DMA API.

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Wed, Nov 30, 2011 at 4:50 PM, Michael S. Tsirkin m...@redhat.com wrote: make headers_install make -C tools/virtio/ (you'll need an empty stub for tools/virtio/linux/module.h,  I just sent a patch to add that) sudo insmod tools/virtio/vhost_test/vhost_test.ko ./tools/virtio/virtio_test

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Michael S. Tsirkin
On Thu, Dec 01, 2011 at 12:43:08AM +0200, Ohad Ben-Cohen wrote: On Wed, Nov 30, 2011 at 4:50 PM, Michael S. Tsirkin m...@redhat.com wrote: make headers_install make -C tools/virtio/ (you'll need an empty stub for tools/virtio/linux/module.h,  I just sent a patch to add that) sudo insmod

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Wed, Nov 30, 2011 at 6:24 PM, Ohad Ben-Cohen o...@wizery.com wrote: On Wed, Nov 30, 2011 at 6:15 PM, Michael S. Tsirkin m...@redhat.com wrote: How are the rings mapped? normal memory, right? No, device memory. Ok, I have more info. Originally remoteproc was mapping the rings using

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Michael S. Tsirkin
On Thu, Dec 01, 2011 at 01:27:10AM +0200, Ohad Ben-Cohen wrote: On Wed, Nov 30, 2011 at 6:24 PM, Ohad Ben-Cohen o...@wizery.com wrote: On Wed, Nov 30, 2011 at 6:15 PM, Michael S. Tsirkin m...@redhat.com wrote: How are the rings mapped? normal memory, right? No, device memory. Ok, I

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Rusty Russell
On Thu, 1 Dec 2011 01:13:07 +0200, Michael S. Tsirkin m...@redhat.com wrote: For x86, stores into memory are ordered. So I think that yes, smp_XXX can be selected at compile time. So let's forget the virtio strangeness for a minute, Hmm, we got away with light barriers because we knew we

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Thu, Dec 1, 2011 at 1:13 AM, Michael S. Tsirkin m...@redhat.com wrote: For x86, stores into memory are ordered. So I think that yes, smp_XXX can be selected at compile time. But then you can't use the same kernel image for both scenarios. It won't take long until people will use virtio on

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Thu, Dec 1, 2011 at 1:43 AM, Michael S. Tsirkin m...@redhat.com wrote: And these accesses need to be ordered with DSB? Or DMB? DMB (i.e. smp barriers) should be enough within Normal memory accesses, though the other issues that were reported to me are a bit concerning. I'm still trying to get

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-30 Thread Ohad Ben-Cohen
On Thu, Dec 1, 2011 at 4:28 AM, Rusty Russell ru...@rustcorp.com.au wrote: Hmm, we got away with light barriers because we knew we were not *really* talking to a device.  But now with virtio-mmio, turns out we are :) I'm really tempted to revert d57ed95 for 3.2, and we can revisit this

[RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-29 Thread Ohad Ben-Cohen
Virtio is using memory barriers to control the ordering of references to the vrings on SMP systems. When the guest is compiled with SMP support, virtio is only using SMP barriers in order to avoid incurring the overhead involved with mandatory barriers. Lately, though, virtio is being

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-29 Thread Michael S. Tsirkin
On Tue, Nov 29, 2011 at 02:31:26PM +0200, Ohad Ben-Cohen wrote: Virtio is using memory barriers to control the ordering of references to the vrings on SMP systems. When the guest is compiled with SMP support, virtio is only using SMP barriers in order to avoid incurring the overhead involved

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-29 Thread Ohad Ben-Cohen
On Tue, Nov 29, 2011 at 3:11 PM, Michael S. Tsirkin m...@redhat.com wrote: On Tue, Nov 29, 2011 at 02:31:26PM +0200, Ohad Ben-Cohen wrote: Virtio is using memory barriers to control the ordering of references to the vrings on SMP systems. When the guest is compiled with SMP support, virtio is

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-29 Thread Michael S. Tsirkin
On Tue, Nov 29, 2011 at 03:57:19PM +0200, Ohad Ben-Cohen wrote: On Tue, Nov 29, 2011 at 3:11 PM, Michael S. Tsirkin m...@redhat.com wrote: On Tue, Nov 29, 2011 at 02:31:26PM +0200, Ohad Ben-Cohen wrote: Virtio is using memory barriers to control the ordering of references to the vrings on

Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

2011-11-29 Thread Michael S. Tsirkin
On Tue, Nov 29, 2011 at 03:57:19PM +0200, Ohad Ben-Cohen wrote: Is an extra branch faster or slower than reverting d57ed95? Sorry, unfortunately I have no way to measure this, as I don't have any virtualization/x86 setup. I'm developing on ARM SoCs, where virtualization hardware is coming,