Re: [virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Sun, Jun 04, 2017 at 06:34:45PM +0800, Wei Wang wrote: > On 05/26/2017 01:57 AM, Michael S. Tsirkin wrote: > > > > I think that's a very valid point. Linux isn't currently optimized to > > handle packets in device BAR. > > There are several issues here and you do need to address them in the > > kernel, no way around that: > > > > 1. lots of drivers set protection to > > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > > > Sorry for my late reply. > > In the implementation tests, I didn't find an issue when letting the > guest directly access the bar MMIO returned by ioremap_cache(). > If that's conventionally improper, we can probably make a new > function similar to ioremap_cache, as the 2nd comment suggests > below. Right. And just disable the driver on architectures that don't support it. > So, in any case, the vhost-pci driver uses ioremap_cache() or a similar > function, which sets the memory type to WB. > And that's great. AFAIK VFIO doesn't though, you will need to teach it to do that to use userspace drivers. > > > vfio certainly does, and so I think does pci sysfs. > > You won't get good performance with this, you want to use > > a cacheable mapping. > > This needs to be addressed for pmd to work well. > > In case it's useful for the discussion here, introduce a little background > about how the bar MMIO is used in vhost-pci: > The device in QEMU sets up the MemoryRegion of the bar as "ram" type, > which will finally have translation mappings created in EPT. So, the memory > setup of the bar is the same as adding a regular RAM. It's like we are > passing through a bar memory to the guest which allows the guest to > directly access to the bar memory. > > Back to the comments, why it is not cacheable memory when the > vhost-pci driver explicitly uses ioremap_cache()? It is. But when you write a userspace driver, you will need to teach vfio to allow cacheable access from userspace. > > > > 2. linux mostly assumes PCI BAR isn't memory, ioremap_cache returns __iomem > > pointers which aren't supposed to be dereferenced directly. > > You want a new API that does direct remap or copy if not possible. > > Alternatively remap or fail, kind of like pci_remap_iospace. > > Maybe there's already something like that - I'm not sure. > > > > For the vhost-pci case, the bar is known to be a portion physical memory. Yes but AFAIK __iomem mappings still can't be portably dereferenced on all architectures. ioremap_cache simply doesn't always give you a dereferencable address. > So, in this case, would it be an issue if the driver directly accesses to > it? > (as mentioned above, the implementation functions correctly) > > Best, > Wei you mean like this: void __iomem *baseptr = ioremap_cache(); unsigned long signature = *(unsigned int *)baseptr; It works on intel. sparse will complain though. See Documentation/bus-virt-phys-mapping.txt -- MST - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/26/2017 01:57 AM, Michael S. Tsirkin wrote: I think that's a very valid point. Linux isn't currently optimized to handle packets in device BAR. There are several issues here and you do need to address them in the kernel, no way around that: 1. lots of drivers set protection to vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); Sorry for my late reply. In the implementation tests, I didn't find an issue when letting the guest directly access the bar MMIO returned by ioremap_cache(). If that's conventionally improper, we can probably make a new function similar to ioremap_cache, as the 2nd comment suggests below. So, in any case, the vhost-pci driver uses ioremap_cache() or a similar function, which sets the memory type to WB. vfio certainly does, and so I think does pci sysfs. You won't get good performance with this, you want to use a cacheable mapping. This needs to be addressed for pmd to work well. In case it's useful for the discussion here, introduce a little background about how the bar MMIO is used in vhost-pci: The device in QEMU sets up the MemoryRegion of the bar as "ram" type, which will finally have translation mappings created in EPT. So, the memory setup of the bar is the same as adding a regular RAM. It's like we are passing through a bar memory to the guest which allows the guest to directly access to the bar memory. Back to the comments, why it is not cacheable memory when the vhost-pci driver explicitly uses ioremap_cache()? 2. linux mostly assumes PCI BAR isn't memory, ioremap_cache returns __iomem pointers which aren't supposed to be dereferenced directly. You want a new API that does direct remap or copy if not possible. Alternatively remap or fail, kind of like pci_remap_iospace. Maybe there's already something like that - I'm not sure. For the vhost-pci case, the bar is known to be a portion physical memory. So, in this case, would it be an issue if the driver directly accesses to it? (as mentioned above, the implementation functions correctly) Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月25日 22:35, Eric Blake wrote: [meta-comment] On 05/25/2017 07:22 AM, Jason Wang wrote: [snip] On 2017年05月16日 15:12, Wei Wang wrote: Hi: 17 levels of '>' when I add my reply. Wow. I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. If vhost-pci to vhost-pci is obvious superior, why not try this consider we're at rather early stage for vhost-pci? I have to scroll a couple of screens past heavily-quoted material before getting to the start of the additions to the thread. It's not only okay, but recommended, to trim your replies down to relevant context so that it is easier to get to your additions (3 or 4 levels of quoted material can still be relevant, but 17 levels is usually a sign that you are including too much). Readers coming in mid-thread can still refer to the public archives if they want more context. Ok, will do. Thanks - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Thu, May 25, 2017 at 08:31:09PM +0800, Jason Wang wrote: > > > On 2017年05月25日 20:22, Jason Wang wrote: > > > > > > > > Even with vhost-pci to virito-net configuration, I think rx > > > > zerocopy could be achieved but not implemented in your driver > > > > (probably more easier in pmd). > > > > > > > Yes, it would be easier with dpdk pmd. But I think it would not be > > > important in the NFV use case, > > > since the data flow goes to one direction often. > > > > > > Best, > > > Wei > > > > > > > I would say let's don't give up on any possible performance optimization > > now. You can do it in the future. > > > > If you still want to keep the copy in both tx and rx, you'd better: > > > > - measure the performance of larger packet size other than 64B > > - consider whether or not it's a good idea to do it in vcpu thread, or > > move it to another one(s) > > > > Thanks > > And what's more important, since you care NFV seriously. I would really > suggest you to draft a pmd for vhost-pci and use it to for benchmarking. > It's real life case. OVS dpdk is known for not optimized for kernel drivers. > > Good performance number can help us to examine the correctness of both > design and implementation. > > Thanks I think that's a very valid point. Linux isn't currently optimized to handle packets in device BAR. There are several issues here and you do need to address them in the kernel, no way around that: 1. lots of drivers set protection to vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vfio certainly does, and so I think does pci sysfs. You won't get good performance with this, you want to use a cacheable mapping. This needs to be addressed for pmd to work well. 2. linux mostly assumes PCI BAR isn't memory, ioremap_cache returns __iomem pointers which aren't supposed to be dereferenced directly. You want a new API that does direct remap or copy if not possible. Alternatively remap or fail, kind of like pci_remap_iospace. Maybe there's already something like that - I'm not sure. Thanks, MST -- MST - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月25日 20:01, Wei Wang wrote: On 05/25/2017 03:59 PM, Jason Wang wrote: On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. If vhost-pci to vhost-pci is obvious superior, why not try this consider we're at rather early stage for vhost-pci? Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Yes, it would be easier with dpdk pmd. But I think it would not be important in the NFV use case, since the data flow goes to one direction often. Best, Wei I would say let's don't give up on any possible performance optimization now. You can do it in the future. If you still want to keep the copy in both tx and rx, you'd better: - measure the performance of larger packet size other than 64B - consider whether or not it's a good idea to do it in vcpu thread, or move it to another one(s) Thanks - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/25/2017 03:59 PM, Jason Wang wrote: On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Yes, it would be easier with dpdk pmd. But I think it would not be important in the NFV use case, since the data flow goes to one direction often. Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Thanks Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? Thanks Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? - Have you tested packet size other than 64B? Not yet. Better to test more since the time spent on 64B copy should be very fast. - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? zerocopy is not used in the test, but I don't think zerocopy can increase the throughput to 2x. I agree, but we need prove this with numbers. Thanks On the other side, we haven't put effort to optimize the draft kernel driver yet. Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. - Have you tested packet size other than 64B? Not yet. - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? zerocopy is not used in the test, but I don't think zerocopy can increase the throughput to 2x. On the other side, we haven't put effort to optimize the draft kernel driver yet. Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? - Have you tested packet size other than 64B? - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? Btw, do you have data about vhost_net v.s. vhost_user? I haven't. Thanks Best, Wei - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] RE: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: > On 2017年05月19日 23:33, Stefan Hajnoczi wrote: > > On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: > >> On 2017年05月18日 11:03, Wei Wang wrote: > >>> On 05/17/2017 02:22 PM, Jason Wang wrote: > On 2017年05月17日 14:16, Jason Wang wrote: > > On 2017年05月16日 15:12, Wei Wang wrote: > >>> Hi: > >>> > >>> Care to post the driver codes too? > >>> > >> OK. It may take some time to clean up the driver code before post > >> it out. You can first have a check of the draft at the repo here: > >> https://github.com/wei-w-wang/vhost-pci-driver > >> > >> Best, > >> Wei > > Interesting, looks like there's one copy on tx side. We used to > > have zerocopy support for tun for VM2VM traffic. Could you please > > try to compare it with your vhost-pci-net by: > > > >>> We can analyze from the whole data path - from VM1's network stack > >>> to send packets -> VM2's network stack to receive packets. The > >>> number of copies are actually the same for both. > >> That's why I'm asking you to compare the performance. The only reason > >> for vhost-pci is performance. You should prove it. > > There is another reason for vhost-pci besides maximum performance: > > > > vhost-pci makes it possible for end-users to run networking or storage > > appliances in compute clouds. Cloud providers do not allow end-users > > to run custom vhost-user processes on the host so you need vhost-pci. > > > > Stefan > > Then it has non NFV use cases and the question goes back to the performance > comparing between vhost-pci and zerocopy vhost_net. If it does not perform > better, it was less interesting at least in this case. > Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Btw, do you have data about vhost_net v.s. vhost_user? Best, Wei
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Thanks - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
[virtio-dev] Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月20日 00:49, Michael S. Tsirkin wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets from its network stack to VM2's RX ring buffer. (we call it "zerocopy" because there is no intermediate copy between VMs) zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies packets from VM1's TX ring buffer to VM2's RX ring buffer. Actually, there's a major difference here. You do copy in guest which consumes time slice of vcpu thread on host. Vhost_net do this in its own thread. So I feel vhost_net is even faster here, maybe I was wrong. Yes but only if you have enough CPUs. The point of vhost-pci is to put the switch in a VM and scale better with # of VMs. Does the overall performance really increase? I suspect the only thing vhost-pci gains here is probably scheduling cost and copying in guest should be slower than doing it in host. That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? And do you use virtio-net kernel driver to compare the performance? If yes, has OVS dpdk optimized for kernel driver (I think not)? What's more important, if vhost-pci is faster, I think its kernel driver should be also faster than virtio-net, no? If you have a vhost CPU per VCPU and can give a host CPU to each using that will be faster. But not everyone has so many host CPUs. If the major use case is NFV, we should have sufficient CPU resources I believe? Thanks - make sure zerocopy is enabled for vhost_net - comment skb_orphan_frags() in tun_net_xmit() Thanks You can even enable tx batching for tun by ethtool -C tap0 rx-frames N. This will greatly improve the performance according to my test. Thanks, but would this hurt latency? Best, Wei I don't see this in my test. Thanks - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
