Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月25日 22:35, Eric Blake wrote: [meta-comment] On 05/25/2017 07:22 AM, Jason Wang wrote: [snip] On 2017年05月16日 15:12, Wei Wang wrote: Hi: 17 levels of '>' when I add my reply. Wow. I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. If vhost-pci to vhost-pci is obvious superior, why not try this consider we're at rather early stage for vhost-pci? I have to scroll a couple of screens past heavily-quoted material before getting to the start of the additions to the thread. It's not only okay, but recommended, to trim your replies down to relevant context so that it is easier to get to your additions (3 or 4 levels of quoted material can still be relevant, but 17 levels is usually a sign that you are including too much). Readers coming in mid-thread can still refer to the public archives if they want more context. Ok, will do. Thanks
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
[meta-comment] On 05/25/2017 07:22 AM, Jason Wang wrote: > [snip] On 2017年05月16日 15:12, Wei Wang wrote: >> Hi: 17 levels of '>' when I add my reply. Wow. >> I think that's another direction or future extension. >> We already have the vhost-pci to virtio-net model on the way, so I >> think it would be better to start from here. >> > > If vhost-pci to vhost-pci is obvious superior, why not try this consider > we're at rather early stage for vhost-pci? > I have to scroll a couple of screens past heavily-quoted material before getting to the start of the additions to the thread. It's not only okay, but recommended, to trim your replies down to relevant context so that it is easier to get to your additions (3 or 4 levels of quoted material can still be relevant, but 17 levels is usually a sign that you are including too much). Readers coming in mid-thread can still refer to the public archives if they want more context. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月25日 20:01, Wei Wang wrote: On 05/25/2017 03:59 PM, Jason Wang wrote: On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. If vhost-pci to vhost-pci is obvious superior, why not try this consider we're at rather early stage for vhost-pci? Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Yes, it would be easier with dpdk pmd. But I think it would not be important in the NFV use case, since the data flow goes to one direction often. Best, Wei I would say let's don't give up on any possible performance optimization now. You can do it in the future. If you still want to keep the copy in both tx and rx, you'd better: - measure the performance of larger packet size other than 64B - consider whether or not it's a good idea to do it in vcpu thread, or move it to another one(s) Thanks
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/25/2017 03:59 PM, Jason Wang wrote: On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? I think that's another direction or future extension. We already have the vhost-pci to virtio-net model on the way, so I think it would be better to start from here. Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Yes, it would be easier with dpdk pmd. But I think it would not be important in the NFV use case, since the data flow goes to one direction often. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月24日 16:31, Wei Wang wrote: On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Then, as has been pointed out. Should we consider a vhost-pci to vhost-pci peer? Even with vhost-pci to virito-net configuration, I think rx zerocopy could be achieved but not implemented in your driver (probably more easier in pmd). Thanks Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/24/2017 11:24 AM, Jason Wang wrote: On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Yes. In both cases, the guests are using kernel drivers. Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? No, we can't. Use this example, VM1's Vhost-pci<->virtio-net of VM2, VM1 sees VM2's memory, but VM2 only sees its own memory. What this copy achieves is to get data from VM1's memory to VM2's memory, so that VM2 can deliver it's own memory to its network stack. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月23日 18:48, Wei Wang wrote: On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Thanks for the pointer. Just to confirm, I think virtio-net kernel driver is used in OVS-dpdk test? Another question is, can we manage to remove the copy in tx? If not, is it a limitation of virtio protocol? Thanks Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Tue, May 23, 2017 at 07:09:05PM +0800, Wei Wang wrote: > On 05/20/2017 04:44 AM, Michael S. Tsirkin wrote: > > On Fri, May 19, 2017 at 05:00:37PM +0800, Wei Wang wrote: > > > > > That being said, we compared to vhost-user, instead of vhost_net, > > > > > because vhost-user is the one > > > > > that is used in NFV, which we think is a major use case for vhost-pci. > > > > If this is true, why not draft a pmd driver instead of a kernel one? > > > Yes, that's right. There are actually two directions of the vhost-pci > > > driver > > > implementation - kernel driver > > > and dpdk pmd. The QEMU side device patches are first posted out for > > > discussion, because when the device > > > part is ready, we will be able to have the related team work on the pmd > > > driver as well. As usual, the pmd > > > driver would give a much better throughput. > > For PMD to work though, the protocol will need to support vIOMMU. > > Not asking you to add it right now since it's work in progress > > for vhost user at this point, but something you will have to > > keep in mind. Further, reviewing vhost user iommu patches might be > > a good idea for you. > > > > For the dpdk pmd case, I'm not sure if vIOMMU is necessary to be used - > Since it only needs to share a piece of memory between the two VMs, we > can only send that piece of memory info for sharing, instead of sending the > entire VM's memory and using vIOMMU to expose that accessible portion. > > Best, > Wei I am not sure I understand what you are saying here. My understanding is that at the moment with VM1 using virtio and VM2 vhost pci, all of VM1's memory is exposed to VM2. If VM1 is using a userspace driver, it needs a way for the kernel to limit the memory regions which are accessible to the device. At the moment this is done by VFIO by means of interacting with a vIOMMU. -- MST
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/20/2017 04:44 AM, Michael S. Tsirkin wrote: On Fri, May 19, 2017 at 05:00:37PM +0800, Wei Wang wrote: That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? Yes, that's right. There are actually two directions of the vhost-pci driver implementation - kernel driver and dpdk pmd. The QEMU side device patches are first posted out for discussion, because when the device part is ready, we will be able to have the related team work on the pmd driver as well. As usual, the pmd driver would give a much better throughput. For PMD to work though, the protocol will need to support vIOMMU. Not asking you to add it right now since it's work in progress for vhost user at this point, but something you will have to keep in mind. Further, reviewing vhost user iommu patches might be a good idea for you. For the dpdk pmd case, I'm not sure if vIOMMU is necessary to be used - Since it only needs to share a piece of memory between the two VMs, we can only send that piece of memory info for sharing, instead of sending the entire VM's memory and using vIOMMU to expose that accessible portion. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/23/2017 02:32 PM, Jason Wang wrote: On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? Oh, that l2fwd is a kernel module from OPNFV vsperf (http://artifacts.opnfv.org/vswitchperf/docs/userguide/quickstart.html) For both legacy and vhost-pci cases, they use the same l2fwd module. No bridge is used, the module already works at L2 to forward packets between two net devices. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月23日 13:47, Wei Wang wrote: On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. Interesting, in the above link, "l2fwd" was used in vhost-pci testing. I want to know more about the test configuration: If l2fwd is the one that dpdk had, want to know how can you make it work for kernel driver. (Maybe packet socket I think?) If not, want to know how do you configure it (e.g through bridge or act_mirred or others). And in OVS dpdk, is dpdk l2fwd + pmd used in the testing? - Have you tested packet size other than 64B? Not yet. Better to test more since the time spent on 64B copy should be very fast. - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? zerocopy is not used in the test, but I don't think zerocopy can increase the throughput to 2x. I agree, but we need prove this with numbers. Thanks On the other side, we haven't put effort to optimize the draft kernel driver yet. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/23/2017 10:08 AM, Jason Wang wrote: On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? Yes, the kernel driver code. - Have you tested packet size other than 64B? Not yet. - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? zerocopy is not used in the test, but I don't think zerocopy can increase the throughput to 2x. On the other side, we haven't put effort to optimize the draft kernel driver yet. Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月22日 19:46, Wang, Wei W wrote: On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Thanks, the number looks good. But I have some questions: - Is the number measured through your vhost-pci kernel driver code? - Have you tested packet size other than 64B? - Is zerocopy supported in OVS-dpdk? If yes, is it enabled in your test? Btw, do you have data about vhost_net v.s. vhost_user? I haven't. Thanks Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Monday, May 22, 2017 10:28 AM, Jason Wang wrote: > On 2017年05月19日 23:33, Stefan Hajnoczi wrote: > > On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: > >> On 2017年05月18日 11:03, Wei Wang wrote: > >>> On 05/17/2017 02:22 PM, Jason Wang wrote: > On 2017年05月17日 14:16, Jason Wang wrote: > > On 2017年05月16日 15:12, Wei Wang wrote: > >>> Hi: > >>> > >>> Care to post the driver codes too? > >>> > >> OK. It may take some time to clean up the driver code before post > >> it out. You can first have a check of the draft at the repo here: > >> https://github.com/wei-w-wang/vhost-pci-driver > >> > >> Best, > >> Wei > > Interesting, looks like there's one copy on tx side. We used to > > have zerocopy support for tun for VM2VM traffic. Could you please > > try to compare it with your vhost-pci-net by: > > > >>> We can analyze from the whole data path - from VM1's network stack > >>> to send packets -> VM2's network stack to receive packets. The > >>> number of copies are actually the same for both. > >> That's why I'm asking you to compare the performance. The only reason > >> for vhost-pci is performance. You should prove it. > > There is another reason for vhost-pci besides maximum performance: > > > > vhost-pci makes it possible for end-users to run networking or storage > > appliances in compute clouds. Cloud providers do not allow end-users > > to run custom vhost-user processes on the host so you need vhost-pci. > > > > Stefan > > Then it has non NFV use cases and the question goes back to the performance > comparing between vhost-pci and zerocopy vhost_net. If it does not perform > better, it was less interesting at least in this case. > Probably I can share what we got about vhost-pci and vhost-user: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost_pci_vs_vhost_user.pdf Right now, I don’t have the environment to add the vhost_net test. Btw, do you have data about vhost_net v.s. vhost_user? Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月19日 23:33, Stefan Hajnoczi wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan Then it has non NFV use cases and the question goes back to the performance comparing between vhost-pci and zerocopy vhost_net. If it does not perform better, it was less interesting at least in this case. Thanks
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月20日 00:49, Michael S. Tsirkin wrote: On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets from its network stack to VM2's RX ring buffer. (we call it "zerocopy" because there is no intermediate copy between VMs) zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies packets from VM1's TX ring buffer to VM2's RX ring buffer. Actually, there's a major difference here. You do copy in guest which consumes time slice of vcpu thread on host. Vhost_net do this in its own thread. So I feel vhost_net is even faster here, maybe I was wrong. Yes but only if you have enough CPUs. The point of vhost-pci is to put the switch in a VM and scale better with # of VMs. Does the overall performance really increase? I suspect the only thing vhost-pci gains here is probably scheduling cost and copying in guest should be slower than doing it in host. That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? And do you use virtio-net kernel driver to compare the performance? If yes, has OVS dpdk optimized for kernel driver (I think not)? What's more important, if vhost-pci is faster, I think its kernel driver should be also faster than virtio-net, no? If you have a vhost CPU per VCPU and can give a host CPU to each using that will be faster. But not everyone has so many host CPUs. If the major use case is NFV, we should have sufficient CPU resources I believe? Thanks - make sure zerocopy is enabled for vhost_net - comment skb_orphan_frags() in tun_net_xmit() Thanks You can even enable tx batching for tun by ethtool -C tap0 rx-frames N. This will greatly improve the performance according to my test. Thanks, but would this hurt latency? Best, Wei I don't see this in my test. Thanks
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Fri, May 19, 2017 at 05:00:37PM +0800, Wei Wang wrote: > > > > > > That being said, we compared to vhost-user, instead of vhost_net, > > > because vhost-user is the one > > > that is used in NFV, which we think is a major use case for vhost-pci. > > > > If this is true, why not draft a pmd driver instead of a kernel one? > > Yes, that's right. There are actually two directions of the vhost-pci driver > implementation - kernel driver > and dpdk pmd. The QEMU side device patches are first posted out for > discussion, because when the device > part is ready, we will be able to have the related team work on the pmd > driver as well. As usual, the pmd > driver would give a much better throughput. For PMD to work though, the protocol will need to support vIOMMU. Not asking you to add it right now since it's work in progress for vhost user at this point, but something you will have to keep in mind. Further, reviewing vhost user iommu patches might be a good idea for you. -- MST
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: > > > On 2017年05月18日 11:03, Wei Wang wrote: > > On 05/17/2017 02:22 PM, Jason Wang wrote: > > > > > > > > > On 2017年05月17日 14:16, Jason Wang wrote: > > > > > > > > > > > > On 2017年05月16日 15:12, Wei Wang wrote: > > > > > > > > > > > > > > > > > > > Hi: > > > > > > > > > > > > Care to post the driver codes too? > > > > > > > > > > > OK. It may take some time to clean up the driver code before > > > > > post it out. You can first > > > > > have a check of the draft at the repo here: > > > > > https://github.com/wei-w-wang/vhost-pci-driver > > > > > > > > > > Best, > > > > > Wei > > > > > > > > Interesting, looks like there's one copy on tx side. We used to > > > > have zerocopy support for tun for VM2VM traffic. Could you > > > > please try to compare it with your vhost-pci-net by: > > > > > > We can analyze from the whole data path - from VM1's network stack to > > send packets -> VM2's > > network stack to receive packets. The number of copies are actually the > > same for both. > > That's why I'm asking you to compare the performance. The only reason for > vhost-pci is performance. You should prove it. > > > > > vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets > > from its network stack to VM2's > > RX ring buffer. (we call it "zerocopy" because there is no intermediate > > copy between VMs) > > zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies > > packets from VM1's TX ring > > buffer to VM2's RX ring buffer. > > Actually, there's a major difference here. You do copy in guest which > consumes time slice of vcpu thread on host. Vhost_net do this in its own > thread. So I feel vhost_net is even faster here, maybe I was wrong. Yes but only if you have enough CPUs. The point of vhost-pci is to put the switch in a VM and scale better with # of VMs. > > > > That being said, we compared to vhost-user, instead of vhost_net, > > because vhost-user is the one > > that is used in NFV, which we think is a major use case for vhost-pci. > > If this is true, why not draft a pmd driver instead of a kernel one? And do > you use virtio-net kernel driver to compare the performance? If yes, has OVS > dpdk optimized for kernel driver (I think not)? > > What's more important, if vhost-pci is faster, I think its kernel driver > should be also faster than virtio-net, no? If you have a vhost CPU per VCPU and can give a host CPU to each using that will be faster. But not everyone has so many host CPUs. > > > > > > > > - make sure zerocopy is enabled for vhost_net > > > > - comment skb_orphan_frags() in tun_net_xmit() > > > > > > > > Thanks > > > > > > > > > > You can even enable tx batching for tun by ethtool -C tap0 rx-frames > > > N. This will greatly improve the performance according to my test. > > > > > > > Thanks, but would this hurt latency? > > > > Best, > > Wei > > I don't see this in my test. > > Thanks
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On Fri, May 19, 2017 at 11:10:33AM +0800, Jason Wang wrote: > On 2017年05月18日 11:03, Wei Wang wrote: > > On 05/17/2017 02:22 PM, Jason Wang wrote: > > > On 2017年05月17日 14:16, Jason Wang wrote: > > > > On 2017年05月16日 15:12, Wei Wang wrote: > > > > > > Hi: > > > > > > > > > > > > Care to post the driver codes too? > > > > > > > > > > > OK. It may take some time to clean up the driver code before > > > > > post it out. You can first > > > > > have a check of the draft at the repo here: > > > > > https://github.com/wei-w-wang/vhost-pci-driver > > > > > > > > > > Best, > > > > > Wei > > > > > > > > Interesting, looks like there's one copy on tx side. We used to > > > > have zerocopy support for tun for VM2VM traffic. Could you > > > > please try to compare it with your vhost-pci-net by: > > > > > > We can analyze from the whole data path - from VM1's network stack to > > send packets -> VM2's > > network stack to receive packets. The number of copies are actually the > > same for both. > > That's why I'm asking you to compare the performance. The only reason for > vhost-pci is performance. You should prove it. There is another reason for vhost-pci besides maximum performance: vhost-pci makes it possible for end-users to run networking or storage appliances in compute clouds. Cloud providers do not allow end-users to run custom vhost-user processes on the host so you need vhost-pci. Stefan signature.asc Description: PGP signature
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月19日 17:00, Wei Wang wrote: On 05/19/2017 11:10 AM, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets from its network stack to VM2's RX ring buffer. (we call it "zerocopy" because there is no intermediate copy between VMs) zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies packets from VM1's TX ring buffer to VM2's RX ring buffer. Actually, there's a major difference here. You do copy in guest which consumes time slice of vcpu thread on host. Vhost_net do this in its own thread. So I feel vhost_net is even faster here, maybe I was wrong. The code path using vhost_net is much longer - the Ping test shows that the zcopy based vhost_net reports around 0.237ms, while using vhost-pci it reports around 0.06 ms. For some environment issue, I can report the throughput number later. Yes, vhost-pci should have better latency by design. But we should measure pps or packet size other than 64 as well. I agree vhost_net has bad latency, but this does not mean it could not be improved (just because few people are working on improve this in the past), especially we know the destination is another VM. That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? Yes, that's right. There are actually two directions of the vhost-pci driver implementation - kernel driver and dpdk pmd. The QEMU side device patches are first posted out for discussion, because when the device part is ready, we will be able to have the related team work on the pmd driver as well. As usual, the pmd driver would give a much better throughput. I think pmd should be easier for a prototype than kernel driver. So, I think at this stage we should focus on the device part review, and use the kernel driver to prove that the device part design and implementation is reasonable and functional. Probably both. And do you use virtio-net kernel driver to compare the performance? If yes, has OVS dpdk optimized for kernel driver (I think not)? We used the legacy OVS+DPDK. Another thing with the existing OVS+DPDK usage is its centralization property. With vhost-pci, we will be able to de-centralize the usage. Right, so I think we should prove: - For usage, prove or make vhost-pci better than existed share memory based solution. (Or is virtio good at shared memory?) - For performance, prove or make vhost-pci better than existed centralized solution. What's more important, if vhost-pci is faster, I think its kernel driver should be also faster than virtio-net, no? Sorry about the confusion. We are actually not trying to use vhost-pci to replace virtio-net. Rather, vhost-pci can be viewed as another type of backend for virtio-net to be used in NFV (the communication channel is vhost-pci-net<->virtio_net). My point is performance number is important for proving the correctness for both design and engineering. If its slow, it has less interesting in NFV. Thanks Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 05/19/2017 11:10 AM, Jason Wang wrote: On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets from its network stack to VM2's RX ring buffer. (we call it "zerocopy" because there is no intermediate copy between VMs) zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies packets from VM1's TX ring buffer to VM2's RX ring buffer. Actually, there's a major difference here. You do copy in guest which consumes time slice of vcpu thread on host. Vhost_net do this in its own thread. So I feel vhost_net is even faster here, maybe I was wrong. The code path using vhost_net is much longer - the Ping test shows that the zcopy based vhost_net reports around 0.237ms, while using vhost-pci it reports around 0.06 ms. For some environment issue, I can report the throughput number later. That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? Yes, that's right. There are actually two directions of the vhost-pci driver implementation - kernel driver and dpdk pmd. The QEMU side device patches are first posted out for discussion, because when the device part is ready, we will be able to have the related team work on the pmd driver as well. As usual, the pmd driver would give a much better throughput. So, I think at this stage we should focus on the device part review, and use the kernel driver to prove that the device part design and implementation is reasonable and functional. And do you use virtio-net kernel driver to compare the performance? If yes, has OVS dpdk optimized for kernel driver (I think not)? We used the legacy OVS+DPDK. Another thing with the existing OVS+DPDK usage is its centralization property. With vhost-pci, we will be able to de-centralize the usage. What's more important, if vhost-pci is faster, I think its kernel driver should be also faster than virtio-net, no? Sorry about the confusion. We are actually not trying to use vhost-pci to replace virtio-net. Rather, vhost-pci can be viewed as another type of backend for virtio-net to be used in NFV (the communication channel is vhost-pci-net<->virtio_net). Best, Wei
Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication
On 2017年05月18日 11:03, Wei Wang wrote: On 05/17/2017 02:22 PM, Jason Wang wrote: On 2017年05月17日 14:16, Jason Wang wrote: On 2017年05月16日 15:12, Wei Wang wrote: Hi: Care to post the driver codes too? OK. It may take some time to clean up the driver code before post it out. You can first have a check of the draft at the repo here: https://github.com/wei-w-wang/vhost-pci-driver Best, Wei Interesting, looks like there's one copy on tx side. We used to have zerocopy support for tun for VM2VM traffic. Could you please try to compare it with your vhost-pci-net by: We can analyze from the whole data path - from VM1's network stack to send packets -> VM2's network stack to receive packets. The number of copies are actually the same for both. That's why I'm asking you to compare the performance. The only reason for vhost-pci is performance. You should prove it. vhost-pci: 1-copy happen in VM1's driver xmit(), which copes packets from its network stack to VM2's RX ring buffer. (we call it "zerocopy" because there is no intermediate copy between VMs) zerocopy enabled vhost-net: 1-copy happen in tun's recvmsg, which copies packets from VM1's TX ring buffer to VM2's RX ring buffer. Actually, there's a major difference here. You do copy in guest which consumes time slice of vcpu thread on host. Vhost_net do this in its own thread. So I feel vhost_net is even faster here, maybe I was wrong. That being said, we compared to vhost-user, instead of vhost_net, because vhost-user is the one that is used in NFV, which we think is a major use case for vhost-pci. If this is true, why not draft a pmd driver instead of a kernel one? And do you use virtio-net kernel driver to compare the performance? If yes, has OVS dpdk optimized for kernel driver (I think not)? What's more important, if vhost-pci is faster, I think its kernel driver should be also faster than virtio-net, no? - make sure zerocopy is enabled for vhost_net - comment skb_orphan_frags() in tun_net_xmit() Thanks You can even enable tx batching for tun by ethtool -C tap0 rx-frames N. This will greatly improve the performance according to my test. Thanks, but would this hurt latency? Best, Wei I don't see this in my test. Thanks
