Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Tue, Aug 11, 2009 at 07:49:37PM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet. Some more detailed description attached to the patch itself. The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment. I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run? Regards, -Greg 1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance) 2. disable tso,gso,lro with ethtool 3. add vhost=ethX -- MST -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
Michael S. Tsirkin wrote: On Tue, Aug 11, 2009 at 07:49:37PM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet. Some more detailed description attached to the patch itself. The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment. I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run? Regards, -Greg 1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance) Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). 2. disable tso,gso,lro with ethtool Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost? 3. add vhost=ethX You mean via ip link I assume? Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 07:56:05AM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: On Tue, Aug 11, 2009 at 07:49:37PM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet. Some more detailed description attached to the patch itself. The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment. I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run? Regards, -Greg 1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance) Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined. 2. disable tso,gso,lro with ethtool Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost? Hmm, makes sense. I'll check this and let you know. 3. add vhost=ethX You mean via ip link I assume? No, that's a new flag for virtio in qemu: -net nic,model=virtio,vhost=veth0 Regards, -Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
Michael S. Tsirkin wrote: On Wed, Aug 12, 2009 at 07:56:05AM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: snip 1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance) Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined. Makes sense. Got it. I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge? snip 3. add vhost=ethX You mean via ip link I assume? No, that's a new flag for virtio in qemu: -net nic,model=virtio,vhost=veth0 Ah, ok. Even better. Thanks! -Greg signature.asc Description: OpenPGP digital signature
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wednesday 12 August 2009, Gregory Haskins wrote: Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined. Makes sense. Got it. I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge? If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 It's a bit more complicated than it need to be, but should work fine. Arnd -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 08:41:31AM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: On Wed, Aug 12, 2009 at 07:56:05AM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: snip 1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance) Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined. Makes sense. Got it. I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge? snip Presumably you mean on the same host? There were also some patches to enable local guest to guest for macvlan, that would be a nice software-only solution. For back to back, I just tried over veth, seems to work fine. 3. add vhost=ethX You mean via ip link I assume? No, that's a new flag for virtio in qemu: -net nic,model=virtio,vhost=veth0 Ah, ok. Even better. Thanks! -Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 02:52:01PM +0200, Arnd Bergmann wrote: On Wednesday 12 August 2009, Gregory Haskins wrote: Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty). SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined. Makes sense. Got it. I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge? If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 It's a bit more complicated than it need to be, but should work fine. Arnd Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 03:40:44PM +0200, Arnd Bergmann wrote: On Wednesday 12 August 2009, Michael S. Tsirkin wrote: If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity. Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance. Arnd Oh, hopefully macvlan will soon allow that. -- MST -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wednesday 12 August 2009, Michael S. Tsirkin wrote: If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity. Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance. Arnd -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 09:51:45AM -0400, Gregory Haskins wrote: Arnd Bergmann wrote: On Wednesday 12 August 2009, Michael S. Tsirkin wrote: If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity. Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance. Arnd Yeah, this would be the config I would be interested in. Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest. Here's one way to do this: -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22 In guests, for simplicity, configure eth1 and eth0 to use separate subnets. Long term, I hope macvlan will be extended to support guest to guest. Regards, -Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
Michael S. Tsirkin wrote: On Wed, Aug 12, 2009 at 09:51:45AM -0400, Gregory Haskins wrote: Arnd Bergmann wrote: On Wednesday 12 August 2009, Michael S. Tsirkin wrote: If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity. Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance. Arnd Yeah, this would be the config I would be interested in. Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest. Here's one way to do this: -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22 In guests, for simplicity, configure eth1 and eth0 to use separate subnets. I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean all hosts (virtual or physical) in the same L2 domain. If I get a chance, though, I will try to also wire them up in isolation as another data point. Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
On Wed, Aug 12, 2009 at 12:13:43PM -0400, Gregory Haskins wrote: Michael S. Tsirkin wrote: On Wed, Aug 12, 2009 at 09:51:45AM -0400, Gregory Haskins wrote: Arnd Bergmann wrote: On Wednesday 12 August 2009, Michael S. Tsirkin wrote: If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like veth0 - veth1 - vhost - guest 1 eth0 - br0-| veth2 - veth3 - vhost - guest 2 Heh, you don't need a bridge in this picture: guest 1 - vhost - veth0 - veth1 - vhost guest 2 Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity. Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance. Arnd Yeah, this would be the config I would be interested in. Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest. Here's one way to do this: -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22 In guests, for simplicity, configure eth1 and eth0 to use separate subnets. I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean all hosts (virtual or physical) in the same L2 domain. If I get a chance, though, I will try to also wire them up in isolation as another data point. Regards, -Greg Or patch macvlan to support guest to guest: http://markmail.org/message/sjy74g57qsvdo2wh That patch needs to be updated to support guest to guest multiast, but it seems functional enough for your purposes. -- MST -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 0/2] vhost: a kernel-level virtio server
Michael S. Tsirkin wrote: This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet. Some more detailed description attached to the patch itself. The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment. I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run? Regards, -Greg signature.asc Description: OpenPGP digital signature
