On Thu, 28 Jun 2012 09:03:51 +0000 "Waskiewicz Jr, Peter P" <[email protected]> wrote:
> On Thu, 2012-06-28 at 08:55 +0000, Anna Fischer wrote: > > Hi, I have 2 VCPUs assigned to the VM. I thought more CPUs wouldn't > > really help because a single TCP connection doesn't span across > > cores anyway? I would have to run multiple connections > > simultaneously I think. > > Do you see 2 Tx and 2 Rx queues in ethtool -S ethX inside your VM or > no? The issue you can run into is head-of-line blocking of ACKs being > sent if you're using a single Tx and Rx queue. TCP connections can't > have their ACKs sent or be received and processed since there's > pending data packets needing to be serviced. > > -PJ The currently released ixgbevf driver uses only a single tx/rx queue pair. The VF has access to two queues but there is no way to direct traffic to multiple queues in the VF anyway, since it does not support RSS, DCB or any other way of directing traffic across multiple receive queues. Still, the numbers that you are seeing are a bit low. Generally we see closer to 6.5 Gbps on a single VF but there are other factors that could come into play to affect that number I suppose. The point of SR-IOV virtual functions is to share I/O. You will not get full line rate performance from a single VF. If you want a single device to get full line rate, just assign the entire 82599 device to the guest VM (with an appropriate number of VCPUs obviously) and then you'll get full line rate in both transmit and receive directions. To get close to full line rate on your 10Gig connection with VFs you'll need to run multiple VFs. Generally two or three should do it. As far as why you are seeing CPU idle time I think you need to see if you're dropping packets. If no packets are being dropped and there is free CPU then it is a result of other overhead in the TCP conversation - probably head of line blocking due to the ACKS that PJ mentioned. - Greg > > > I use the standard packet size, I don't do jumbo frames or > > anything. I don't tweak the OS either, I just want to see how it > > would perform by default. > > > > I'm just surprised that CPU isn't maxed out at all. It must be > > because polling/NAPI is too slow fetching packets? It must be a > > timer thing like this? > > Otherwise, is it that hardware resources are limited with just a > > single queue and the RX ring buffer cannot be filled/refilled > > quickly enough? Why else would it be slow if CPU resources are > > there. > > > > When I say maxed out I mean the VCPU of course, so those cores are > > not running at full capability. > > > > Anna > > > > > > ______________________________________________________________________ > > From: "Waskiewicz Jr, Peter P" <[email protected]> > > Date: Thu, 28 Jun 2012 08:45:23 +0000 > > To: Fischer, Anna<[email protected]> > > Cc: > > [email protected]<[email protected]> > > Subject: Re: [E1000-devel] 82599 receive performance > > > > > > Hi Anna, > > > > On Thu, 2012-06-28 at 08:32 +0000, Fischer, Anna wrote: > > > Hi, I'm running a 10G 82599 controller in PCI SR-IOV mode, and the > > VF is assigned to a VM (PCI pass-through). I only enable a single VF > > (max_vfs=1). On my machine I run KVM and just a single VM. > > > The machine has 4 Xeon X5550 cores. > > > > > > I've tested RX and TX throughput using iperf (TCP). Basically I > > > want > > to see how fast I can send and receive packets to/from the VM. > > > > How many CPU's are provisioned to your VM? Also, what packet size > > are you using with iperf? In many cases, a single Rx/Tx queue pair > > is not enough to run 10GbE line rate. > > > > -PJ > > > > > TX performance comes to about 8.5Gbps. At that point, CPU is maxed > > out. > > > > > > RX performance only comes to about 5.5Gbps. CPU isn't maxed out at > > all, and I wonder, if this is the expected RX throughput? I have to > > say that I only run a single TX queue and a single RX queue. (I am > > testing against something else that also just runs a single queue, > > so I need to keep it fair.) I haven't tuned any OS parameters, I run > > Fedora Core 16 and a 3.4 Linux kernel. > > > > > > Thanks, > > > Anna > > > > > > > > ------------------------------------------------------------------------------ > > > Live Security Virtual Conference > > > Exclusive live event will cover all the ways today's security and > > > threat landscape has changed and how IT managers can respond. > > Discussions > > > will include endpoint security, mobile security and the latest in > > malware > > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > > _______________________________________________ > > > E1000-devel mailing list > > > [email protected] > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > To learn more about Intel® Ethernet, visit > > http://communities.intel.com/community/wired > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. > Discussions will include endpoint security, mobile security and the > latest in malware threats. > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ E1000-devel mailing > list [email protected] > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
