Re: Network performance with small packets - continued

2011-03-18 Thread Tom Lendacky
On Thursday, March 10, 2011 11:16:11 am Tom Lendacky wrote: > On Thursday, March 10, 2011 09:34:22 am Michael S. Tsirkin wrote: > > On Thu, Mar 10, 2011 at 09:23:42AM -0600, Tom Lendacky wrote: > > > On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote: > > > > On Wed, Mar 09, 2011 at 0

Re: Network performance with small packets - continued

2011-03-10 Thread Tom Lendacky
On Thursday, March 10, 2011 09:34:22 am Michael S. Tsirkin wrote: > On Thu, Mar 10, 2011 at 09:23:42AM -0600, Tom Lendacky wrote: > > On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote: > > > On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote: > > > > As for which CPU the in

Re: Network performance with small packets - continued

2011-03-10 Thread Michael S. Tsirkin
On Thu, Mar 10, 2011 at 09:23:42AM -0600, Tom Lendacky wrote: > On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote: > > On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote: > > > As for which CPU the interrupt gets pinned to, that doesn't matter - see > > > below. > > > > So

Re: Network performance with small packets - continued

2011-03-10 Thread Tom Lendacky
On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote: > On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote: > > As for which CPU the interrupt gets pinned to, that doesn't matter - see > > below. > > So what hurts us the most is that the IRQ jumps between the VCPUs? Yes, it

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote: > As for which CPU the interrupt gets pinned to, that doesn't matter - see > below. So what hurts us the most is that the IRQ jumps between the VCPUs? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a m

Re: Network performance with small packets - continued

2011-03-09 Thread Rick Jones
On Wed, 2011-03-09 at 16:59 -0800, Shirley Ma wrote: > In theory, for lots of TCP_RR streams, the guest should be able to keep > sending xmit skbs to send vq, so vhost should be able to disable > notification most of the time, then number of guest exits should be > significantly reduced? Why we saw

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 23:56 +0200, Michael S. Tsirkin wrote: > > Txn Rate: 153,696.59 Txn/Sec, Pkt Rate: 305,358 Pkgs/Sec > > Exits: 62,603.37 Exits/Sec > > TxCPU: 3.73% RxCPU: 98.52% > > Virtio1-input Interrupts/Sec (CPU0/CPU1): 11,564/0 > > Virtio1-output Interrupts/Sec (CPU0/CPU1): 0

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
On Wednesday, March 09, 2011 03:56:15 pm Michael S. Tsirkin wrote: > On Wed, Mar 09, 2011 at 02:11:07PM -0600, Tom Lendacky wrote: > > Here are the results again with the addition of the interrupt rate that > > occurred on the guest virtio_net device: > > > > Here is the KVM baseline (average of s

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
On Wednesday, March 09, 2011 04:45:12 pm Shirley Ma wrote: > Hello Tom, > > Do you also have Rusty's virtio stat patch results for both send queue > and recv queue to share here? Let me see what I can do about getting the data extracted, averaged and in a form that I can put in an email. > > T

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
On Wednesday, March 09, 2011 10:09:26 am Tom Lendacky wrote: > On Wednesday, March 09, 2011 01:15:58 am Michael S. Tsirkin wrote: > > On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > > > We've been doing some more experimenting with the small packet network > > > performance problem

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
Hello Tom, Do you also have Rusty's virtio stat patch results for both send queue and recv queue to share here? Thanks Shirley -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/ma

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 02:11:07PM -0600, Tom Lendacky wrote: > Here are the results again with the addition of the interrupt rate that > occurred on the guest virtio_net device: > > Here is the KVM baseline (average of six runs): > Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec > Ex

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
Here are the results again with the addition of the interrupt rate that occurred on the guest virtio_net device: Here is the KVM baseline (average of six runs): Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec Exits: 148,444.58 Exits/Sec TxCPU: 2.40% RxCPU: 99.35% Virtio1-input I

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 19:16 +0200, Michael S. Tsirkin wrote: > On Wed, Mar 09, 2011 at 08:51:33AM -0800, Shirley Ma wrote: > > On Wed, 2011-03-09 at 10:09 -0600, Tom Lendacky wrote: > > > > > Vhost is receiving a lot of notifications for packets that are > to > > > be > > > > > transmitted (over 60

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 08:51:33AM -0800, Shirley Ma wrote: > On Wed, 2011-03-09 at 10:09 -0600, Tom Lendacky wrote: > > > > Vhost is receiving a lot of notifications for packets that are to > > be > > > > transmitted (over 60% of the packets generate a kick_notify). > > This is guest TX send not

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 10:09 -0600, Tom Lendacky wrote: > > > Vhost is receiving a lot of notifications for packets that are to > be > > > transmitted (over 60% of the packets generate a kick_notify). This is guest TX send notification when vhost enables notification. In TCP_STREAM test, vhost ex

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 18:32 +0200, Michael S. Tsirkin wrote: > I think your issues are with TX overrun. > Besides delaying IRQ on TX, I don't have many ideas. > > The one interesting thing is that you see better speed > if you drop packets. netdev crowd says this should not happen, > so could be a

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 08:25:34AM -0800, Shirley Ma wrote: > On Wed, 2011-03-09 at 18:10 +0200, Michael S. Tsirkin wrote: > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index 82dba5a..4477b9a 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 10:09:26AM -0600, Tom Lendacky wrote: > On Wednesday, March 09, 2011 01:15:58 am Michael S. Tsirkin wrote: > > On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > > > We've been doing some more experimenting with the small packet network > > > performance problem

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 18:10 +0200, Michael S. Tsirkin wrote: > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 82dba5a..4477b9a 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -514,11 +514,11 @@ static unsigned int free_old_xmit_skbs(struct

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 10:09 -0600, Tom Lendacky wrote: > > > > > This spread out the kick_notify but still resulted in alot of > them. I > > > decided to build on the delayed Tx buffer freeing and code up an > > > "ethtool" like coalescing patch in order to delay the kick_notify > until > > > the

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
On Wednesday, March 09, 2011 01:17:44 am Michael S. Tsirkin wrote: > On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > > I used the uperf tool to do this after verifying the results against > > netperf. Uperf allows the specification of the number of connections as > > a parameter in

Re: Network performance with small packets - continued

2011-03-09 Thread Michael S. Tsirkin
On Wed, Mar 09, 2011 at 07:45:43AM -0800, Shirley Ma wrote: > On Wed, 2011-03-09 at 09:15 +0200, Michael S. Tsirkin wrote: > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index 82dba5a..ebe3337 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.c > >

Re: Network performance with small packets - continued

2011-03-09 Thread Tom Lendacky
On Wednesday, March 09, 2011 01:15:58 am Michael S. Tsirkin wrote: > On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > > We've been doing some more experimenting with the small packet network > > performance problem in KVM. I have a different setup than what Steve D. > > was using so

Re: Network performance with small packets - continued

2011-03-09 Thread Shirley Ma
On Wed, 2011-03-09 at 09:15 +0200, Michael S. Tsirkin wrote: > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 82dba5a..ebe3337 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -514,11 +514,11 @@ static unsigned int free_old_xmit_skbs(struct > v

Re: Network performance with small packets - continued

2011-03-08 Thread Michael S. Tsirkin
On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > I used the uperf tool to do this after verifying the results against netperf. > > Uperf allows the specification of the number of connections as a parameter in > an XML file as opposed to launching, in this case, 100 separate insta

Re: Network performance with small packets - continued

2011-03-08 Thread Michael S. Tsirkin
On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote: > We've been doing some more experimenting with the small packet network > performance problem in KVM. I have a different setup than what Steve D. was > using so I re-baselined things on the kvm.git kernel on both the host and > gues

Re: Network performance with small packets - continued

2011-03-08 Thread Chigurupati, Chaks
Hi Tom, My two cents. Please look for [Chaks] > >Comparing the transmit path to the receive path, the guest disables >notifications after the first kick and vhost re-enables notifications >after >completing processing of the tx ring. Can a similar thing be done for >the >receive path? Once vhos

Network performance with small packets - continued

2011-03-07 Thread Tom Lendacky
We've been doing some more experimenting with the small packet network performance problem in KVM. I have a different setup than what Steve D. was using so I re-baselined things on the kvm.git kernel on both the host and guest with a 10GbE adapter. I also made use of the virtio-stats patch. T