RE: vhost-net patches

[Shirley Ma]

Hello Xiaohui,

On Tue, 2009-11-03 at 09:06 +0800, Xin, Xiaohui wrote:
 Hi, Michael,
 What's your deferring skb allocation patch mentioned here, may you
 elaborate it a little more detailed?

That's my patch. It was submitted a few month ago. Here is the link to
this RFC patch:
http://www.mail-archive.com/kvm@vger.kernel.org/msg20777.html

It is a patch for guest receiving. Right now, kvm guest did pre-skb
allocation, the worse case when receiving large packet for mergable
buffers, 15/16 skbs need to be freed. Avi and Michale gave me some
comments. I will post the updated patch after running a few more test in
a few days.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 29, 2009 at 12:11:33AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 I am able to get 63xxMb/s throughput with 10% less cpu utilization when
 I apply deferring skb patch on top of your most recent vhost patch. The
 userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 
 
 After applying your recent vhost patch, it goes up to 53xxMb/s.

Thanks for the report!  I have pushed this patch (after cleaning it up
and fixing raw sockets), and updated the numbers in the wiki here
http://www.linux-kvm.org/page/VhostNet

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: vhost-net patches

[Xin, Xiaohui]

Hi, Michael,
What's your deferring skb allocation patch mentioned here, may you elaborate it 
a little more detailed?

Thanks
Xiaohui

-Original Message-
From: kvm-ow...@vger.kernel.org [mailto:kvm-ow...@vger.kernel.org] On Behalf Of 
Michael S. Tsirkin
Sent: Tuesday, November 03, 2009 3:47 AM
To: Shirley Ma
Cc: Sridhar Samudrala; Shirley Ma; David Stevens; kvm@vger.kernel.org; 
s...@linux.vnet.ibm.com; mashi...@linux.vnet.ibm.com
Subject: Re: vhost-net patches

On Thu, Oct 29, 2009 at 12:11:33AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 I am able to get 63xxMb/s throughput with 10% less cpu utilization when
 I apply deferring skb patch on top of your most recent vhost patch. The
 userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 
 
 After applying your recent vhost patch, it goes up to 53xxMb/s.

Thanks for the report!  I have pushed this patch (after cleaning it up
and fixing raw sockets), and updated the numbers in the wiki here
http://www.linux-kvm.org/page/VhostNet

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

I am able to get 63xxMb/s throughput with 10% less cpu utilization when
I apply deferring skb patch on top of your most recent vhost patch. The
userspace TCP_STREAM BW used to be 3xxxMb/s from upper stream git tree. 

After applying your recent vhost patch, it goes up to 53xxMb/s. Now it
can reach 63xxMb/s, pretty good news. (this test with transmission
descriptors = 1K)

netperf output with recent vhost patch:

[r...@localhost ~]# netperf -H 192.168.122.1 -c -C -l60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.1
(192.168.122.1) port 0 AF_INET
Recv   SendSend  Utilization   Service
Demand
Socket Socket  Message  Elapsed  Send Recv Send
Recv
Size   SizeSize Time Throughput  localremote   local
remote
bytes  bytes   bytessecs.10^6bits/s  % S  % S  us/KB
us/KB

 87380  16384  1638460.03  5325.82   99.5883.331.532
2.564

netperf output with vhost + deferring skb allocation patch

[r...@localhost linux-2.6.32-rc5]# netperf -H 192.168.122.1 -c -C -l60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.1
(192.168.122.1) port 0 AF_INET
Recv   SendSend  Utilization   Service
Demand
Socket Socket  Message  Elapsed  Send Recv Send
Recv
Size   SizeSize Time Throughput  localremote   local
remote
bytes  bytes   bytessecs.10^6bits/s  % S  % S  us/KB
us/KB

 87380  16384  1638460.02  6332.38   99.3376.161.285
1.970

Thanks
Shirley


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm memory leak (Was Re: vhost-net patches)

[Avi Kivity]

On 10/28/2009 06:46 PM, Michael S. Tsirkin wrote:

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
   

Hello Michael,

On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
 

Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
Is there a chance you are tight on guest memory for some reason?
with vhost, virtio does currently consume a bit more memory than
with userspace backend.
   

I did see memory leak on host every time after exiting guest. I don't
know where. Do you see it?
 

I don't see a leak specific to vhost. However, even without vhost,
starting up guest increases kmalloc-128 counter in /proc/slabinfo
and this fails to go down when qemu is killed.

Anyone else seeing this?
   


I'm seeing it, but starting a new guest seems to reclaim this memory.  
I'll enable debug and see what's the cause.


--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm memory leak (Was Re: vhost-net patches)

[Avi Kivity]

On 10/29/2009 02:21 PM, Avi Kivity wrote:

On 10/28/2009 06:46 PM, Michael S. Tsirkin wrote:

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:

Hello Michael,

On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:

Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
Is there a chance you are tight on guest memory for some reason?
with vhost, virtio does currently consume a bit more memory than
with userspace backend.

I did see memory leak on host every time after exiting guest. I don't
know where. Do you see it?

I don't see a leak specific to vhost. However, even without vhost,
starting up guest increases kmalloc-128 counter in /proc/slabinfo
and this fails to go down when qemu is killed.

Anyone else seeing this?


I'm seeing it, but starting a new guest seems to reclaim this memory.  
I'll enable debug and see what's the cause.




kmemleak doesn't see any kvm memory leaks.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
  Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
  Is there a chance you are tight on guest memory for some reason?
  with vhost, virtio does currently consume a bit more memory than
  with userspace backend.
 
 I did see memory leak on host every time after exiting guest. I don't
 know where. Do you see it?
 
 Anyway after I reboot host and restart guest with large memory
 allocation, I do see performance improves to 3xxxMb/s, and occasionally
 reaches 40xxMb/s. But queue full still exists, I can avoid the problem
 by increasing send queue size from qemu.

Here's another hack to try. It will break raw sockets,
but just as a test:

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index a140dad..e830b30 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -106,22 +106,41 @@ static void handle_tx(struct vhost_net *net)
.msg_flags = MSG_DONTWAIT,
};
size_t len, total_len = 0;
-   int err;
+   int err, wmem;
size_t hdr_size;
struct socket *sock = rcu_dereference(vq-private_data);
-   if (!sock || !sock_writeable(sock-sk))
+   if (!sock)
+   return;
+
+   wmem = atomic_read(sock-sk-sk_wmem_alloc);
+   if (wmem = sock-sk-sk_sndbuf)
return;
 
use_mm(net-dev.mm);
mutex_lock(vq-mutex);
-   tx_poll_stop(net);
+   vhost_no_notify(vq);
+   
+   if (wmem = sock-sk-sk_sndbuf * 3 / 4) {
+   //tx_poll_start(net);
+   } else {
+   //tx_poll_stop(net);
+   }
hdr_size = vq-hdr_size;
 
for (;;) {
head = vhost_get_vq_desc(net-dev, vq, vq-iov, out, in);
/* Nothing new?  Wait for eventfd to tell us they refilled. */
-   if (head == vq-num)
+   if (head == vq-num) {
+   wmem = atomic_read(sock-sk-sk_wmem_alloc);
+   if (wmem = sock-sk-sk_sndbuf * 3 / 4) {
+   set_bit(SOCK_ASYNC_NOSPACE, sock-flags);
+   break;
+   }
+   if (vhost_notify(vq)) {
+   continue;
+   }
break;
+   }
if (in) {
vq_err(vq, Unexpected descriptor format for TX: 
   out %d, int %d\n, out, in);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 30708c6..67bfc08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -775,7 +775,7 @@ void vhost_no_notify(struct vhost_virtqueue *vq)
 
 int vhost_init(void)
 {
-   vhost_workqueue = create_workqueue(vhost);
+   vhost_workqueue = create_singlethread_workqueue(vhost);
if (!vhost_workqueue)
return -ENOMEM;
return 0;
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Tue, 2009-10-27 at 22:58 +0200, Michael S. Tsirkin wrote:
 How large is large here? I usually allocate 1G.

I used to have 512, for this run I allocated 1G.

  I do see performance improves to 3xxxMb/s, and occasionally
  reaches 40xxMb/s.
 
 This is same as userspace, isn't it?

A little bit better than userspace.

  But queue full still exists, I can avoid the problem
  by increasing send queue size from qemu.
 
 And what performance do you get then?

5-10% better than userspace.

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

On Wed, 2009-10-28 at 17:39 +0200, Michael S. Tsirkin wrote:
 Here's another hack to try. It will break raw sockets,
 but just as a test:

This patch looks better than previous one for guest to host TCP_STREAM
performance. The transmission queue full still exists, but TCP_STREAM
results is 43xxMb/s (userspace is about 3500Mb/s). When I increase the
transmission queue size to 1K, The performance burst up to 53xxMb/s. But
some runs I still hit several transmission queue full in 60 secs run
even with 1K queue size.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

kvm memory leak (Was Re: vhost-net patches)

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
  Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
  Is there a chance you are tight on guest memory for some reason?
  with vhost, virtio does currently consume a bit more memory than
  with userspace backend.
 
 I did see memory leak on host every time after exiting guest. I don't
 know where. Do you see it?

I don't see a leak specific to vhost. However, even without vhost,
starting up guest increases kmalloc-128 counter in /proc/slabinfo
and this fails to go down when qemu is killed.

Anyone else seeing this?

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 28, 2009 at 09:45:37AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 On Wed, 2009-10-28 at 17:39 +0200, Michael S. Tsirkin wrote:
  Here's another hack to try. It will break raw sockets,
  but just as a test:
 
 This patch looks better than previous one for guest to host TCP_STREAM
 performance. The transmission queue full still exists, but TCP_STREAM
 results is 43xxMb/s (userspace is about 3500Mb/s). When I increase the
 transmission queue size to 1K,
 The performance burst up to 53xxMb/s. But
 some runs I still hit several transmission queue full in 60 secs run
 even with 1K queue size.

what exactly do you mean by transmission queue size?
tx_queue_len?
I think what should help with transmission queue full is
actually sndbuf parameter for tap in qemu.

 
 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Arnd Bergmann]

On Monday 26 October 2009, Shirley Ma wrote:
 On Sun, 2009-10-25 at 11:11 +0200, Michael S. Tsirkin wrote:
  What is vnet0?
 
 That's a tap interface. I am binding raw socket to a tap interface and
 it doesn't work. Does it support?


Is the tap device connected to a bridge as you'd normally do
with qemu? That won't work because then the data you send to
the socket will be queued at the /dev/tun chardev.

You can probably connect it like this:

qemu - vhost_net - vnet0 == /dev/tun - qemu

To connect two guests.

I've also used a bidirectional pipe before, to connect two tap
interfaces to each other. However, if you want to connect to
a bridge, the easier interface would be to use a veth pair,
with one end on the bridge and the other end used for the packet
socket.

Arnd 
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Miachel,

On Wed, 2009-10-28 at 18:53 +0200, Michael S. Tsirkin wrote:
 what exactly do you mean by transmission queue size?
 tx_queue_len?
 I think what should help with transmission queue full is
 actually sndbuf parameter for tap in qemu.

I didn't see my email out, I resend the response here.
What I meant is the number of descriptors in transmission queue. To be
more specific, it is 256-1024 in hw/virtio-net.c file

   n-tx_vq = virtio_add_queue(n-vdev, 256, virtio_net_handle_tx);

I had a patch to have trans/recv descriptors configurable rather than
fixed before. But the test results doens't show any difference for both
TCP and UDP, so I didn't submit the patch. I think it's better to make
it configurable based on how much memory virtio-net can be used. 

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Arnd,

On Wed, 2009-10-28 at 18:46 +0100, Arnd Bergmann wrote:
 You can probably connect it like this:
 
 qemu - vhost_net - vnet0 == /dev/tun - qemu
 
 To connect two guests.
 
 I've also used a bidirectional pipe before, to connect two tap
 interfaces to each other. However, if you want to connect to
 a bridge, the easier interface would be to use a veth pair,
 with one end on the bridge and the other end used for the packet
 socket.

Thanks for the tip, I will make a try.

Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

When I am testing deferring skb allocation patch, I found this problem.
Simply removing and reloading guest virtio_net module would cause guest
exit with errors. It is easy to reproduce it:

[r...@localhost ~]# rmmod virtio_net

[r...@localhost ~]# modprobe virtio_net

vhost_net_init returned -17

Any idea?

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 26, 2009 at 02:34:49PM -0700, Shirley Ma wrote:
 Hello Miachel,
 
 On Mon, 2009-10-26 at 22:05 +0200, Michael S. Tsirkin wrote:
  Shirley, could you please test the following patch?
 
 With this patch, the performance has gained from 1xxx to 2xxx Mb/s,
 still has some performance gap compared to without vhost. It was
 3xxxMb/s before from guest to host on my set up.

At some point my guest had a runaway nash-hotplug process
consuming 100% CPU. Could you please verify this
does not happen to you?

 Looks like your git tree virtio_net has fixed the skb_xmit panic I have
 seen before as well, good news.
 
 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

On Tue, 2009-10-27 at 08:43 +0200, Michael S. Tsirkin wrote:
 At some point my guest had a runaway nash-hotplug process
 consuming 100% CPU. Could you please verify this
 does not happen to you?

What I have found that the start_xmit stopped and restarted too often.
There is no vring descriptor available for adding the new buf. The buf
release is not able to keep up after vhost patch?

I saw lots of Unexpected full queue from dmesg. I haven't had time to
debug it yet. You might have some idea here?

/* This can happen with OOM and indirect buffers. */
if (unlikely(capacity  0)) {
netif_stop_queue(dev);
dev_warn(dev-dev, Unexpected full queue\n);
if (unlikely(!vi-svq-vq_ops-enable_cb(vi-svq))) {
vi-svq-vq_ops-disable_cb(vi-svq);
netif_start_queue(dev);
goto again;
}
return NETDEV_TX_BUSY;
}

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Tue, 2009-10-27 at 08:38 +0200, Michael S. Tsirkin wrote:
 Yes but you need to make host send packets out to tap as well,
 somehow. One way to do this is to assign IP address in
 a separate subnet to tap in host and to eth device in guest.

Thanks for the hint, I will make a try.

Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 07:46:59AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 On Tue, 2009-10-27 at 08:43 +0200, Michael S. Tsirkin wrote:
  At some point my guest had a runaway nash-hotplug process
  consuming 100% CPU. Could you please verify this
  does not happen to you?
 
 What I have found that the start_xmit stopped and restarted too often.
 There is no vring descriptor available for adding the new buf. The buf
 release is not able to keep up after vhost patch?

Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
Is there a chance you are tight on guest memory for some reason?
with vhost, virtio does currently consume a bit more memory than
with userspace backend.

 I saw lots of Unexpected full queue from dmesg. I haven't had time to
 debug it yet. You might have some idea here?
 
 /* This can happen with OOM and indirect buffers. */
 if (unlikely(capacity  0)) {
 netif_stop_queue(dev);
 dev_warn(dev-dev, Unexpected full queue\n);
 if (unlikely(!vi-svq-vq_ops-enable_cb(vi-svq))) {
 vi-svq-vq_ops-disable_cb(vi-svq);
 netif_start_queue(dev);
 goto again;
 }
 return NETDEV_TX_BUSY;
 }
 
 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote:
  Possibly GFP_ATOMIC allocations in vring_add_indirect are failing?
  Is there a chance you are tight on guest memory for some reason?
  with vhost, virtio does currently consume a bit more memory than
  with userspace backend.
 
 I did see memory leak on host every time after exiting guest. I don't
 know where. Do you see it?

I didn't notice. I'll check this.

 Anyway after I reboot host and restart guest with large memory
 allocation,

How large is large here? I usually allocate 1G.

 I do see performance improves to 3xxxMb/s, and occasionally
 reaches 40xxMb/s.

This is same as userspace, isn't it?

 But queue full still exists, I can avoid the problem
 by increasing send queue size from qemu.

And what performance do you get then?

 I will apply deferring skb allocation patch on guest to see any
 performance gain after your vhost patch.
 
 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
 I also hit guest skb_xmit panic.

If these are the same panics I have seen myself,
they are probably fixed with recent virtio patches
I sent to Rusty. I put them on my vhost.git tree to make
it easier for you to test.
If you see any more crashes, please holler, preferably
with a backtrace.

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 Some initial vhost test netperf results on my T61 laptop from the
 working tap device are here, latency has been significant decreased, but
 throughput from guest to host has huge regression. I also hit guest
 skb_xmit panic.
 
 netperf TCP_STREAM, default setup, 60 secs run
 
 guest-host drops from 3XXXMb/s to 1XXXMb/s (regression)
 host-guest increases from 3XXXMb/s to 4Mb/s 
 
 TCP_RR, 60 secs run (very impressive)
 
 guest-host trans/s increases from 2XXX/s to 13XXX/s
 host-guest trans/s increases from 2XXX/s to 13XXX/s
 
 Thanks
 Shirley

Shirley, could you please test the following patch?
It is surprising to me that it should improve
performance, but seems to do this in my setup.
Please comment.


diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 30708c6..67bfc08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -775,7 +775,7 @@ void vhost_no_notify(struct vhost_virtqueue *vq)
 
 int vhost_init(void)
 {
-   vhost_workqueue = create_workqueue(vhost);
+   vhost_workqueue = create_singlethread_workqueue(vhost);
if (!vhost_workqueue)
return -ENOMEM;
return 0;
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index a140dad..49026bb 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -106,10 +106,14 @@ static void handle_tx(struct vhost_net *net)
.msg_flags = MSG_DONTWAIT,
};
size_t len, total_len = 0;
-   int err;
+   int err, wmem;
size_t hdr_size;
struct socket *sock = rcu_dereference(vq-private_data);
-   if (!sock || !sock_writeable(sock-sk))
+   if (!sock)
+   return;
+
+   wmem = atomic_read(sock-sk-sk_wmem_alloc);
+   if (wmem = sock-sk-sk_sndbuf)
return;
 
use_mm(net-dev.mm);
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Miachel,

On Mon, 2009-10-26 at 22:05 +0200, Michael S. Tsirkin wrote:
 Shirley, could you please test the following patch?

With this patch, the performance has gained from 1xxx to 2xxx Mb/s,
still has some performance gap compared to without vhost. It was
3xxxMb/s before from guest to host on my set up.

Looks like your git tree virtio_net has fixed the skb_xmit panic I have
seen before as well, good news.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Pulled your git tree, didn't see the panic.

Thanks
Shirley


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Sun, 2009-10-25 at 11:11 +0200, Michael S. Tsirkin wrote:
 What is vnet0?

That's a tap interface. I am binding raw socket to a tap interface and
it doesn't work. Does it support?

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 12:24:15PM -0700, Shirley Ma wrote:
 Hello Michael,
 
 Some update,
 
 On Fri, 2009-10-23 at 08:12 -0700, Shirley Ma wrote:
  Tested raw packet, it didn't work; 
 
 Tested option -net raw,ifname=eth0, attached to a real device, raw works
 to remote node. I was expecting raw worked to local host. 
 
 Does this option -net raw,ifname=vnet0 support?

What is vnet0?

 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
 Hello Michael,
 
 Some initial vhost test netperf results on my T61 laptop from the
 working tap device are here, latency has been significant decreased, but
 throughput from guest to host has huge regression.

Could you please try adding ,sndbuf=0 to tap options, and re-testing?
This would make the -net option look like something like this:
-net tap,vlan=1,ifname=msttap0,script=/home/mst/ifup,downscript=no,sndbuf=0

 I also hit guest skb_xmit panic.

I have not seen panics in a while.
Could you provide more information please?
Is the panic reproducible?
Do you see one of these messages:
 id %u out of range
 id %u is not a head!


 
 netperf TCP_STREAM, default setup, 60 secs run
 
 guest-host drops from 3XXXMb/s to 1XXXMb/s (regression)
 host-guest increases from 3XXXMb/s to 4Mb/s 
 
 TCP_RR, 60 secs run (very impressive)
 
 guest-host trans/s increases from 2XXX/s to 13XXX/s
 host-guest trans/s increases from 2XXX/s to 13XXX/s
 
 Thanks
 Shirley
 
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Fri, Oct 23, 2009 at 09:23:40AM -0700, Shirley Ma wrote:
 I also hit guest skb_xmit panic.

OK, I have fixed a couple of reasons for panic. Will push and
look at host to guest performance soon.

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 11:00:20AM -0700, Sridhar Samudrala wrote:
 On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:
 
  
  Possibly we'll have to debug this in vhost in host kernel.
  I would debug this directly, it's just that my setup is somehow
  different and I do not see this issue, otherwise I would not
  waste your time.
  
  Can we add some printks?
  handle_tx has this at the top:
  
  if (!sock || !sock_writeable(sock-sk))
  return;
 
 I added some debug printks in handle_rx and handle_tx
 get_user() calls are failing with EFAULT.
 
 Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_rx: net:f5494800
 Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to enable 
 notification: -14
 Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail 
 idx at 0002 ret:-14
 Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to disable 
 notification: -14
 Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail 
 idx at 0002 ret:-14

Aha, we are getting somewhere!
Thanks very much for looking into this.  This suggests strongly that
available index pointer is not set up correctly in vhost.
I would like to verify this and understand how it came to be
incorrect.

Sridhar, Shirley, could you please try this:
- make sure to enable CONFIG_DYNAMIC_DEBUG=y in .config
  (if it's not there, you'll have to rebuild kernel, or
   hack drivers/vhost/vhost.h to use pr_err instead of
   pr_debug).
- echo 'module vhost_net +p'  /sys/kernel/debug/dynamic_debug/control

the above will make vhost print address which we tried to access, on error,
and they will apear in the output of the dmesg command.

In addition, please apply the following debugging patch to qemu.
this will make it output data about available index on startup.
Thanks!


diff --git a/hw/vhost_net.c b/hw/vhost_net.c
index bc179ab..d651504 100644
--- a/hw/vhost_net.c
+++ b/hw/vhost_net.c
@@ -66,6 +66,11 @@ static int vhost_virtqueue_init(struct vhost_dev *dev,
r = ioctl(dev-control, VHOST_SET_VRING_AVAIL, addr);
if (r  0)
return -errno;
+
+   fprintf(stderr, VQ %d, avail va %p(=%llx), pa %llx, num %x size 
%llx\n,
+   idx, vq-avail, (unsigned long long)addr.user_addr,
+   (unsigned long long)q-vring.avail, (unsigned)q-vring.num, s);
+
s = l = offsetof(struct vring_used, ring) +
sizeof(struct vring_used_elem) * q-vring.num;
vq-used = cpu_physical_memory_map(q-vring.used, l, 1);
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 11:00:20AM -0700, Sridhar Samudrala wrote:
 On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:
 
  
  Possibly we'll have to debug this in vhost in host kernel.
  I would debug this directly, it's just that my setup is somehow
  different and I do not see this issue, otherwise I would not
  waste your time.
  
  Can we add some printks?
  handle_tx has this at the top:
  
  if (!sock || !sock_writeable(sock-sk))
  return;
 
 I added some debug printks in handle_rx and handle_tx
 get_user() calls are failing with EFAULT.

Sridhar, Shirley,
Could you please test the following patch?
It should fix a bug on 32 bit hosts - is this what you have?


diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 5b57697..9d856e9 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -286,11 +286,11 @@ static long vhost_set_vring(struct vhost_dev *d, int 
ioctl, void __user *argp)
r = -EOPNOTSUPP;
break;
}
-   if ((u64)(long)a.user_addr != a.user_addr) {
+   if ((u64)(unsigned long)a.user_addr != a.user_addr) {
r = -EFAULT;
break;
}
-   vq-desc = (void __user *)(long)a.user_addr;
+   vq-desc = (void __user *)(unsigned long)a.user_addr;
break;
case VHOST_SET_VRING_AVAIL:
r = copy_from_user(a, argp, sizeof a);
@@ -300,11 +300,11 @@ static long vhost_set_vring(struct vhost_dev *d, int 
ioctl, void __user *argp)
r = -EOPNOTSUPP;
break;
}
-   if ((u64)(long)a.user_addr != a.user_addr) {
+   if ((u64)(unsigned long)a.user_addr != a.user_addr) {
r = -EFAULT;
break;
}
-   vq-avail = (void __user *)(long)a.user_addr;
+   vq-avail = (void __user *)(unsigned long)a.user_addr;
/* Forget the cached index value. */
vq-avail_idx = vq-last_avail_idx;
break;
@@ -316,11 +316,11 @@ static long vhost_set_vring(struct vhost_dev *d, int 
ioctl, void __user *argp)
r = -EOPNOTSUPP;
break;
}
-   if ((u64)(long)a.user_addr != a.user_addr) {
+   if ((u64)(unsigned long)a.user_addr != a.user_addr) {
r = -EFAULT;
break;
}
-   vq-used = (void __user *)(long)a.user_addr;
+   vq-used = (void __user *)(unsigned long)a.user_addr;
r = init_used(vq);
if (r)
break;
@@ -389,7 +389,7 @@ static long vhost_set_vring(struct vhost_dev *d, int ioctl, 
void __user *argp)
 
if (pollstop  vq-handle_kick)
vhost_poll_flush(vq-poll);
-   return 0;
+   return r;
 }
 
 long vhost_dev_ioctl(struct vhost_dev *d, unsigned int ioctl, unsigned long 
arg)
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Fri, 2009-10-23 at 13:04 +0200, Michael S. Tsirkin wrote:
 Sridhar, Shirley,
 Could you please test the following patch?
 It should fix a bug on 32 bit hosts - is this what you have?

Yes, it's 32 bit host. I checked out your recent git tree. Looks like
the patch is already there, but vhost doesn't work. 

I will apply the debug one you sent out in previous email.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Tested raw packet, it didn't work; switching to tap device, it is
working. Qemu command is:

x86_64-softmmu/qemu-system-x86_64 -s /home/xma/images/fedora10-2-vm -m
512 -drive file=/home/xma/images/fedora10-2-vm,if=virtio,index=0,boot=on
-net tap,ifname=vnet0,script=no,downscript=no -net
nic,macaddr=54:52:00:35:e3:73,model=virtio,vhost -no-acpi

Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Some initial vhost test netperf results on my T61 laptop from the
working tap device are here, latency has been significant decreased, but
throughput from guest to host has huge regression. I also hit guest
skb_xmit panic.

netperf TCP_STREAM, default setup, 60 secs run

guest-host drops from 3XXXMb/s to 1XXXMb/s (regression)
host-guest increases from 3XXXMb/s to 4Mb/s 

TCP_RR, 60 secs run (very impressive)

guest-host trans/s increases from 2XXX/s to 13XXX/s
host-guest trans/s increases from 2XXX/s to 13XXX/s

Thanks
Shirley


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

Hello Michael,

Some update,

On Fri, 2009-10-23 at 08:12 -0700, Shirley Ma wrote:
 Tested raw packet, it didn't work; 

Tested option -net raw,ifname=eth0, attached to a real device, raw works
to remote node. I was expecting raw worked to local host. 

Does this option -net raw,ifname=vnet0 support?

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
 Hello Michael,
 
 There was a recent bugfix in qemu-kvm I pushed.
 Could you please verify that you have 
 cec75e39151e49cc90c849eab5d0d729667c9e68
 ?
 
 Yes, I cloned your qemu-kvm and kernel git. Here is git log output:
 
 commit cec75e39151e49cc90c849eab5d0d729667c9e68
 Author: Michael S. Tsirkin m...@redhat.com
 Date:   Wed Oct 21 12:34:36 2009 +0200
 
 vhost: use acked features, not all features
 
 This was recently reported without vhost, did not
 reproduce it here yet.
 And you say you do not see the above without vhost?
 
 I saw above with and without vhost, but the network worked with the errors
 without vhost.

Could you please supply more information?
What is the qemu command line you use?
What guest do you use?
How is the network configured?

Please attach tcpdump to interfaces in host and guest,
and run ping from guest.
Do you see any traffic? Does it look sane?



 Thanks
 Shirley
 
 
 Inactive hide details for Michael S. Tsirkin ---10/21/2009 01:24:58 PM---On
 Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley MaMichael S. Tsirkin 
 ---10/21/
 2009 01:24:58 PM---On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
 
 Michael S.[cid]   *
 TsirkinTo Shirley Ma/Beaverton/i...@ibmus
 m...@redhat.com   [cid]   *
 cc David Stevens/Beaverton/i...@ibmus,
 10/21/2009 01:19   kvm@vger.kernel.org, 
 s...@linux.vnet.ibm.com
 PM [cid]   *
Subject Re: vhost-net patches
**
 
 On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
  Hello Micahel,
 
  I have set up guest kernel 2.6.32-rc5 with MSI configured. Here are errors
 what
  I have got:
 
  1. First, qemu complained extboot.bin not found, I copied the file from
  optionrom/ dir to pc-bios/ dir, this problem is gone.
 
  2. Second, when guest boot up, it has lots of errors as below. Without vhost
  support, I still saw same errors but the guest interface can communicate 
  with
  host, but with vhost, it doesn't work.
 
 There was a ecent bugfix in qemu-kvm I pushed.
 Could you please verify that you have cec75e39151e49cc90c849eab5d0d729667c9e68
 ?
 
 
  I am posting the errors from /var/log/
  messages here:
 
  virtio-pci :00:03.0: can't find IRQ for PCI INT A; probably buggy MP
 table
  virtio-pci :00:04.0: can't find IRQ for PCI INT A; probably buggy MP
 table
  virtio-pci :00:04.0: irq 24 for MSI/MSI-X
  virtio-pci :00:04.0: irq 25 for MSI/MSI-X
  IRQ handler type mismatch for IRQ 1
  current handler: i8042
  Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
  Call Trace:
  __setup_irq+0x24c/0x2ac
  request_threaded_irq+0x113/0x179
  ? vring_interrupt+0x0/0x2f
  vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
  ? blk_done+0x0/0xa7 [virtio_blk]
  vp_find_vqs+0x1b/0x62 [virtio_pci]
  virtblk_probe+0xbd/0x3d0 [virtio_blk]
  ? sysfs_do_create_link+0xbb/0xfd
  ? blk_done+0x0/0xa7 [virtio_blk]
  ? add_status+0x1f/0x24
  virtio_dev_probe+0x91/0xb0
  driver_probe_device+0x79/0x105
  __driver_attach+0x43/0x5f
  bus_for_each_dev+0x3d/0x67
  driver_attach+0x14/0x16
  ? __driver_attach+0x0/0x5f
  bus_add_driver+0xa2/0x1c9
  driver_register+0x8b/0xeb
  ? init+0x0/0x24 [virtio_blk]
  register_virtio_driver+0x1f/0x22
  init+0x22/0x24 [virtio_blk]
  do_one_initcall+0x4c/0x13a
  sys_init_module+0xa7/0x1db
  syscall_call+0x7/0xb
  virtio-pci :00:04.0: irq 24 for MSI/MSI-X
  virtio-pci :00:04.0: irq 25 for MSI/MSI-X
 
  vda: vda1 vda2
  EXT3-fs: INFO: recovery required on readonly filesystem.
  EXT3-fs: write access will be enabled during recovery.
  kjournald starting. Commit interval 5 seconds
  EXT3-fs: recovery complete.
  EXT3-fs: mounted filesystem with writeback data mode.
  udevd version 127 started
  virtio-pci :00:03.0: irq 26 for MSI/MSI-X
  virtio-pci :00:03.0: irq 27 for MSI/MSI-X
  virtio-pci :00:03.0: irq 28 for MSI/MSI-X
  IRQ handler type mismatch for IRQ 1
  current handler: i8042
  Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
  Call Trace:
  __setup_irq+0x24c/0x2ac
  request_threaded_irq+0x113/0x179
  ? vring_interrupt+0x0/0x2f
  vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
  ? skb_recv_done+0x0/0x36 [virtio_net]
  vp_find_vqs+0x1b/0x62 [virtio_pci]
  virtnet_probe+0x265/0x347 [virtio_net]
  ? skb_recv_done+0x0/0x36 [virtio_net]
  ? skb_xmit_done+0x0/0x1e [virtio_net]
  ? add_status+0x1f/0x24
  virtio_dev_probe+0x91/0xb0
  driver_probe_device+0x79/0x105
  __driver_attach+0x43/0x5f
  bus_for_each_dev+0x3d/0x67
  driver_attach+0x14/0x16
  ? __driver_attach+0x0/0x5f
  bus_add_driver+0xa2/0x1c9
  driver_register+0x8b/0xeb
  ? init+0x0/0xf [virtio_net]
  register_virtio_driver+0x1f/0x22
  init+0xd/0xf [virtio_net]
  do_one_initcall+0x4c/0x13a
  sys_init_module+0xa7/0x1db
  syscall_call+0x7/0xb

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
 
 Hello Michael,
 
 There was a recent bugfix in qemu-kvm I pushed.
 Could you please verify that you have
 cec75e39151e49cc90c849eab5d0d729667c9e68 ?
 
 Yes, I cloned your qemu-kvm and kernel git.
  I am posting the errors from /var/log/
  messages here:
 
  virtio-pci :00:03.0: can't find IRQ for PCI INT A; probably buggy MP
 table
  virtio-pci :00:04.0: can't find IRQ for PCI INT A; probably buggy MP
 table
  virtio-pci :00:04.0: irq 24 for MSI/MSI-X
  virtio-pci :00:04.0: irq 25 for MSI/MSI-X
  IRQ handler type mismatch for IRQ 1


So I think these are not related.

  current handler: i8042
  Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
  Call Trace:
  __setup_irq+0x24c/0x2ac
  request_threaded_irq+0x113/0x179
  ? vring_interrupt+0x0/0x2f
  vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
  ? blk_done+0x0/0xa7 [virtio_blk]
  vp_find_vqs+0x1b/0x62 [virtio_pci]
  virtblk_probe+0xbd/0x3d0 [virtio_blk]
  ? sysfs_do_create_link+0xbb/0xfd
  ? blk_done+0x0/0xa7 [virtio_blk]
  ? add_status+0x1f/0x24
  virtio_dev_probe+0x91/0xb0
  driver_probe_device+0x79/0x105
  __driver_attach+0x43/0x5f
  bus_for_each_dev+0x3d/0x67
  driver_attach+0x14/0x16
  ? __driver_attach+0x0/0x5f
  bus_add_driver+0xa2/0x1c9
  driver_register+0x8b/0xeb
  ? init+0x0/0x24 [virtio_blk]
  register_virtio_driver+0x1f/0x22
  init+0x22/0x24 [virtio_blk]
  do_one_initcall+0x4c/0x13a
  sys_init_module+0xa7/0x1db
  syscall_call+0x7/0xb
  virtio-pci :00:04.0: irq 24 for MSI/MSI-X
  virtio-pci :00:04.0: irq 25 for MSI/MSI-X
 
  vda: vda1 vda2
  EXT3-fs: INFO: recovery required on readonly filesystem.
  EXT3-fs: write access will be enabled during recovery.
  kjournald starting. Commit interval 5 seconds
  EXT3-fs: recovery complete.
  EXT3-fs: mounted filesystem with writeback data mode.
  udevd version 127 started
  virtio-pci :00:03.0: irq 26 for MSI/MSI-X
  virtio-pci :00:03.0: irq 27 for MSI/MSI-X
  virtio-pci :00:03.0: irq 28 for MSI/MSI-X
  IRQ handler type mismatch for IRQ 1
  current handler: i8042
  Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
  Call Trace:
  __setup_irq+0x24c/0x2ac
  request_threaded_irq+0x113/0x179
  ? vring_interrupt+0x0/0x2f
  vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
  ? skb_recv_done+0x0/0x36 [virtio_net]
  vp_find_vqs+0x1b/0x62 [virtio_pci]
  virtnet_probe+0x265/0x347 [virtio_net]
  ? skb_recv_done+0x0/0x36 [virtio_net]
  ? skb_xmit_done+0x0/0x1e [virtio_net]
  ? add_status+0x1f/0x24
  virtio_dev_probe+0x91/0xb0
  driver_probe_device+0x79/0x105
  __driver_attach+0x43/0x5f
  bus_for_each_dev+0x3d/0x67
  driver_attach+0x14/0x16
  ? __driver_attach+0x0/0x5f
  bus_add_driver+0xa2/0x1c9
  driver_register+0x8b/0xeb
  ? init+0x0/0xf [virtio_net]
  register_virtio_driver+0x1f/0x22
  init+0xd/0xf [virtio_net]
  do_one_initcall+0x4c/0x13a
  sys_init_module+0xa7/0x1db
  syscall_call+0x7/0xb
  virtio-pci :00:03.0: irq 26 for MSI/MSI-X
  virtio-pci :00:03.0: irq 27 for MSI/MSI-X
 
  3. The guest interface is up, and cat /proc/interrupts outputs:
 
  24: 0 PCI-MSI-edge virtio1-config
  25: 2571 PCI-MSI-edge virtio1-virtqueues
  26: 0 PCI-MSI-edge virtio0-config
  27: 0 PCI-MSI-edge virtio0-virtqueues

Aha. I notice that we failed to allocate 3 vectors, and have fallen back
on 2 vectors here.  This also should work, but is not the configuration
that I am testing, and will be slower anyway.  Why are there 2 virtio
devices here? Is the second one for storage? Can you try switching to
another hard drive model please?


  Thanks
  Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
 
 Hello Michael,
 
 There was a recent bugfix in qemu-kvm I pushed.
 Could you please verify that you have
 cec75e39151e49cc90c849eab5d0d729667c9e68 ?
 
 Yes, I cloned your qemu-kvm and kernel git.


It seems that the errors you observe are a result of changes
in guest virtio. I work to fix these, for now please
revert this commit f68d24082e22ccee3077d11aeb6dc5354f0ca7f1
in guest kernel.

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 02:34:56PM +0200, Michael S. Tsirkin wrote:
 On Wed, Oct 21, 2009 at 04:59:20PM -0700, Shirley Ma wrote:
  
  Hello Michael,
  
  There was a recent bugfix in qemu-kvm I pushed.
  Could you please verify that you have
  cec75e39151e49cc90c849eab5d0d729667c9e68 ?
  
  Yes, I cloned your qemu-kvm and kernel git.
 
 
 It seems that the errors you observe are a result of changes
 in guest virtio. I work to fix these, for now please
 revert this commit f68d24082e22ccee3077d11aeb6dc5354f0ca7f1
 in guest kernel.

OK, I sent a patch that should fix the errors for you.
Could you please confirm, preferably on-list, whether
the patch makes the errors go away for you with
userspace virtio?

However, as I said previously, it's good to fix them but I think
they are unrelated to the fact that vhost does not work
for you.

Shirley, if you have the time, please send details
on your setup and observations on tcpdump output
in both guest and host with vhost enabled.

Thanks!

-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 15:13 +0200, Michael S. Tsirkin wrote:
 OK, I sent a patch that should fix the errors for you.
 Could you please confirm, preferably on-list, whether
 the patch makes the errors go away for you with
 userspace virtio?

Confirmed, your patch has fixed irq handler mismatch errors.

 However, as I said previously, it's good to fix them but I think
 they are unrelated to the fact that vhost does not work
 for you.

Yes, agreed. One observation is when I enable PCI MSI in guest kernel, I
found that even without vhost supportin host kernel the network doesn't
work either. So I think this is nothing related to vhost. I need to find
why PCI MSI doesn't work for me.

 Shirley, if you have the time, please send details
 on your setup and observations on tcpdump output
 in both guest and host with vhost enabled.

Yes, here are details:

HW and distros:
T61 laptop, cpu is 
vendor_id   : GenuineIntel
cpu family  : 6
model   : 15
model name  : Intel(R) Core(TM)2 Duo CPU T7700  @ 2.40GHz
stepping: 11
cpu MHz : 2393.902
cache size  : 4096 KB

Guest installed Fedora 10, Host installed Fedora 10 and updated to
Fedora11, both doesn't work with guest kernel MSI enabled w/i, w/o vhost
support.


Source trees:
clone your qemu-kvm and vhost git. host kernel is vhost git, guest
kernel 2.6.32-rc5

Compile option:
qemu: ./configure (default target-list = x86_64-softmmu)
guest kernel: PCI MSI, virio enabled
host kernel: vhost, kvm, evenfd all enabled

I tried tap, raw, vnet0, eth0, here are some qemu commandline example:

mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
-s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
1 -monitor pty -drive
file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
raw,ifname=eth0 -net nic,macaddr=54:52:00:46:26:80,model=virtio,vhost

mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
-s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
1 -monitor pty -drive
file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
raw,ifname=vnet0 -net nic,model,macaddr=54:52:00:46:26:80=virtio,vhost 

Please let me know what I can do next. I am eager to test the vhost
performance for both latency and bw.

tcpdump output:
When I ping from guest to host, I didn't see any packets from any
interfaces, not even an arp request.

When I ping from host to guest, guest interface didn't see any packets
either, not even an arp request.

Thanks for your help!
Shirley





--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 10:23 -0700, Shirley Ma wrote:
 Yes, agreed. One observation is when I enable PCI MSI in guest kernel,
 I
 found that even without vhost supportin host kernel the network
 doesn't
 work either. So I think this is nothing related to vhost. I need to
 find
 why PCI MSI doesn't work for me.

Let me clarify this, here is the table:

1. upstream qemu + guest kernel MSI support --- doesn't work
2. upstream qemu + guest kernel no MSI support --- work
3. mst qemu git tree + guest kernel MSI support - vhost --- work
4. mst qemu git tree + guest kernel MSI support + vhost -- doesn't work

I hope it's clear.

thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:32:55AM -0700, Shirley Ma wrote:
 On Thu, 2009-10-22 at 10:23 -0700, Shirley Ma wrote:
  Yes, agreed. One observation is when I enable PCI MSI in guest kernel,
  I
  found that even without vhost supportin host kernel the network
  doesn't
  work either. So I think this is nothing related to vhost. I need to
  find
  why PCI MSI doesn't work for me.
 
 Let me clarify this, here is the table:
 
 1. upstream qemu + guest kernel MSI support --- doesn't work
 2. upstream qemu + guest kernel no MSI support --- work
 3. mst qemu git tree + guest kernel MSI support - vhost --- work
 4. mst qemu git tree + guest kernel MSI support + vhost -- doesn't work

Upstream is Avi's qemu-kvm.git?
So, for a moment taking vhost out of the equation, it seems that MSI was
broken in Avi's tree again, after I forked my tree?


 
 I hope it's clear.
 
 thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 19:36 +0200, Michael S. Tsirkin wrote:
 Upstream is Avi's qemu-kvm.git?
 So, for a moment taking vhost out of the equation, it seems that MSI
 was
 broken in Avi's tree again, after I forked my tree?

The upper stream qemu git tree never worked for me w/i MSI, the boot
hung on booting. I tried to debug this, hasn't found any hint. But
Sridhar has same configure, it works for him.

But with your git tree, guest kernel works without vhost; it doesn't
work with vhost. So the vhost might have some issues.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:23:44AM -0700, Shirley Ma wrote:
 On Thu, 2009-10-22 at 15:13 +0200, Michael S. Tsirkin wrote:
  OK, I sent a patch that should fix the errors for you.
  Could you please confirm, preferably on-list, whether
  the patch makes the errors go away for you with
  userspace virtio?
 
 Confirmed, your patch has fixed irq handler mismatch errors.
 
  However, as I said previously, it's good to fix them but I think
  they are unrelated to the fact that vhost does not work
  for you.
 
 Yes, agreed. One observation is when I enable PCI MSI in guest kernel, I
 found that even without vhost supportin host kernel the network doesn't
 work either.

Yes, but that's on Avi's qemu-kvm.git, so probably merge problems again.

 So I think this is nothing related to vhost. I need to find
 why PCI MSI doesn't work for me.

Probably just bisect until you find the bad commit.

  Shirley, if you have the time, please send details
  on your setup and observations on tcpdump output
  in both guest and host with vhost enabled.
 
 Yes, here are details:
 
 HW and distros:
 T61 laptop, cpu is 
 vendor_id : GenuineIntel
 cpu family: 6
 model : 15
 model name: Intel(R) Core(TM)2 Duo CPU T7700  @ 2.40GHz
 stepping  : 11
 cpu MHz   : 2393.902
 cache size: 4096 KB
 
 Guest installed Fedora 10, Host installed Fedora 10 and updated to
 Fedora11, both doesn't work with guest kernel MSI enabled w/i, w/o vhost
 support.

So, IIUC, my tree with MSI without vhost works?

 
 Source trees:
 clone your qemu-kvm and vhost git. host kernel is vhost git, guest
 kernel 2.6.32-rc5
 
 Compile option:
 qemu: ./configure (default target-list = x86_64-softmmu)
 guest kernel: PCI MSI, virio enabled
 host kernel: vhost, kvm, evenfd all enabled
 
 I tried tap, raw, vnet0, eth0,

what are vnet0 and eth0?
Do you have a bridge in host?
What do brctl show and ifconfig -a show in host?

 here are some qemu commandline example:
 
 mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
 -s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
 1 -monitor pty -drive
 file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
 raw,ifname=eth0 -net nic,macaddr=54:52:00:46:26:80,model=virtio,vhost
 
 mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64
 -s /home/xma/images/fedora10-2-vm -name guest2 -boot c -M pc -m 512 -smp
 1 -monitor pty -drive
 file=/home/xma/images/fedora10-2-vm,if=virtio,boot=on -net
 raw,ifname=vnet0 -net nic,model,macaddr=54:52:00:46:26:80=virtio,vhost 
 
 Please let me know what I can do next. I am eager to test the vhost
 performance for both latency and bw.
 tcpdump output:
 When I ping from guest to host, I didn't see any packets from any
 interfaces, not even an arp request.
 
 When I ping from host to guest, guest interface didn't see any packets
 either, not even an arp request.

Did you try tcpdump on the tap interface in host?

 Thanks for your help!
 Shirley
 
 
 
 

Possibly we'll have to debug this in vhost in host kernel.
I would debug this directly, it's just that my setup is somehow
different and I do not see this issue, otherwise I would not
waste your time.

Can we add some printks?
handle_tx has this at the top:

if (!sock || !sock_writeable(sock-sk))
return;

Could you please add

 printk(KERN_ERR %s: !sock = %d, !sock || !sock_writeable(sock-sk) =
%d,
__func__, !sock , !sock || !sock_writeable(sock-sk));

*Before* these checks?
Then make modules modules_install
rmmod vhost_net
insmod vhost_net
and re-run.
If you want me to send a patch that does this, let me know please.

Thanks!
-- 
MST
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 10:44:29AM -0700, Shirley Ma wrote:
 On Thu, 2009-10-22 at 19:36 +0200, Michael S. Tsirkin wrote:
  Upstream is Avi's qemu-kvm.git?
  So, for a moment taking vhost out of the equation, it seems that MSI
  was
  broken in Avi's tree again, after I forked my tree?
 
 The upper stream qemu git tree never worked for me w/i MSI, the boot
 hung on booting. I tried to debug this, hasn't found any hint. But
 Sridhar has same configure, it works for him.
 
 But with your git tree, guest kernel works without vhost;

What happens if you reset my tree to commit
47e465f031fc43c53ea8f08fa55cc3482c6435c8?

 it doesn't work with vhost.
 So the vhost might have some issues.

Looks like there are 2 issues:
- upstream issue
- vhost issue
Let's try to debug both.

 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 19:47 +0200, Michael S. Tsirkin wrote:
 What happens if you reset my tree to commit
 47e465f031fc43c53ea8f08fa55cc3482c6435c8?

I am going to clean up my upperstream git tree and retest first. Then I
will try back up this commit.

 Looks like there are 2 issues:
 - upstream issue
 - vhost issue
 Let's try to debug both.

Sounds good.

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Sridhar Samudrala]

On Thu, 2009-10-22 at 19:43 +0200, Michael S. Tsirkin wrote:

 
 Possibly we'll have to debug this in vhost in host kernel.
 I would debug this directly, it's just that my setup is somehow
 different and I do not see this issue, otherwise I would not
 waste your time.
 
 Can we add some printks?
 handle_tx has this at the top:
 
 if (!sock || !sock_writeable(sock-sk))
 return;

I added some debug printks in handle_rx and handle_tx
get_user() calls are failing with EFAULT.

Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_rx: net:f5494800
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to enable 
notification: -14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to disable 
notification: -14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:41 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:51:48 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:48 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:51:49 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:49 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:51:53 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:51:53 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:52:03 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:03 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:52:22 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:22 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:52:23 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:52:23 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:53:17 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:53:17 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14
Oct 21 10:56:56 IBM-19B5263ED41-009047018100 kernel: handle_tx: net:f5494800
Oct 21 10:56:56 IBM-19B5263ED41-009047018100 kernel: Failed to access avail idx 
at 0002 ret:-14

Thanks
Sridhar
 Could you please add
 
  printk(KERN_ERR %s: !sock = %d, !sock || !sock_writeable(sock-sk) =
 %d,
 __func__, !sock , !sock || !sock_writeable(sock-sk));
 
 *Before* these checks?
 Then make modules modules_install
 rmmod vhost_net
 insmod vhost_net
 and re-run.
 If you want me to send a patch that does this, let me know please.
 
 Thanks!

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Shirley Ma]

On Thu, 2009-10-22 at 10:56 -0700, Shirley Ma wrote:
 On Thu, 2009-10-22 at 19:47 +0200, Michael S. Tsirkin wrote:
  What happens if you reset my tree to commit
  47e465f031fc43c53ea8f08fa55cc3482c6435c8?
 
 I am going to clean up my upperstream git tree and retest first. Then
 I
 will try back up this commit.

Cleaned my local qemu upperstream git tree and guest kernel works with
MSI enabled.

So only vhost does seem to have some issues in my configuration. 

Thanks
Shirley

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Wed, Oct 21, 2009 at 12:59:50PM -0700, Shirley Ma wrote:
 Hello Micahel,
 
 I have set up guest kernel 2.6.32-rc5 with MSI configured. Here are errors 
 what
 I have got:
 
 1. First, qemu complained extboot.bin not found, I copied the file from
 optionrom/ dir to pc-bios/ dir, this problem is gone.
 
 2. Second, when guest boot up, it has lots of errors as below. Without vhost
 support, I still saw same errors but the guest interface can communicate with
 host, but with vhost, it doesn't work.

There was a ecent bugfix in qemu-kvm I pushed.
Could you please verify that you have cec75e39151e49cc90c849eab5d0d729667c9e68 ?


 I am posting the errors from /var/log/
 messages here:
 
 virtio-pci :00:03.0: can't find IRQ for PCI INT A; probably buggy MP table
 virtio-pci :00:04.0: can't find IRQ for PCI INT A; probably buggy MP table
 virtio-pci :00:04.0: irq 24 for MSI/MSI-X
 virtio-pci :00:04.0: irq 25 for MSI/MSI-X
 IRQ handler type mismatch for IRQ 1
 current handler: i8042
 Pid: 335, comm: modprobe Not tainted 2.6.32-rc5 #3
 Call Trace:
 __setup_irq+0x24c/0x2ac
 request_threaded_irq+0x113/0x179
 ? vring_interrupt+0x0/0x2f
 vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
 ? blk_done+0x0/0xa7 [virtio_blk]
 vp_find_vqs+0x1b/0x62 [virtio_pci]
 virtblk_probe+0xbd/0x3d0 [virtio_blk]
 ? sysfs_do_create_link+0xbb/0xfd
 ? blk_done+0x0/0xa7 [virtio_blk]
 ? add_status+0x1f/0x24
 virtio_dev_probe+0x91/0xb0
 driver_probe_device+0x79/0x105
 __driver_attach+0x43/0x5f
 bus_for_each_dev+0x3d/0x67
 driver_attach+0x14/0x16
 ? __driver_attach+0x0/0x5f
 bus_add_driver+0xa2/0x1c9
 driver_register+0x8b/0xeb
 ? init+0x0/0x24 [virtio_blk]
 register_virtio_driver+0x1f/0x22
 init+0x22/0x24 [virtio_blk]
 do_one_initcall+0x4c/0x13a
 sys_init_module+0xa7/0x1db
 syscall_call+0x7/0xb
 virtio-pci :00:04.0: irq 24 for MSI/MSI-X
 virtio-pci :00:04.0: irq 25 for MSI/MSI-X

This was recently reported without vhost, did not
reproduce it here yet.
And you say you do not see the above without vhost?

 vda: vda1 vda2
 EXT3-fs: INFO: recovery required on readonly filesystem.
 EXT3-fs: write access will be enabled during recovery.
 kjournald starting. Commit interval 5 seconds
 EXT3-fs: recovery complete.
 EXT3-fs: mounted filesystem with writeback data mode.
 udevd version 127 started
 virtio-pci :00:03.0: irq 26 for MSI/MSI-X
 virtio-pci :00:03.0: irq 27 for MSI/MSI-X
 virtio-pci :00:03.0: irq 28 for MSI/MSI-X
 IRQ handler type mismatch for IRQ 1
 current handler: i8042
 Pid: 440, comm: modprobe Not tainted 2.6.32-rc5 #3
 Call Trace:
 __setup_irq+0x24c/0x2ac
 request_threaded_irq+0x113/0x179
 ? vring_interrupt+0x0/0x2f
 vp_try_to_find_vqs+0x4a3/0x4e0 [virtio_pci]
 ? skb_recv_done+0x0/0x36 [virtio_net]
 vp_find_vqs+0x1b/0x62 [virtio_pci]
 virtnet_probe+0x265/0x347 [virtio_net]
 ? skb_recv_done+0x0/0x36 [virtio_net]
 ? skb_xmit_done+0x0/0x1e [virtio_net]
 ? add_status+0x1f/0x24
 virtio_dev_probe+0x91/0xb0
 driver_probe_device+0x79/0x105
 __driver_attach+0x43/0x5f
 bus_for_each_dev+0x3d/0x67
 driver_attach+0x14/0x16
 ? __driver_attach+0x0/0x5f
 bus_add_driver+0xa2/0x1c9
 driver_register+0x8b/0xeb
 ? init+0x0/0xf [virtio_net]
 register_virtio_driver+0x1f/0x22
 init+0xd/0xf [virtio_net]
 do_one_initcall+0x4c/0x13a
 sys_init_module+0xa7/0x1db
 syscall_call+0x7/0xb
 virtio-pci :00:03.0: irq 26 for MSI/MSI-X
 virtio-pci :00:03.0: irq 27 for MSI/MSI-X
 
 3. The guest interface is up, and cat /proc/interrupts outputs:
 
 24: 0 PCI-MSI-edge virtio1-config
 25: 2571 PCI-MSI-edge virtio1-virtqueues
 26: 0 PCI-MSI-edge virtio0-config
 27: 0 PCI-MSI-edge virtio0-virtqueues
 
 Thanks
 Shirley
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 19, 2009 at 03:56:54PM -0700, Sridhar Samudrala wrote:
 On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
  On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
   On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
Hi Michael,

We are trying out your vhost-net patches from your git trees on 
kernel.org.
I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.

I am using the following qemu script to start the guest using userspace 
tap backend.

home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 
/home/sridhar/kvm_images/fedora10-1-vm -m 512 -drive 
file=/home/sridhar/kvm_images/fedora10-1-vm,if=virtio,index=0,boot=on 
-net nic,macaddr=54:52:00:35:e3:73,model=virtio -net 
tap,ifname=vnet0,script=no,downscript=no

Now that i got the default backend to work, i wanted to try vhost in 
kernel. But
could not figure out the right -net option to pass to qemu.

Can you let me know the right syntax to start a guest using vhost.

Thanks
Sridhar
   
   Here's an example with raw socket:
   
   /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
   /boot/vmlinuz-$release -append \
   'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
   console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
   $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
   -balloon none -redir tcp:8023::22
   
   As you see, I changed the command line.
   You now simply add ,vhost after model, and it will locate
   host network interface specified earlier and attach to it.
   This should have been clear from running  qemu with -help
   flag. Could you please suggest how can that text
   be clarified?
 
 I updated to your latest git trees and the default user-space tap backend 
 using the
 following -net options worked fine.
 -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
 
 But i could not get vhost to work with either raw or tap backends.
 I tried the following combinations.
 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
 3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost
 
 They all failed with the following error
 vhost_net_init returned -7
 This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
 vhost_net_start() fails. It looks like dev-binding-irqfd() is failing in
 vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
 enabled in the host kernel.
 
 Are all the above -net options supposed to work?
 
 In your descriptions, you say that checksum/tso offload is not supported.

They should work with tap but not raw sockets yet.

 Isn't it
 possible to send/receive large packets without checksum using AF_PACKET 
 sockets if
 the attached interface supports these offloads.
 Do you see the same offload issue even when using tap backend via vhost?
 
 Thanks
 Sridhar
 
 
 
 
 
 
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
 Hello Michael,
 
 They all failed with the following error
 vhost_net_init returned -7
 This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
 vhost_net_start() fails. It looks like dev-binding-irqfd() is failing in
 vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
 enabled in the host kernel.
 
 From the debug output, looks like the vnet-vector is not defined,

what is vnet-vector?
And what do you mean by not defined?

 and the
 default msix_entries_nr is 3, so it returned EINVAL from virtio_pci_irqfd.
 Looks we need to either disable QEMU_PCI_CAP_MSIX or define vector in QEMU
 configuration?

You shouldn't have to do anything.

 I am not familiar with MSIX stuffs.
 
 Thanks
 Shirley
 
 
 Inactive hide details for sri---10/19/2009 03:56:57 PM---On Sun, 2009-10-18 at
 19:32 +0200, Michael S. Tsirkin wrote:sri---10/19/2009 03:56:57 PM---On Sun,
 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
 
 s...@linux.vnet.ibm.com [cid]   *
 To Michael S. Tsirkin 
 m...@redhat.com,
 10/19/2009 03:56 PMkvm@vger.kernel.org
[cid]   *
 cc David 
 Stevens/Beaverton/i...@ibmus, Shirley Ma/
Beaverton/i...@ibmus
[cid]   *
Subject Re: vhost-net patches
**
 
 On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
  On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
   On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
Hi Michael,
   
We are trying out your vhost-net patches from your git trees on
 kernel.org.
I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.
   
I am using the following qemu script to start the guest using userspace
 tap backend.
   
home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/
 sridhar/kvm_images/fedora10-1-vm -m 512 -drive file=/home/sridhar/kvm_images/
 fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=
 54:52:00:35:e3:73,model=virtio -net tap,ifname=vnet0,script=no,downscript=no
   
Now that i got the default backend to work, i wanted to try vhost in
 kernel. But
could not figure out the right -net option to pass to qemu.
   
Can you let me know the right syntax to start a guest using vhost.
   
Thanks
Sridhar
  
   Here's an example with raw socket:
  
   /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
   /boot/vmlinuz-$release -append \
   'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
   console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
   $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
   -balloon none -redir tcp:8023::22
  
   As you see, I changed the command line.
   You now simply add ,vhost after model, and it will locate
   host network interface specified earlier and attach to it.
   This should have been clear from running  qemu with -help
   flag. Could you please suggest how can that text
   be clarified?
 
 I updated to your latest git trees and the default user-space tap backend 
 using
 the
 following -net options worked fine.
 -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio
 
 But i could not get vhost to work with either raw or tap backends.
 I tried the following combinations.
 1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
 2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
 3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost

Yes, should work.

 
 They all failed with the following error
vhost_net_init returned -7
 This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
 vhost_net_start() fails. It looks like dev-binding-irqfd() is failing in
 vhost_virtqueue_init().

what parameters are passed in?

 Haven't yet debugged further.

this calls into virtio_pci_irqfd.

 I have CONFIG_EVENTFD
 enabled in the host kernel.

Note you need to also enable eventfd support under kvm menu.

 Are all the above -net options supposed to work?
 
 In your descriptions, you say that checksum/tso offload is not supported. 
 Isn't
 it
 possible to send/receive large packets without checksum using AF_PACKET 
 sockets
 if
 the attached interface supports these offloads.
 Do you see the same offload issue even when using tap backend via vhost?
 
 Thanks
 Sridhar
 
 
 
 
 
 
 
 

--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 20, 2009 at 10:14:55AM -0700, Shirley Ma wrote:
 
 Hello Michael,
 
 what is vnet-vector?
 And what do you mean by not defined?
 
 In funcation:
 
 static int vhost_virtqueue_init()
 {
 ..
   r = vdev-binding-irqfd(vdev-binding_opaque, q-vector,
 vq-call);
 ..
 }.
 
 q-vector is 65535,

Thanks for debugging this.
I think this means that guest does not use MSI-X.

You can verify this by booting guest without vhost,
and performing the following command:
 cat /proc/interrupts

Please note that you currently need recent kernel in guest,
so that it uses MSI-X. I plan on implementing regular IRQ,
but not yet, and it will be slower anyway.

 in   static int virtio_pci_irqfd()
 {
 ..
 if (vector = proxy-pci_dev.msix_entries_nr) {
 fprintf(stderr,  pci irq fd returned vector %d, msix_entries_nr %d
 \n,
 vector, proxy-pci_dev.msix_entries_nr); --- I added
 one output line here.
 return -EINVAL;
 
 }...
 The output is:
 
pci irq fd returned vector 65535, msix_entries_nr 3, EINVAL is
 returned.
 
 thanks
 Shirley Ma
 IBM Linux Technology Center
 15300 SW Koll Parkway
 Beaverton, OR 97006-6063
 Phone(Fax): (503) 578-7638
 
 
 
 

  Michael S.   
  Tsirkin  
  m...@redhat.com   To
Shirley Ma/Beaverton/i...@ibmus  
  10/20/2009 04:34   cc
  AMs...@linux.vnet.ibm.com, David   
Stevens/Beaverton/i...@ibmus,
kvm@vger.kernel.org 
Subject
Re: vhost-net patches   






 
 
 
 
 On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
  Hello Michael,
 
  They all failed with the following error
  vhost_net_init returned -7
  This is an error message from hw/virtio-net.c:virtio_net_driver_ok()
 when
  vhost_net_start() fails. It looks like dev-binding-irqfd() is failing
 in
  vhost_virtqueue_init(). Haven't yet debugged further. I have
 CONFIG_EVENTFD
  enabled in the host kernel.
 
  From the debug output, looks like the vnet-vector is not defined,
 
 what is vnet-vector?
 And what do you mean by not defined?
 
  and the
  default msix_entries_nr is 3, so it returned EINVAL from
 virtio_pci_irqfd.
  Looks we need to either disable QEMU_PCI_CAP_MSIX or define vector in
 QEMU
  configuration?
 
 You shouldn't have to do anything.
 
  I am not familiar with MSIX stuffs.
 
  Thanks
  Shirley
 
 
  Inactive hide details for sri---10/19/2009 03:56:57 PM---On Sun,
 2009-10-18 at
  19:32 +0200, Michael S. Tsirkin wrote:sri---10/19/2009 03:56:57 PM---On
 Sun,
  2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
 
  s...@linux.vnet.ibm.com [cid]   *
  To Michael S. Tsirkin
 m...@redhat.com,
  10/19/2009 03:56 PMkvm@vger.kernel.org
 [cid]   *
  cc David
 Stevens/Beaverton/i...@ibmus, Shirley Ma/
 Beaverton/i...@ibmus
 [cid]   *
 Subject Re: vhost-net patches
 **
 
  On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
   On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
 Hi Michael,

 We are trying out your vhost-net patches from your git trees on
  kernel.org.
 I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for
 qemu.

 I am using the following qemu script to start the guest using
 userspace
  tap backend.


 home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 /home/
  sridhar/kvm_images/fedora10-1-vm -m 512 -drive
 file=/home/sridhar/kvm_images/
  fedora10-1-vm,if=virtio,index=0,boot=on -net nic,macaddr=
  54:52:00:35:e3:73,model=virtio -net
 tap,ifname=vnet0,script=no,downscript=no

 Now that i got the default backend to work, i

Re: vhost-net patches

[Michael S. Tsirkin]

On Tue, Oct 20, 2009 at 10:27:38AM -0700, Shirley Ma wrote:
 
 Hello Michael,
 
 Here are the output, I am using guest 2.6.32-rc3 kernel. It doesn't use
 MSIX. So which guest kernel I should use??
 
 [...@localhost ~]$ cat /proc/interrupts.
CPU0
   0:299   IO-APIC-edge  timer.
   1:  2   IO-APIC-edge  i8042.
   2:  0XT-PIC-XTcascade
   4: 76   IO-APIC-edge  serial
  11:   2126   IO-APIC-edge  virtio1, virtio0 - here is the
 virtio for both disk and network i/o??

Yes, this is regular shared IRQ, no good.
I think your guest is too old, please use kernel 2.6.31 and up in guest.
I will work to improve the error message as well.

  12: 89   IO-APIC-edge  i8042
 NMI:  0   Non-maskable interrupts
 LOC:   5146   Local timer interrupts
 SPU:  0   Spurious interrupts
 CNT:  0   Performance counter interrupts
 PND:  0   Performance pending work
 RES:  0   Rescheduling interrupts
 CAL:  0   Function call interrupts
 TLB:  0   TLB shootdowns
 TRM:  0   Thermal event interrupts
 MCE:  0   Machine check exceptions
 MCP:  1   Machine check polls
 ERR:  0
 MIS:  0
 [...@localhost ~]$ uname -r
 2.6.32-rc3
 
 
 Shirley Ma
 IBM Linux Technology Center
 15300 SW Koll Parkway
 Beaverton, OR 97006-6063
 Phone(Fax): (503) 578-7638
 
 
 
 

  Michael S.   
  Tsirkin  
  m...@redhat.com   To
Shirley Ma/Beaverton/i...@ibmus  
  10/20/2009 10:18   cc
  AMDavid Stevens/Beaverton/i...@ibmus,
kvm@vger.kernel.org,
s...@linux.vnet.ibm.com  
Subject
Re: vhost-net patches   






 
 
 
 
 On Tue, Oct 20, 2009 at 10:14:55AM -0700, Shirley Ma wrote:
 
  Hello Michael,
 
  what is vnet-vector?
  And what do you mean by not defined?
 
  In funcation:
 
  static int vhost_virtqueue_init()
  {
  ..
r = vdev-binding-irqfd(vdev-binding_opaque, q-vector,
  vq-call);
  ..
  }.
 
  q-vector is 65535,
 
 Thanks for debugging this.
 I think this means that guest does not use MSI-X.
 
 You can verify this by booting guest without vhost,
 and performing the following command:
   cat /proc/interrupts
 
 Please note that you currently need recent kernel in guest,
 so that it uses MSI-X. I plan on implementing regular IRQ,
 but not yet, and it will be slower anyway.
 
  in   static int virtio_pci_irqfd()
  {
  ..
  if (vector = proxy-pci_dev.msix_entries_nr) {
  fprintf(stderr,  pci irq fd returned vector %d, msix_entries_nr
 %d
  \n,
  vector, proxy-pci_dev.msix_entries_nr); --- I added
  one output line here.
  return -EINVAL;
  
  }...
  The output is:
 
 pci irq fd returned vector 65535, msix_entries_nr 3, EINVAL is
  returned.
 
  thanks
  Shirley Ma
  IBM Linux Technology Center
  15300 SW Koll Parkway
  Beaverton, OR 97006-6063
  Phone(Fax): (503) 578-7638
 
 
 
 
 
   Michael S.
   Tsirkin
   m...@redhat.com
 To
 Shirley Ma/Beaverton/i...@ibmus
   10/20/2009 04:34
 cc
   AMs...@linux.vnet.ibm.com, David
 Stevens/Beaverton/i...@ibmus,
 kvm@vger.kernel.org
 
 Subject
 Re: vhost-net patches
 
 
 
 
 
 
 
 
 
 
  On Mon, Oct 19, 2009 at 04:08:24PM -0700, Shirley Ma wrote:
   Hello Michael,
  
   They all failed with the following error
   vhost_net_init returned -7
   This is an error message from hw/virtio-net.c:virtio_net_driver_ok()
  when
   vhost_net_start() fails. It looks like dev-binding-irqfd() is
 failing
  in
   vhost_virtqueue_init(). Haven't yet debugged further. I have
  CONFIG_EVENTFD
   enabled in the host kernel.
  
   From the debug output, looks like the vnet-vector is not defined,
 
  what is vnet-vector

Re: vhost-net patches

[Sridhar Samudrala]

On Sun, 2009-10-18 at 19:32 +0200, Michael S. Tsirkin wrote:
 On Sun, Oct 18, 2009 at 12:53:56PM +0200, Michael S. Tsirkin wrote:
  On Fri, Oct 16, 2009 at 12:29:29PM -0700, Sridhar Samudrala wrote:
   Hi Michael,
   
   We are trying out your vhost-net patches from your git trees on 
   kernel.org.
   I am using mst/vhost.git as host kernel and mst/qemu-kvm.git for qemu.
   
   I am using the following qemu script to start the guest using userspace 
   tap backend.
   
   home/sridhar/git/mst/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 
   /home/sridhar/kvm_images/fedora10-1-vm -m 512 -drive 
   file=/home/sridhar/kvm_images/fedora10-1-vm,if=virtio,index=0,boot=on 
   -net nic,macaddr=54:52:00:35:e3:73,model=virtio -net 
   tap,ifname=vnet0,script=no,downscript=no
   
   Now that i got the default backend to work, i wanted to try vhost in 
   kernel. But
   could not figure out the right -net option to pass to qemu.
   
   Can you let me know the right syntax to start a guest using vhost.
   
   Thanks
   Sridhar
  
  Here's an example with raw socket:
  
  /root/kvm-test/bin/qemu-system-x86_64 -m 1G -kernel \
  /boot/vmlinuz-$release -append \
  'root=UUID=d5d2d201-d086-42ad-bb1d-32fbe40eda71 ro quiet nosplash \
  console=tty0 console=ttyS0,9600n8' -initrd /boot/guest-initrd.img \
  $HOME/disk.raw.copy -net raw,ifname=eth3 -net nic,model=virtio,vhost \
  -balloon none -redir tcp:8023::22
  
  As you see, I changed the command line.
  You now simply add ,vhost after model, and it will locate
  host network interface specified earlier and attach to it.
  This should have been clear from running  qemu with -help
  flag. Could you please suggest how can that text
  be clarified?

I updated to your latest git trees and the default user-space tap backend using 
the
following -net options worked fine.
-net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio

But i could not get vhost to work with either raw or tap backends.
I tried the following combinations.
1) -net raw,ifname=eth0 -net nic,model=virtio,vhost
2) -net raw,ifname=vnet0, -net nic,model=virtio,vhost
3) -net tap,ifname=vnet0,script=no,downscript=no -net nic,model=virtio,vhost

They all failed with the following error
vhost_net_init returned -7
This is an error message from hw/virtio-net.c:virtio_net_driver_ok() when
vhost_net_start() fails. It looks like dev-binding-irqfd() is failing in
vhost_virtqueue_init(). Haven't yet debugged further. I have CONFIG_EVENTFD
enabled in the host kernel.

Are all the above -net options supposed to work?

In your descriptions, you say that checksum/tso offload is not supported. Isn't 
it
possible to send/receive large packets without checksum using AF_PACKET sockets 
if
the attached interface supports these offloads.
Do you see the same offload issue even when using tap backend via vhost?

Thanks
Sridhar







--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html