Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
the patch has solved my problem. thanks a lot On Wed, Apr 25, 2012 at 10:52 AM, junaid khalid junaid.kha...@seecs.nust.edu.pk wrote: Thanks a lot. I will update you after testing it On Wed, Apr 25, 2012 at 10:45 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 10:38:45AM +0500, junaid khalid wrote: we are talking about the recv failed problem (system@br0: recv failed (No buffer space available)). Try applying the following patch from master to your tree. It may help. --8--cut here--8-- From: Ben Pfaff b...@nicira.com Date: Thu, 15 Mar 2012 21:15:38 -0700 Subject: [PATCH] netlink-socket: Increase Netlink socket receive buffer size. Open vSwitch userspace can set up flows at a high rate, but it is somewhat bursty in opportunities to set up flows, by which I mean that OVS sets up a batch of flows, then goes off and does some other work for a while, then sets up another batch of flows, and so on. The result is that, if a large number of packets that need flow setups come in all at once, then some of them can overflow the relatively small kernel-to-user buffers. This commit increases the kernel-to-user buffers from the default of approximately 120 kB each to 1 MB each. In one somewhat synthetic test case that I ran based on an hping3 that generated a load of about 20,000 new flows per second (including both requests and replies), this reduced the packets dropped at the kernel-to-user interface from about 30% to none. I expect that it will similarly improve packet loss in workloads where flow arrival is not easily predictable. (This has little effect on workloads generated by ovs-benchmark rate because that benchmark is effectively self-clocking, that is, a new flow is triggered only by a reply to a request made earlier, which means that the number of buffered packets at any given has a known, constant upper limit.) Bug #10210. Signed-off-by: Ben Pfaff b...@nicira.com --- include/sparse/sys/socket.h |5 +++-- lib/netlink-socket.c| 10 +- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/include/sparse/sys/socket.h b/include/sparse/sys/socket.h index 89e3c2d..1ed195b 100644 --- a/include/sparse/sys/socket.h +++ b/include/sparse/sys/socket.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2011 Nicira Networks. + * Copyright (c) 2011, 2012 Nicira Networks. * * Licensed under the Apache License, Version 2.0 (the License); * you may not use this file except in compliance with the License. @@ -74,7 +74,8 @@ enum { SO_SNDBUF, SO_SNDLOWAT, SO_SNDTIMEO, -SO_TYPE +SO_TYPE, +SO_RCVBUFFORCE }; enum { diff --git a/lib/netlink-socket.c b/lib/netlink-socket.c index bc46235..df6f1d8 100644 --- a/lib/netlink-socket.c +++ b/lib/netlink-socket.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008, 2009, 2010, 2011 Nicira Networks. + * Copyright (c) 2008, 2009, 2010, 2011, 2012 Nicira Networks. * * Licensed under the Apache License, Version 2.0 (the License); * you may not use this file except in compliance with the License. @@ -89,6 +89,7 @@ nl_sock_create(int protocol, struct nl_sock **sockp) struct nl_sock *sock; struct sockaddr_nl local, remote; socklen_t local_size; +int rcvbuf; int retval = 0; if (!max_iovs) { @@ -122,6 +123,13 @@ nl_sock_create(int protocol, struct nl_sock **sockp) sock-protocol = protocol; sock-dump = NULL; +rcvbuf = 1024 * 1024; +if (setsockopt(sock-fd, SOL_SOCKET, SO_RCVBUFFORCE, + rcvbuf, sizeof rcvbuf)) { +VLOG_WARN_RL(rl, setting %d-byte socket receive buffer failed (%s), + rcvbuf, strerror(errno)); +} + retval = get_socket_rcvbuf(sock-fd); if (retval 0) { retval = -retval; -- 1.7.2.5 ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
Hi, Do you mean the setup of flows in cached flow table in kernel module? We are sending packets after setting up the flows. Therefore, this problem should be in the fast path or in other words, between the ovs-vswitchd and kernel module. About traffic passing from the switch, MTU size UDP packets are passed from the switch; received from one interface and sent out from the other interface of NIC. On Tue, Apr 24, 2012 at 2:55 AM, Ben Pfaff b...@nicira.com wrote: On Mon, Apr 23, 2012 at 03:54:09PM +0600, junaid khalid wrote: 1) Apr 23 13:19:27|00019|dpif|WARN|system@br0: recv failed (No buffer space available) Packets are arriving at your interfaces faster than ovs-vswitchd can set up flows. What traffic is going through the switch? 2) Apr 23 13:19:37|00033|pktbuf|WARN|cookie mismatch: 01fa != 02fa OVS is sending buffered packets to your OpenFlow controller. The OpenFlow controller is sending back replies to use those buffers, but the replies are arriving slowly enough that by the time that they arrive the switch has already discarded those buffers. Apr 23 13:19:37|00034|connmgr|INFO|br0-tcp:127.0.0.1:6633: sending type OFPET_BAD_REQUEST, code OFPBRC_BUFFER_UNKNOWN error reply to OFPT_FLOW_MOD message Also a consequence of the above. traffic type: MTU size UDP packets @7.3 Gbps, generating 100,000 flows by changing src IPs. NOX isn't going to be able to keep up with that rate. OVS 1.4.0 isn't, either, but OVS 1.6.90 from the tip of master should be able to handle it without a controller. Throwing in an OpenFlow controller that sees every packet will probably bog things down a lot. Apr 23 13:19:16|00010|ofp_util|WARN|received Nicira extension message of unknown type 8 That message has been obsolete for ages, why are you using it? ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
On Tue, Apr 24, 2012 at 07:02:37PM +0600, junaid khalid wrote: Do you mean the setup of flows in cached flow table in kernel module? We are sending packets after setting up the flows. Therefore, this problem should be in the fast path or in other words, between the ovs-vswitchd and kernel module. You said you have 100,000 flows. How many packets in each flow? ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
On Tue, Apr 24, 2012 at 9:06 PM, Ben Pfaff b...@nicira.com wrote: On Tue, Apr 24, 2012 at 07:02:37PM +0600, junaid khalid wrote: Do you mean the setup of flows in cached flow table in kernel module? We are sending packets after setting up the flows. Therefore, this problem should be in the fast path or in other words, between the ovs-vswitchd and kernel module. You said you have 100,000 flows. How many packets in each flow? we are generating packets in a round robin fashion, approx. 6 packets per flow per sec. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
On Wed, Apr 25, 2012 at 10:14 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 09:43:50AM +0500, junaid khalid wrote: On Tue, Apr 24, 2012 at 9:06 PM, Ben Pfaff b...@nicira.com wrote: On Tue, Apr 24, 2012 at 07:02:37PM +0600, junaid khalid wrote: Do you mean the setup of flows in cached flow table in kernel module? We are sending packets after setting up the flows. Therefore, this problem should be in the fast path or in other words, between the ovs-vswitchd and kernel module. You said you have 100,000 flows. How many packets in each flow? we are generating packets in a round robin fashion, approx. 6 packets per flow per sec. You might want to increase the flow eviction threshold to 10 then. See the documentation in ovs-vswitchd.conf.db(5). We have also tried that. We set the flow eviction threshold to 100,000 and added a periodic print in the kernel module to check the number of entries in the flow table in kernel module. We noticed that, although the flow table size in userspace is 100,000 (from dump-aggregate command), but the flow table size in kernel module increases gradually and the error continues to come up till the kernel table is completely filled with 10 entries. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
On Wed, Apr 25, 2012 at 10:28:05AM +0500, junaid khalid wrote: On Wed, Apr 25, 2012 at 10:14 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 09:43:50AM +0500, junaid khalid wrote: On Tue, Apr 24, 2012 at 9:06 PM, Ben Pfaff b...@nicira.com wrote: On Tue, Apr 24, 2012 at 07:02:37PM +0600, junaid khalid wrote: Do you mean the setup of flows in cached flow table in kernel module? We are sending packets after setting up the flows. Therefore, this problem should be in the fast path or in other words, between the ovs-vswitchd and kernel module. You said you have 100,000 flows. How many packets in each flow? we are generating packets in a round robin fashion, approx. 6 packets per flow per sec. You might want to increase the flow eviction threshold to 10 then. See the documentation in ovs-vswitchd.conf.db(5). We have also tried that. We set the flow eviction threshold to 100,000 and added a periodic print in the kernel module to check the number of entries in the flow table in kernel module. We noticed that, although the flow table size in userspace is 100,000 (from dump-aggregate command), but the flow table size in kernel module increases gradually and the error continues to come up till the kernel table is completely filled with 10 entries. I'm not sure what's the error. If it's the one about unknown buffers, then that's probably because you've got some kind of packet loss between the switch and your controller. Or your controller can't keep up with 100,000 packet-ins in the first 166 ms (1/6 of a second)? Is the controller in-band? ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
Thanks a lot. I will update you after testing it On Wed, Apr 25, 2012 at 10:45 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 10:38:45AM +0500, junaid khalid wrote: we are talking about the recv failed problem (system@br0: recv failed (No buffer space available)). Try applying the following patch from master to your tree. It may help. --8--cut here--8-- From: Ben Pfaff b...@nicira.com Date: Thu, 15 Mar 2012 21:15:38 -0700 Subject: [PATCH] netlink-socket: Increase Netlink socket receive buffer size. Open vSwitch userspace can set up flows at a high rate, but it is somewhat bursty in opportunities to set up flows, by which I mean that OVS sets up a batch of flows, then goes off and does some other work for a while, then sets up another batch of flows, and so on. The result is that, if a large number of packets that need flow setups come in all at once, then some of them can overflow the relatively small kernel-to-user buffers. This commit increases the kernel-to-user buffers from the default of approximately 120 kB each to 1 MB each. In one somewhat synthetic test case that I ran based on an hping3 that generated a load of about 20,000 new flows per second (including both requests and replies), this reduced the packets dropped at the kernel-to-user interface from about 30% to none. I expect that it will similarly improve packet loss in workloads where flow arrival is not easily predictable. (This has little effect on workloads generated by ovs-benchmark rate because that benchmark is effectively self-clocking, that is, a new flow is triggered only by a reply to a request made earlier, which means that the number of buffered packets at any given has a known, constant upper limit.) Bug #10210. Signed-off-by: Ben Pfaff b...@nicira.com --- include/sparse/sys/socket.h |5 +++-- lib/netlink-socket.c| 10 +- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/include/sparse/sys/socket.h b/include/sparse/sys/socket.h index 89e3c2d..1ed195b 100644 --- a/include/sparse/sys/socket.h +++ b/include/sparse/sys/socket.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2011 Nicira Networks. + * Copyright (c) 2011, 2012 Nicira Networks. * * Licensed under the Apache License, Version 2.0 (the License); * you may not use this file except in compliance with the License. @@ -74,7 +74,8 @@ enum { SO_SNDBUF, SO_SNDLOWAT, SO_SNDTIMEO, -SO_TYPE +SO_TYPE, +SO_RCVBUFFORCE }; enum { diff --git a/lib/netlink-socket.c b/lib/netlink-socket.c index bc46235..df6f1d8 100644 --- a/lib/netlink-socket.c +++ b/lib/netlink-socket.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008, 2009, 2010, 2011 Nicira Networks. + * Copyright (c) 2008, 2009, 2010, 2011, 2012 Nicira Networks. * * Licensed under the Apache License, Version 2.0 (the License); * you may not use this file except in compliance with the License. @@ -89,6 +89,7 @@ nl_sock_create(int protocol, struct nl_sock **sockp) struct nl_sock *sock; struct sockaddr_nl local, remote; socklen_t local_size; +int rcvbuf; int retval = 0; if (!max_iovs) { @@ -122,6 +123,13 @@ nl_sock_create(int protocol, struct nl_sock **sockp) sock-protocol = protocol; sock-dump = NULL; +rcvbuf = 1024 * 1024; +if (setsockopt(sock-fd, SOL_SOCKET, SO_RCVBUFFORCE, + rcvbuf, sizeof rcvbuf)) { +VLOG_WARN_RL(rl, setting %d-byte socket receive buffer failed (%s), + rcvbuf, strerror(errno)); +} + retval = get_socket_rcvbuf(sock-fd); if (retval 0) { retval = -retval; -- 1.7.2.5 ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] warning messages regarding buffer space and unknown buffer
On Mon, Apr 23, 2012 at 03:54:09PM +0600, junaid khalid wrote: 1) Apr 23 13:19:27|00019|dpif|WARN|system@br0: recv failed (No buffer space available) Packets are arriving at your interfaces faster than ovs-vswitchd can set up flows. What traffic is going through the switch? 2) Apr 23 13:19:37|00033|pktbuf|WARN|cookie mismatch: 01fa != 02fa OVS is sending buffered packets to your OpenFlow controller. The OpenFlow controller is sending back replies to use those buffers, but the replies are arriving slowly enough that by the time that they arrive the switch has already discarded those buffers. Apr 23 13:19:37|00034|connmgr|INFO|br0-tcp:127.0.0.1:6633: sending type OFPET_BAD_REQUEST, code OFPBRC_BUFFER_UNKNOWN error reply to OFPT_FLOW_MOD message Also a consequence of the above. traffic type: MTU size UDP packets @7.3 Gbps, generating 100,000 flows by changing src IPs. NOX isn't going to be able to keep up with that rate. OVS 1.4.0 isn't, either, but OVS 1.6.90 from the tip of master should be able to handle it without a controller. Throwing in an OpenFlow controller that sees every packet will probably bog things down a lot. Apr 23 13:19:16|00010|ofp_util|WARN|received Nicira extension message of unknown type 8 That message has been obsolete for ages, why are you using it? ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss