Re: [ovs-discuss] ovs-ofctl fails occasionally
Ben,
> The following stopgap appears to fix the problem on master. I suspect
> it could be easily backported as far as necessary. Does it work for you
> too?
Yes, it worked. Thank you for your support.
Thanks,
Ken
On Thu, 28 Feb 2019 00:12:54 +,
b...@ovn.org wrote
in E-mail "Re: [ovs-discuss] ovs-ofctl fails occasionally":
> On Wed, Feb 27, 2019 at 12:45:58PM +, Ken Ajiro wrote:
> >
> > Hello,
> >
> > When I use ovs-ofctl on OVS 2.10.1, it failed occasionally by error:
> >
> > ovs-ofctl: br0: failed to connect to socket (Broken pipe)
> >
> > This error is possible to be reproduced with:
> >
> > # while :; do /opt/nec/pf/ds/bin/ovs-ofctl --timeout=5 dump-flows br0 >
> > /tmp/log || break; done; cat /tmp/log
> > ovs-ofctl: br0: failed to connect to socket (Broken pipe)
> >
> > Also I tried with OVS 2.4.1 and could not reproduce this error.
> >
> > I think that recently rconn.c was modified on commit
> > 476d2551abd2871696a64203f78d658ac2d7f32c
> > and the connection for ofctl was changed it begins on state S_CONNECTING
> > instead of S_ACTIVE.
> > So timeout_CONNECTING will be applied for ofctl connection, however
> > timeout_CONNECTING is 1
> > and this is too short (e.g. if transition time of CONNCTING was
> > 12:00:00.999, it will be timed out
> > after 1ms). I think that this is cause of occasionally disconnection of
> > ofctl.
>
> Thanks for the report.
>
> It seems to me that the right long-term fix is to use higher resolution
> (e.g. millisecond resolution) to measure the timeouts. However, that
> will be a relatively large fix.
>
> The following stopgap appears to fix the problem on master. I suspect
> it could be easily backported as far as necessary. Does it work for you
> too?
>
> diff --git a/lib/rconn.c b/lib/rconn.c
> index 48ae8c6a72e5..8ca23cbcfe90 100644
> --- a/lib/rconn.c
> +++ b/lib/rconn.c
> @@ -502,7 +502,7 @@ static unsigned int
> timeout_CONNECTING(const struct rconn *rc)
> OVS_REQUIRES(rc->mutex)
> {
> -return MAX(1, rc->backoff);
> +return MAX(2, rc->backoff);
> }
>
> static void
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Router not allowing traffic in reverse
OK. The OVS FAQ has some advice: Q: I have a sophisticated network setup involving Open vSwitch, VMs or multiple hosts, and other components. The behavior isn't what I expect. Help! A: To debug network behavior problems, trace the path of a packet, hop-by-hop, from its origin in one host to a remote host. If that's correct, then trace the path of the response packet back to the origin. The open source tool called ``plotnetcfg`` can help to understand the relationship between the networking devices on a single host. Usually a simple ICMP echo request and reply (``ping``) packet is good enough. Start by initiating an ongoing ``ping`` from the origin host to a remote host. If you are tracking down a connectivity problem, the "ping" will not display any successful output, but packets are still being sent. (In this case the packets being sent are likely ARP rather than ICMP.) Tools available for tracing include the following: - ``tcpdump`` and ``wireshark`` for observing hops across network devices, such as Open vSwitch internal devices and physical wires. - ``ovs-appctl dpif/dump-flows `` in Open vSwitch 1.10 and later or ``ovs-dpctl dump-flows `` in earlier versions. These tools allow one to observe the actions being taken on packets in ongoing flows. See ovs-vswitchd(8) for ``ovs-appctl dpif/dump-flows`` documentation, ovs-dpctl(8) for ``ovs-dpctl dump-flows`` documentation, and "Why are there so many different ways to dump flows?" above for some background. - ``ovs-appctl ofproto/trace`` to observe the logic behind how ovs-vswitchd treats packets. See ovs-vswitchd(8) for documentation. You can out more details about a given flow that ``ovs-dpctl dump-flows`` displays, by cutting and pasting a flow from the output into an ``ovs-appctl ofproto/trace`` command. - SPAN, RSPAN, and ERSPAN features of physical switches, to observe what goes on at these physical hops. Starting at the origin of a given packet, observe the packet at each hop in turn. For example, in one plausible scenario, you might: 1. ``tcpdump`` the ``eth`` interface through which an ARP egresses a VM, from inside the VM. 2. ``tcpdump`` the ``vif`` or ``tap`` interface through which the ARP ingresses the host machine. 3. Use ``ovs-dpctl dump-flows`` to spot the ARP flow and observe the host interface through which the ARP egresses the physical machine. You may need to use ``ovs-dpctl show`` to interpret the port numbers. If the output seems surprising, you can use ``ovs-appctl ofproto/trace`` to observe details of how ovs-vswitchd determined the actions in the ``ovs-dpctl dump-flows`` output. 4. ``tcpdump`` the ``eth`` interface through which the ARP egresses the physical machine. 5. ``tcpdump`` the ``eth`` interface through which the ARP ingresses the physical machine, at the remote host that receives the ARP. 6. Use ``ovs-dpctl dump-flows`` to spot the ARP flow on the remote host remote host that receives the ARP and observe the VM ``vif`` or ``tap`` interface to which the flow is directed. Again, ``ovs-dpctl show`` and ``ovs-appctl ofproto/trace`` might help. 7. ``tcpdump`` the ``vif`` or ``tap`` interface to which the ARP is directed. 8. ``tcpdump`` the ``eth`` interface through which the ARP ingresses a VM, from inside the VM. It is likely that during one of these steps you will figure out the problem. If not, then follow the ARP reply back to the origin, in reverse. On Thu, Feb 28, 2019 at 06:34:17PM -0600, John Carew wrote: > I think there is some misunderstanding, I don't think OVS is working > incorrectly or has a problem. I believe it is working exactly how it is > configured. I believe OpenStack is just configuring OVS, and it has put in a > flow rule or something that is allowing outbound for a network/subnet but not > inbound. I'm asking how to figure out where in OVS it is stopping at. > > > On Feb 28, 2019, at 5:51 PM, Ben Pfaff wrote: > > > > Do you have a reason to believe that it's a problem in OVS, rather than > > a problem in what OpenStack is telling OVS to do? > > > > On Thu, Feb 28, 2019 at 04:48:44PM -0600, John Carew wrote: > >> I agree, but I want to diagnose what is causing it in ovs first before I > >> go to them. > >> > >> John > >> > On Feb 28, 2019, at 4:33 PM, Ben Pfaff wrote: > > On Wed, Feb 27, 2019 at 09:26:16PM -0600, John Carew wrote: > I have setup OpenStack with OVS. I have a single Hyper-V server running > the > controller and three CentOS instances(10.0.0.x) on a private subnet. I > created a router in OpenStack with SNAT disabled, as I only want it to > route > traffic between the private subnet(10.0.0.x) and the external >
Re: [ovs-discuss] Router not allowing traffic in reverse
Do you have a reason to believe that it's a problem in OVS, rather than a problem in what OpenStack is telling OVS to do? On Thu, Feb 28, 2019 at 04:48:44PM -0600, John Carew wrote: > I agree, but I want to diagnose what is causing it in ovs first before I go > to them. > > John > > > On Feb 28, 2019, at 4:33 PM, Ben Pfaff wrote: > > > >> On Wed, Feb 27, 2019 at 09:26:16PM -0600, John Carew wrote: > >> I have setup OpenStack with OVS. I have a single Hyper-V server running the > >> controller and three CentOS instances(10.0.0.x) on a private subnet. I > >> created a router in OpenStack with SNAT disabled, as I only want it to > >> route > >> traffic between the private subnet(10.0.0.x) and the external > >> subnet(172.16.1.x)/internet. All of the instances can ping each other along > >> with the external network(172.16.1.x). From the external network, I can > >> ping > >> the interface of the ovs router on the external network. I can not though > >> ping inside the private network. A trace route stops at the IP of the OVS > >> router. With wireshark, I do not see anything coming from the external pc’s > >> IP. If I trace route it, I see packets making all the way to the OVS router > >> and then stop. Since I can ping one way, and not the other; I believe there > >> is something in the router/OVS that is stopping the packets to route into > >> the private subnet. What do I need to look at? (I have disabled all > >> firewalls on all OSes involved.) > > > > An OpenStack mailing list might be a better place to track this down. > > OpenStack programs and configures OVS; OVS by itself doesn't have much > > to do with what's going on. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Router not allowing traffic in reverse
On Wed, Feb 27, 2019 at 09:26:16PM -0600, John Carew wrote: > I have setup OpenStack with OVS. I have a single Hyper-V server running the > controller and three CentOS instances(10.0.0.x) on a private subnet. I > created a router in OpenStack with SNAT disabled, as I only want it to route > traffic between the private subnet(10.0.0.x) and the external > subnet(172.16.1.x)/internet. All of the instances can ping each other along > with the external network(172.16.1.x). From the external network, I can ping > the interface of the ovs router on the external network. I can not though > ping inside the private network. A trace route stops at the IP of the OVS > router. With wireshark, I do not see anything coming from the external pcs > IP. If I trace route it, I see packets making all the way to the OVS router > and then stop. Since I can ping one way, and not the other; I believe there > is something in the router/OVS that is stopping the packets to route into > the private subnet. What do I need to look at? (I have disabled all > firewalls on all OSes involved.) An OpenStack mailing list might be a better place to track this down. OpenStack programs and configures OVS; OVS by itself doesn't have much to do with what's going on. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Open vSwitch 2.11.0 Available
Whoops. It should be fixed now. Thanks, Edouard! --Justin > On Feb 28, 2019, at 12:26 AM, Madko wrote: > > Hi Justin, > > http://openvswitch.org/releases/NEWS-2.11.0 is a 404 (as all previous NEWS > release pages). Is that normal ? > > Best regards, > Edouard > > Le jeu. 28 févr. 2019 à 07:57, Justin Pettit a écrit : > The Open vSwitch team is pleased to announce the release of Open vSwitch > 2.11.0: > > http://openvswitch.org/releases/openvswitch-2.11.0.tar.gz > > A few other feature highlights of 2.11.0 include: > > - OVN support for encrypted tunnels between hypervisors. > > - Improved IPAM support in OVN. > > - New OpenFlow feature support. > > - Support for DPDK 18.11. > > - Linux kernel support up to 4.18. > > - And many others. See the full change log here: > >http://openvswitch.org/releases/NEWS-2.11.0 > > Enjoy! > > --The Open vSwitch Team > > > Open vSwitch is a production quality, multilayer open source virtual switch. > It is designed to enable massive network automation through programmatic > extension, while still supporting standard management interfaces. Open > vSwitch can operate both as a soft switch running within the hypervisor, and > as the control stack for switching silicon. It has been ported to multiple > virtualization platforms and switching chipsets. > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > -- > Edouard Bourguignon ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Open vSwitch 2.11.0 Available
The correct URLs look like this: https://www.openvswitch.org/releases/NEWS-2.11.0.txt On Thu, Feb 28, 2019 at 10:15:13AM -0500, Flaviof wrote: > Heh, looks like a 404 indeed. > > For now, if you are looking for the actual content of 2.11.0, try this link: > > https://github.com/openvswitch/ovs/blob/997f2b583f49d1a52b41958b88acf4f23a49eba6/NEWS > > -- flaviof > > > On Thu, Feb 28, 2019 at 3:39 AM Madko wrote: > > > Hi Justin, > > > > http://openvswitch.org/releases/NEWS-2.11.0 is a 404 (as all previous > > NEWS release pages). Is that normal ? > > > > Best regards, > > Edouard > > > > Le jeu. 28 févr. 2019 à 07:57, Justin Pettit a écrit : > > > >> The Open vSwitch team is pleased to announce the release of Open vSwitch > >> 2.11.0: > >> > >> http://openvswitch.org/releases/openvswitch-2.11.0.tar.gz > >> > >> A few other feature highlights of 2.11.0 include: > >> > >> - OVN support for encrypted tunnels between hypervisors. > >> > >> - Improved IPAM support in OVN. > >> > >> - New OpenFlow feature support. > >> > >> - Support for DPDK 18.11. > >> > >> - Linux kernel support up to 4.18. > >> > >> - And many others. See the full change log here: > >> > >>http://openvswitch.org/releases/NEWS-2.11.0 > >> > >> Enjoy! > >> > >> --The Open vSwitch Team > >> > >> > >> Open vSwitch is a production quality, multilayer open source virtual > >> switch. It is designed to enable massive network automation through > >> programmatic extension, while still supporting standard management > >> interfaces. Open vSwitch can operate both as a soft switch running within > >> the hypervisor, and as the control stack for switching silicon. It has been > >> ported to multiple virtualization platforms and switching chipsets. > >> > >> > >> ___ > >> discuss mailing list > >> disc...@openvswitch.org > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > >> > > > > > > -- > > Edouard Bourguignon > > ___ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd process huge memory consumption
On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote: > Hi, > > thanks for the scripts, so here's the output for a 24G core dump: > https://pastebin.com/hWa3R9Fx > there's 271 entries of 4MB - does it seem something we should take a closer > look at? I think that this output really just indicates that the script failed. It analyzed a lot of regions but didn't output anything useful. If it had worked properly, it would have told us a lot about data blocks that had been allocated and freed. The next step would have to be to debug the script. It definitely worked for me before, because I have fixed at least 3 or 4 bugs based on it, but it also definitely is a quick hack and not something that I can stand behind. I'm not sure how to debug it at a distance. It has a large comment that describes what it's trying to do. Maybe that would help you, if you want to try to debug it yourself. I guess it's also possible that glibc has changed its malloc implementation; if so, then it would probably be necessary to start over and build a new script. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] OVN: availability zones concept
Hi folks, Just wanted to throw an idea here about introducing availability zones (AZ) concept in OVN and get implementation ideas. From a CMS perspective, it makes sense to be able to implement some sort of logical division of resources into failure domains to maximize their availability. In this sense, establishing a full mesh of Geneve tunnels is not needed (and possibly undesired when strict firewalls are used between AZs) as L2 connectivity will be constrained to the AZ boundaries. A possibility would be to let the deployer of the CMS set a key on the OpenvSwitch table of the local OVS instance like 'external_ids:ovn_az=' and if it's set, ovn-controller will register itself as a Chassis with the same external ID and establish tunnels to those Chassis within the same AZ, otherwise it'll keep the current behavior. It'll be responsibility of the CMS to schedule gateway ports in the right AZ as well to provide L3 AZ awareness. Does that make sense? Thoughts? Thanks a lot!! Daniel ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Open vSwitch 2.11.0 Available
Heh, looks like a 404 indeed. For now, if you are looking for the actual content of 2.11.0, try this link: https://github.com/openvswitch/ovs/blob/997f2b583f49d1a52b41958b88acf4f23a49eba6/NEWS -- flaviof On Thu, Feb 28, 2019 at 3:39 AM Madko wrote: > Hi Justin, > > http://openvswitch.org/releases/NEWS-2.11.0 is a 404 (as all previous > NEWS release pages). Is that normal ? > > Best regards, > Edouard > > Le jeu. 28 févr. 2019 à 07:57, Justin Pettit a écrit : > >> The Open vSwitch team is pleased to announce the release of Open vSwitch >> 2.11.0: >> >> http://openvswitch.org/releases/openvswitch-2.11.0.tar.gz >> >> A few other feature highlights of 2.11.0 include: >> >> - OVN support for encrypted tunnels between hypervisors. >> >> - Improved IPAM support in OVN. >> >> - New OpenFlow feature support. >> >> - Support for DPDK 18.11. >> >> - Linux kernel support up to 4.18. >> >> - And many others. See the full change log here: >> >>http://openvswitch.org/releases/NEWS-2.11.0 >> >> Enjoy! >> >> --The Open vSwitch Team >> >> >> Open vSwitch is a production quality, multilayer open source virtual >> switch. It is designed to enable massive network automation through >> programmatic extension, while still supporting standard management >> interfaces. Open vSwitch can operate both as a soft switch running within >> the hypervisor, and as the control stack for switching silicon. It has been >> ported to multiple virtualization platforms and switching chipsets. >> >> >> ___ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> > > > -- > Edouard Bourguignon > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd process huge memory consumption
Hi Ben,
so here're examples of how those repeatable blocks look like:
https://pastebin.com/JBUaeX44
https://pastebin.com/wKreHDJf
https://pastebin.com/f41knqgn
All those blocks are mostly filled with such sequence:
" fa16 3e39 83c4 ..>9
0030 4014 ...0..@.
0fff
6500 e...
4014 ..@."
where "fa 16 3e xx xx xx" are openstack neutron ports mac addresses
(actually VMs MACs as confirmed on the env with issue).
Can we figure something out of this?
Thanks,
Oleg
On Wed, Feb 27, 2019 at 8:41 AM Oleg Bondarev
wrote:
>
>
> On Tue, Feb 26, 2019 at 1:41 PM Oleg Bondarev
> wrote:
>
>> Hi,
>>
>> thanks for the scripts, so here's the output for a 24G core dump:
>> https://pastebin.com/hWa3R9Fx
>> there's 271 entries of 4MB - does it seem something we should take a
>> closer look at?
>>
>
> not 4 but ~67Mb of course.
>
>
>>
>> Thanks,
>> Oleg
>>
>> On Tue, Feb 26, 2019 at 3:26 AM Ben Pfaff wrote:
>>
>>> Some combinations of kernel bonding with Open vSwitch don't necessarily
>>> work that well. I have forgotten which ones are problematic or why.
>>> However, the problems are functional ones (the bonds don't work well),
>>> not operational ones like memory leaks. A memory leak would be a bug
>>> whether or not the configuration used kernel bonds in a way that we
>>> discourage.
>>>
>>> With small OpenFlow flow tables (maybe a few thousand flows) and a small
>>> number of ports (maybe up to 100 or so), using the Linux kernel module
>>> (not DPDK), I'd be surprised to see OVS use more than 100 MB. I'd also
>>> be surprised to see OVS memory usage grow past, say, the first day or
>>> usage.
>>>
>>> With large numbers of OpenFlow flows (e.g. hundreds of thousands), my
>>> rule of thumb is that OVS tends to use about 1 to 2 kB RAM per flow.
>>>
>>> A dump of a 500 MB process could still be enlightening. Usually, when
>>> there is a memory leak, one sees an extraordinary number of unfreed
>>> memory blocks of a single size, and by looking at their size and their
>>> contents one can often figure out what allocated them, and that usually
>>> leads one to figure out why they did not get freed.
>>>
>>> On Mon, Feb 25, 2019 at 09:46:28PM +, Fernando Casas Schössow wrote:
>>> >
>>> > I read in a few places that mixing OS networking features (like
>>> bonding) and OVS is not a good idea and that the recommendation is to do
>>> everything at OVS level. That's why I assumed the configuration was not ok
>>> (even when it worked correctly for around two years albeit the high memory
>>> usage I detected).
>>> >
>>> > How many MB of RAM would you consider normal in a small setup like
>>> this one? Just to make myself an idea.
>>> >
>>> > I just finished a maintenance window on this server that required a
>>> reboot.
>>> > Right after reboot ovs-vswitchd is using 14MB of RAM.
>>> > I will keep monitoring the process memory usage usage and report back
>>> after two weeks or so.
>>> >
>>> > Would it make sense to get a process dump for analysis even if memory
>>> usage is not going as high (several GBs) as before the config change? In
>>> other words, if I find that the process memory usage grows up to around
>>> 500MB but then becomes steady and is not growing anymore would it make
>>> sense to collect a dump for analysis?
>>> >
>>> > On lun, feb 25, 2019 at 5:48 PM, Ben Pfaff wrote:
>>> > Both configurations should work, so probably you did find a bug
>>> causing a memory leak in the former configuration. 464 MB actually sounds
>>> like a lot also. On Sun, Feb 24, 2019 at 02:58:02PM +, Fernando Casas
>>> Schössow wrote:
>>> > Hi Ben, In my case I think I found the cause of the issue, and it was
>>> indeed a misconfiguration on my side. Yet I'm not really sure why the
>>> misconfiguration was causing the high memory usage on OVS. The server has 4
>>> NICs. Bonded in two bonds of two. The problem I think it was that the
>>> bonding was done at OS level (Linux kernel bonding) instead of at OVS
>>> level. So there were two interfaces at OS level (bond0 and bond1) with
>>> bond0 added to OVS as an uplink port. I changed that configuration, removed
>>> all the bonding at OS level and instead created the bonds at OVS level.
>>> Then I restarted the service so I can monitor memory usage. After this
>>> change, memory usage growth from 10MB (at service start) to 464MB after a
>>> few hours and then stayed at that level until today (a week later). I'm
>>> still monitoring the process memory usage but as I said is steady for
>>> almost a week so I will keep monitoring it for a couple more weeks just in
>>> case and report back. Thanks. Kind regards, Fernando On sáb, feb 23, 2019
>>> at 12:23 AM, Ben Pfaff mailto:b...@ovn.org>> wrote: It's
>>> odd that
Re: [ovs-discuss] Open vSwitch 2.11.0 Available
Hi Justin, http://openvswitch.org/releases/NEWS-2.11.0 is a 404 (as all previous NEWS release pages). Is that normal ? Best regards, Edouard Le jeu. 28 févr. 2019 à 07:57, Justin Pettit a écrit : > The Open vSwitch team is pleased to announce the release of Open vSwitch > 2.11.0: > > http://openvswitch.org/releases/openvswitch-2.11.0.tar.gz > > A few other feature highlights of 2.11.0 include: > > - OVN support for encrypted tunnels between hypervisors. > > - Improved IPAM support in OVN. > > - New OpenFlow feature support. > > - Support for DPDK 18.11. > > - Linux kernel support up to 4.18. > > - And many others. See the full change log here: > >http://openvswitch.org/releases/NEWS-2.11.0 > > Enjoy! > > --The Open vSwitch Team > > > Open vSwitch is a production quality, multilayer open source virtual > switch. It is designed to enable massive network automation through > programmatic extension, while still supporting standard management > interfaces. Open vSwitch can operate both as a soft switch running within > the hypervisor, and as the control stack for switching silicon. It has been > ported to multiple virtualization platforms and switching chipsets. > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > -- Edouard Bourguignon ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
