On 7/28/23 18:14, Vsevolod Vayner via discuss wrote: > 1. What you did that made the problem appear. > > In our env we use openvswitch exporter to collect dump-flows and data from > production ovs. > It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to > prometheus format. > We run vswitchd, db & exporter in separate containers. > > In our production env we have: > > Open vSwitch: > > ovs-vswitchd (Open vSwitch) 2.12.0 > DPDK 18.11.2 > ovs-vsctl (Open vSwitch) 2.12.0 > DB Schema 8.0.0 > > and we also have an installed OpenvSwitch on our exporter to collect > ovs-ofctl and ovs-dpctl data: > > ovs-vswitchd (Open vSwitch) 3.1.3 > DPDK 22.11.1 > ovs-vsctl (Open vSwitch) 3.1.3 > DB Schema 8.3.1 > > 2. What you expected to happen. > > We expected the exporter to collect statistics using the ovs-ofctl utility > with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect > lookups. > > 3. What actually happened. > > openvswitch stopped to forward traffic. I see only ingress traffic on ports > of ovs, no traffic outs. > > 4. The kernel version on which Open vSwitch is running (from /proc/version) > and the distribution and version number of your OS (e.g. "Centos 5.0"). > > Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org > <mailto:mockbu...@kbuilder.bsys.centos.org>) (gcc version 8.5.0 20210514 (Red > Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022 > > > In some cases, only a host reboot helps. > > If this is not well-known bug, I'll try to find resource in production to > reproduce and provide more debug information.
Hi, Vsevolod. The problem is that you're using drastically different versions of ovs-vswitchd and ovs-dpctl. Every time you run ovs-dpctl, it will open the kernel datapath and set flags to it in the process. In your particular case, these flags change the way upcalls are delivered to ovs-vswitchd, so no traffic can go to userspace afterwards and no new flows will be installed to a datapath as a result, because ovs-vswitchd is too old to understand a new datapath configuration. The issue should be resolved by re-starting ovs-vswitchd. In general, use of ovs-dpctl while ovs-vswitchd is running is not safe. Use ovs-appctl dpctl/* commands to get the same information via ovs-vswitchd instead. Or at least make sure that the version of ovs-dpctl is exactly the same as version of ovs-vswitchd. Best regards, Ilya Maximets. _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
