Re: [ovs-discuss] OVS Stopped forwarding

[Vsevolod Vayner via discuss]

Hello Ilya --

Thank you for your message and clarification.

Could you please clarify which method/function called ovs-vswitchd and 
ovs-dpctl to open the kernel datapath and set flags?

Thank you.

[https://my.gcdn.co/images/web_3253/logo_g.png]<https://gcorelabs.com/>
Vsevolod Vayner


Product Director

vsevolod.vay...@gcore.com<mailto:vsevolod.vay...@gcore.com>

gcore.com<https://gcore.com/>
[https://my.gcdn.co/images/web_3253/face.png]<https://www.facebook.com/gcorelabscom/>
   [https://my.gcdn.co/images/web_3253/in.png] 
<https://www.linkedin.com/company/g-core>   
[https://my.gcdn.co/images/web_3253/yout.png] 
<https://www.youtube.com/channel/UCFhtQE7HB097a1_Nfv-Ma_Q>
[https://my.gcdn.co/images/web_3253/weib.png] 
<https://www.weibo.com/p/1005056986184457>




This message is intended solely for the addressee and may contain confidential 
information. If you have received this message in error, please send it back to 
us, and immediately and permanently delete it. Do not use, copy or disclose the 
information contained in this message or in any attachment.
From: Ilya Maximets 
Date: Monday, 31 July 2023 at 20:27
To: Vsevolod Vayner , b...@openvswitch.org 

Cc: [Partner] Fedor Tarasenko , [Partner] Ivan 
Kramarev , Dmitri Polyakov 
, [Partner] Ivan Suchkov , 
Aleksandr Fedotov , i.maxim...@ovn.org 

Subject: Re: [ovs-discuss] OVS Stopped forwarding
On 7/28/23 18:14, Vsevolod Vayner via discuss wrote:
> 1. What you did that made the problem appear.
>
> In our env we use openvswitch exporter to collect dump-flows and data from 
> production ovs.
> It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to 
> prometheus format.
> We run vswitchd, db & exporter in separate containers.
>
> In our production env we have:
>
> Open vSwitch:
>
> ovs-vswitchd (Open vSwitch) 2.12.0
> DPDK 18.11.2
> ovs-vsctl (Open vSwitch) 2.12.0
> DB Schema 8.0.0
>
> and we also have an installed OpenvSwitch on our exporter to collect 
> ovs-ofctl and ovs-dpctl data:
>
> ovs-vswitchd (Open vSwitch) 3.1.3
> DPDK 22.11.1
> ovs-vsctl (Open vSwitch) 3.1.3
> DB Schema 8.3.1
>
> 2. What you expected to happen.
>
> We expected the exporter to collect statistics using the ovs-ofctl utility 
> with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect 
> lookups.
>
> 3. What actually happened.
>
> openvswitch stopped to forward traffic. I see only ingress traffic on ports 
> of ovs, no traffic outs.
>
> 4. The kernel version on which Open vSwitch is running (from /proc/version) 
> and the distribution and version number of your OS (e.g. "Centos 5.0").
>
> Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org 
> <mailto:mockbu...@kbuilder.bsys.centos.org>) (gcc version 8.5.0 20210514 (Red 
> Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022
>
>
> In some cases, only a host reboot helps.
>
> If this is not well-known bug, I'll try to find resource in production to 
> reproduce and provide more debug information.

Hi, Vsevolod.

The problem is that you're using drastically different versions
of ovs-vswitchd and ovs-dpctl.  Every time you run ovs-dpctl,
it will open the kernel datapath and set flags to it in the process.
In your particular case, these flags change the way upcalls are
delivered to ovs-vswitchd, so no traffic can go to userspace
afterwards and no new flows will be installed to a datapath as
a result, because ovs-vswitchd is too old to understand a new
datapath configuration.  The issue should be resolved by re-starting
ovs-vswitchd.

In general, use of ovs-dpctl while ovs-vswitchd is running is not
safe.  Use ovs-appctl dpctl/* commands to get the same information
via ovs-vswitchd instead.  Or at least make sure that the version
of ovs-dpctl is exactly the same as version of ovs-vswitchd.

Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS Stopped forwarding

[Ilya Maximets via discuss]

On 8/4/23 14:14, Vsevolod Vayner wrote:
> Hello Ilya --
> 
> Thank you for your message and clarification.
> 
> Could you please clarify which method/function called ovs-vswitchd and 
> ovs-dpctlto open the kernel datapath and set flags?

Both are using the same lib/dpif-netlink.c:dpif_netlink_open().
In your particular case, the OVS_DP_F_DISPATCH_UPCALL_PER_CPU
breaks upcall delivery, because old OVS 2.12 doesn't support it.

> 
> Thank you.
> 
>  
> 
> GCORE <https://gcorelabs.com/>
> *Vsevolod Vayner*
>  
> Product Director
>  
> vsevolod.vay...@gcore.com <mailto:vsevolod.vay...@gcore.com>
> gcore.com <https://gcore.com/>
> <https://www.facebook.com/gcorelabscom/>  
> <https://www.linkedin.com/company/g-core>   
> <https://www.youtube.com/channel/UCFhtQE7HB097a1_Nfv-Ma_Q>  
> <https://www.weibo.com/p/1005056986184457>
> 
>  
> This message is intended solely for the addressee and may contain 
> confidential information. If you have received this message in error, please 
> send it back to us, and immediately and permanently delete it. Do not use, 
> copy or disclose the information contained in this message or in any 
> attachment.
> 
> *From: *Ilya Maximets 
> *Date: *Monday, 31 July 2023 at 20:27
> *To: *Vsevolod Vayner , b...@openvswitch.org 
> 
> *Cc: *[Partner] Fedor Tarasenko , [Partner] 
> Ivan Kramarev , Dmitri Polyakov 
> , [Partner] Ivan Suchkov , 
> Aleksandr Fedotov , i.maxim...@ovn.org 
> 
> *Subject: *Re: [ovs-discuss] OVS Stopped forwarding
> 
> On 7/28/23 18:14, Vsevolod Vayner via discuss wrote:
>> 1. What you did that made the problem appear.
>> 
>> In our env we use openvswitch exporter to collect dump-flows and data from 
>> production ovs.
>> It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to 
>> prometheus format.
>> We run vswitchd, db & exporter in separate containers.
>> 
>> In our production env we have:
>> 
>> Open vSwitch:
>> 
>> ovs-vswitchd (Open vSwitch) 2.12.0
>> DPDK 18.11.2
>> ovs-vsctl (Open vSwitch) 2.12.0
>> DB Schema 8.0.0
>> 
>> and we also have an installed OpenvSwitch on our exporter to collect 
>> ovs-ofctl and ovs-dpctl data:
>> 
>> ovs-vswitchd (Open vSwitch) 3.1.3
>> DPDK 22.11.1
>> ovs-vsctl (Open vSwitch) 3.1.3
>> DB Schema 8.3.1
>> 
>> 2. What you expected to happen.
>> 
>> We expected the exporter to collect statistics using the ovs-ofctl utility 
>> with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect 
>> lookups.
>> 
>> 3. What actually happened.
>> 
>> openvswitch stopped to forward traffic. I see only ingress traffic on ports 
>> of ovs, no traffic outs.
>> 
>> 4. The kernel version on which Open vSwitch is running (from /proc/version) 
>> and the distribution and version number of your OS (e.g. "Centos 5.0").
>> 
>> Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org 
>> <mailto:mockbu...@kbuilder.bsys.centos.org 
>> <mailto:mockbu...@kbuilder.bsys.centos.org>>) (gcc version 8.5.0 20210514 
>> (Red Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022
>> 
>> 
>> In some cases, only a host reboot helps.
>> 
>> If this is not well-known bug, I'll try to find resource in production to 
>> reproduce and provide more debug information.
> 
> Hi, Vsevolod.
> 
> The problem is that you're using drastically different versions
> of ovs-vswitchd and ovs-dpctl.  Every time you run ovs-dpctl,
> it will open the kernel datapath and set flags to it in the process.
> In your particular case, these flags change the way upcalls are
> delivered to ovs-vswitchd, so no traffic can go to userspace
> afterwards and no new flows will be installed to a datapath as
> a result, because ovs-vswitchd is too old to understand a new
> datapath configuration.  The issue should be resolved by re-starting
> ovs-vswitchd.
> 
> In general, use of ovs-dpctl while ovs-vswitchd is running is not
> safe.  Use ovs-appctl dpctl/* commands to get the same information
> via ovs-vswitchd instead.  Or at least make sure that the version
> of ovs-dpctl is exactly the same as version of ovs-vswitchd.
> 
> Best regards, Ilya Maximets.
> 

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS Stopped forwarding

[Ilya Maximets via discuss]

On 7/28/23 18:14, Vsevolod Vayner via discuss wrote:
> 1. What you did that made the problem appear.
> 
> In our env we use openvswitch exporter to collect dump-flows and data from 
> production ovs.
> It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to 
> prometheus format.
> We run vswitchd, db & exporter in separate containers.
> 
> In our production env we have:
> 
> Open vSwitch:
> 
> ovs-vswitchd (Open vSwitch) 2.12.0
> DPDK 18.11.2
> ovs-vsctl (Open vSwitch) 2.12.0
> DB Schema 8.0.0
> 
> and we also have an installed OpenvSwitch on our exporter to collect 
> ovs-ofctl and ovs-dpctl data:
> 
> ovs-vswitchd (Open vSwitch) 3.1.3
> DPDK 22.11.1
> ovs-vsctl (Open vSwitch) 3.1.3
> DB Schema 8.3.1
> 
> 2. What you expected to happen.
> 
> We expected the exporter to collect statistics using the ovs-ofctl utility 
> with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect 
> lookups.
> 
> 3. What actually happened.
> 
> openvswitch stopped to forward traffic. I see only ingress traffic on ports 
> of ovs, no traffic outs.
> 
> 4. The kernel version on which Open vSwitch is running (from /proc/version) 
> and the distribution and version number of your OS (e.g. "Centos 5.0").
> 
> Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org 
> ) (gcc version 8.5.0 20210514 (Red 
> Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022
> 
> 
> In some cases, only a host reboot helps.
> 
> If this is not well-known bug, I'll try to find resource in production to 
> reproduce and provide more debug information.

Hi, Vsevolod.

The problem is that you're using drastically different versions
of ovs-vswitchd and ovs-dpctl.  Every time you run ovs-dpctl,
it will open the kernel datapath and set flags to it in the process.
In your particular case, these flags change the way upcalls are
delivered to ovs-vswitchd, so no traffic can go to userspace
afterwards and no new flows will be installed to a datapath as
a result, because ovs-vswitchd is too old to understand a new
datapath configuration.  The issue should be resolved by re-starting
ovs-vswitchd.

In general, use of ovs-dpctl while ovs-vswitchd is running is not
safe.  Use ovs-appctl dpctl/* commands to get the same information
via ovs-vswitchd instead.  Or at least make sure that the version
of ovs-dpctl is exactly the same as version of ovs-vswitchd.

Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss