Re: [ovs-discuss] Can we choose the encryption algorithms in IPSec?
On 5/16/24 19:33, Jim C via discuss wrote: > Hi, > > I'm looking into this IPSec tutorial > (https://docs.openvswitch.org/en/latest/tutorials/ipsec/). > I wonder if there is an option for us to choose the encryption algorithm that > we want given we > want FIPS approved algorithms. Is there a manual? Hi. The algorithms are not configurable today. StrongSwan is configured with the following options: keyexchange=ikev2 ike=aes256gcm16-sha256-modp2048 esp=aes256gcm16-modp2048 And for Libreswan: ike=aes_gcm256-sha2_256 esp=aes_gcm256 ikev2=insist Changing these options will require changing the code of the ovs-monitor-ipsec daemon. Which is a python script, so should not be difficult if necessary. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Is OVS FIPS compliant?
On 5/16/24 19:51, Jim C wrote: > Thanks Ilya. I think there are 2 things we are interested in: > 1) If the crypto is done with a FIPS validated module > 2) Are the crypto algorithms all FIPS approved > > For 2), I think we can choose the encryption algorithms we use in IPSec > (Please correct me if I was wrong). > For 1), do you have more information on that? Maybe I also need to clarify that OVS itself doesn't perform any crypto operations on traffic it forwards. That is entirely handled outside of OVS by Libreswan or StrongSwan in case of IPSec. The only crypto operations OVS does are operations on SSL/TLS connections that control ovs-vswitchd daemon and ovsdb-server. These are performed fully by OpenSSL. So, again, not OVS itself. By default whatever algorithms are in the OpenSSL's default list will be used. You can choose a subset with --ssl-cyphers command line argument or equivalent database configuration. But I would expect that compliant OpenSSL build will not contain non-compliant algorithms. Best regards, Ilya Maximets. > > On Mon, May 13, 2024 at 2:39 AM Ilya Maximets <mailto:i.maxim...@ovn.org>> wrote: > > On 5/12/24 08:17, Jim C via discuss wrote: > > We want to use Open vSwitch to build our network and enable IPSec > > for encryption in-transit. I wonder if there is a document that > > describes if the OVS package itself is FIPS compliant? > > Hi, Jim. > > If I'm not mistaken, FIPS compliant can only be a built binary and > Open vSwitch project doesn't release binaries. You need to ask > the distribution where you get your binary packages from. > > However, all the important crypto in OVS is performed by OpenSSL, > so it should be compliant as long as you're linking with compliant > version of OpenSSL. But again, you need to ask your distribution. > > Best regards, Ilya Maximets. > > > > > Maybe my question is not described accurately. Please let me know > > what more information is needed. > > > > Thanks. > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Is OVS FIPS compliant?
On 5/12/24 08:17, Jim C via discuss wrote: > We want to use Open vSwitch to build our network and enable IPSec > for encryption in-transit. I wonder if there is a document that > describes if the OVS package itself is FIPS compliant? Hi, Jim. If I'm not mistaken, FIPS compliant can only be a built binary and Open vSwitch project doesn't release binaries. You need to ask the distribution where you get your binary packages from. However, all the important crypto in OVS is performed by OpenSSL, so it should be compliant as long as you're linking with compliant version of OpenSSL. But again, you need to ask your distribution. Best regards, Ilya Maximets. > > Maybe my question is not described accurately. Please let me know > what more information is needed. > > Thanks. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
On 4/26/24 22:05, Gavin McKee wrote: > Thanks again for coming back on this Ilya, > > Another option I am looking at here is to switch the kernal path (Open > vSwitch kernel module) with OVS-DOCA as we are using the CX6/7 card > https://docs.nvidia.com/doca/archive/doca-v2.0.2/ovs-doca/index.html > > I'm trying to wrangle the documented Known Limitations > > - Only one insertion thread is supported (n-offload-threads=1). > - Only a single PF is currently supported. > - Geneve options are not supported. > - Only 250K connections are offloaded by CT offload. > - Only 8 CT zones are supported by CT offload. > - OVS restart on non-tunnel configuration is not supported. Remove all > ports before restarting. > > The one that concerns me is the 8 CT zones supported by CT offload, as > potentially with OVN I may have many CT zones if we have many customer > colocated on the same compute node. > > Seems any way I turn I'm getting kicked :D You'll also need a special binary build from NVIDIA. The docs you linked are closed-source fork and not an upstream OVS. OVS doesn't support offloading of connection tracking in userspace datapath. > > Gav > > On Fri, 26 Apr 2024 at 12:42, Ilya Maximets wrote: >> >> On 4/26/24 20:12, Gavin McKee wrote: >>> Thanks for coming back to me on this. >>> >>> Moving kernal versions around is not a straightforward option here - >>> especially when you are using hardware offload . The OFED driver >>> version is coupled to the kernal so if we move from that we are out of >>> support coverage . >>> >>> Doing an ovn-appctl -t ovn-controller recompute does not resolve the >>> problem , again just taking a big hammer like restarting openvswitch >>> does. >>> >>> How would we proceed here ? Are there any Open vSwitch kernal module >>> patches we could try to get a resolution ? >> >> You can try the commit I linked below. That will mean that you'll >> need to re-build your kernel. There is no other way. >> >> In the past we had out-of-tree module, but it is deprecated for a >> long time, contains multiple issues and is unlikely to work with >> new kernels, especially heavily modified ones, like RHEL kernels. >> >> Note that the issue is not localized to OVS, but affects TC as well >> as they now share the NAT implementation. So, even if just swapping >> the openvswitch kernel module was possible, it wouldn't help much. >> >>> >>> One option we are looking at is regressing the entire stack back to Rocky >>> 9.1 . >> >> This may be an option. The bug I mentioned in a previous email exists >> in RHEL 9.3, so it exists in Rocky 9.3 as well, at least it should >> since they claim "bug-for-bug compatibility". So, 3 options to fix this >> particular bug (I don't know if it is causing your issue, but it is >> a severe bug that can potentially be a cause): >> >> 1. Re-build the kernel to include the fix. >> 2. Downgrade from 9.3 to an earlier release. >> 3. Wait for 9.4. >> >> Best regards, Ilya Maximets. >> >>> >>> Gav >>> >>> On Wed, 24 Apr 2024 at 04:44, Ilya Maximets wrote: >>>> >>>> On 4/23/24 17:39, Gavin McKee wrote: >>>>> If you look at both traces (non working and working) the thing that >>>>> stands out to me is this >>>>> >>>>> At line 10 in the working file the following entry exists >>>>> ct_state NEW tcp (SYN_SENT) orig [172.27.16.11.38793 > >>>>> 172.27.31.189.9100] reply [172.27.31.189.9100 > 172.27.16.11.38793] >>>>> zone 195 >>>>> >>>>> his doesn't happen in the non working file - I just see the following >>>>> >>>>> 3904992932904126 [swapper/125] 0 [kr] queue_userspace_packet >>>>> #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 4 >>>>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>>>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>>>> (6) flags [S] seq 2266263186 win 42340 >>>>> upcall_enqueue (miss) (125/3904992932890052) q 3682874017 ret 0 >>>>> + 3904992932907247 [swapper/125] 0 [kr] ovs_dp_upcall >>>>> #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 5 >>>>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>>>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>>>> (6) flags [S] seq 2266263186 wi
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
On 4/26/24 20:12, Gavin McKee wrote: > Thanks for coming back to me on this. > > Moving kernal versions around is not a straightforward option here - > especially when you are using hardware offload . The OFED driver > version is coupled to the kernal so if we move from that we are out of > support coverage . > > Doing an ovn-appctl -t ovn-controller recompute does not resolve the > problem , again just taking a big hammer like restarting openvswitch > does. > > How would we proceed here ? Are there any Open vSwitch kernal module > patches we could try to get a resolution ? You can try the commit I linked below. That will mean that you'll need to re-build your kernel. There is no other way. In the past we had out-of-tree module, but it is deprecated for a long time, contains multiple issues and is unlikely to work with new kernels, especially heavily modified ones, like RHEL kernels. Note that the issue is not localized to OVS, but affects TC as well as they now share the NAT implementation. So, even if just swapping the openvswitch kernel module was possible, it wouldn't help much. > > One option we are looking at is regressing the entire stack back to Rocky 9.1 > . This may be an option. The bug I mentioned in a previous email exists in RHEL 9.3, so it exists in Rocky 9.3 as well, at least it should since they claim "bug-for-bug compatibility". So, 3 options to fix this particular bug (I don't know if it is causing your issue, but it is a severe bug that can potentially be a cause): 1. Re-build the kernel to include the fix. 2. Downgrade from 9.3 to an earlier release. 3. Wait for 9.4. Best regards, Ilya Maximets. > > Gav > > On Wed, 24 Apr 2024 at 04:44, Ilya Maximets wrote: >> >> On 4/23/24 17:39, Gavin McKee wrote: >>> If you look at both traces (non working and working) the thing that >>> stands out to me is this >>> >>> At line 10 in the working file the following entry exists >>> ct_state NEW tcp (SYN_SENT) orig [172.27.16.11.38793 > >>> 172.27.31.189.9100] reply [172.27.31.189.9100 > 172.27.16.11.38793] >>> zone 195 >>> >>> his doesn't happen in the non working file - I just see the following >>> >>> 3904992932904126 [swapper/125] 0 [kr] queue_userspace_packet >>> #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 4 >>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>> (6) flags [S] seq 2266263186 win 42340 >>> upcall_enqueue (miss) (125/3904992932890052) q 3682874017 ret 0 >>> + 3904992932907247 [swapper/125] 0 [kr] ovs_dp_upcall >>> #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 5 >>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>> (6) flags [S] seq 2266263186 win 42340 >>> upcall_ret (125/3904992932890052) ret 0 >>> >>> I am wondering if a failure to track the ct_state SYN is causing the >>> returning ACK to drop ? >>> >>> + 3904992936344421 [swapper/125] 0 [tp] skb:kfree_skb >>> #ddf9204d21c48ff1d0b676c330c00 (skb 18382861792850913792) n 3 drop >>> (TC_INGRESS) >>> if 33 (genev_sys_6081) rxif 33 172.27.31.189.9100 > >>> 172.27.16.11.42303 ttl 64 tos 0x0 id 0 off 0 [DF] len 52 proto TCP (6) >>> flags [S.] seq 605271182 ack 2266263187 win 42340 >>> >>> On Mon, 22 Apr 2024 at 18:54, Gavin McKee wrote: >>>> >>>> Ok @Adrian Moreno @Flavio Leitner >>>> >>>> Two more detailed Retis traces attached. One is not working - the >>>> same session that I can't establish a TCP session to on port 9010 >>>> 172.27.16.11.42303 > 172.27.31.189.9100 >>>> >>>> Then I restart Open vSwtich and try again >>>> 172.27.16.11.38793 > 172.27.31.189.9100 (this works post restart) >>>> >>>> It looks to me in the non working example that we - >>>> SEND SYN -> exits the tunnel interface genev_sys_6081 via >>>> enp148s0f0np0 - exactly as expected >>>> RECV ACK -> tcp_gro_receive then -> net:netif_receive_skb where we hit >>>> drop (TC_INGRESS) >>>> >>>> After a restart things seem to be very different >>>> >>>> Any ideas where to look next ? >> >> You mentioned that you're using 5.14.0-362.8.1.el9_3.x86 kernel. >> RHEL 9.3 contains a large refactor for OVS connection tracking, >> but it doesn't contain at least one fix for this refa
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
On 4/25/24 11:51, Vladislav Odintsov wrote:
>
>
>> On 25 Apr 2024, at 12:20, Ilya Maximets wrote:
>>
>> On 4/25/24 10:53, Vladislav Odintsov wrote:
>>> Hi Ilya,
>>>
>>> I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
>>> 3.2+ version.
>>> ovsdb(7) states that the recommended path for upgrade is to upgrade
>>> ovsdb-servers one-by-one
>>> and that after upgrade ovsdb-server should be started with option
>>> --disable-file-no-data-conversion
>>> and after whole cluster is upgradede, it is needed to enable no-data
>>> conversion via appctl.
>>>
>>> I’ve ran through code and did some upgrade tests, so my question is:
>>> Do I understand correctly, that if there is no necessity to call schema
>>> conversion after start
>>> and before end of cluster upgrade, so it is allowed just to restart
>>> ovsdb-servers without
>>> --disable-file-no-data-conversion option and this will not have any
>>> downsides?
>>
>> Simply re-starting removing the option is enough. There is no need
>> to enable specifically via appctl in this case.
>
> I’m talking about a bit another thing:
> I want not to start each upgraded ovsdb-server with
> --disable-file-no-data-conversion at all,
> because it guaranteed in my case that there will be no schema conversions
> before full cluster
> upgrade is finished.
> So, in this case there will be no need to enable it back via appctl or
> removing ''disable"
> option and restart.
>
> Am I right, or I’m missing something?
Ah, sorry. If you're sure that there will be no conversion
before all the servers are upgraded, then it should be fine
to just upgrade as usual.
>
>>
>> Also, we did actually backport parts of the format change to 3.1.
>> It should be in v3.1.2 release and newer. So, technically, if you're
>> performing upgrade from v3.1.2+, it should be safe to just upgrade as
>> usual. ovsdb-servr v3.1.2 understands new database format, though it
>> can't produce it on its own. See the following commit on branch-3.1:
>>
>> 9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
>> storage.")
>
> Hmm, nice. But mine source version is 3.0 and this is not applicable :)
The same change should be available in v3.0.5.
>
>>
>> It would be painful to document all the combinations of minor versions,
>> so it's not an officially supported upgrade path, but it is there if
>> you know what are you doing.
>
> Sure.
>
>>
>> Best regards, Ilya Maximets.
>
>
> Regards,
> Vladislav Odintsov
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
On 4/25/24 10:53, Vladislav Odintsov wrote:
> Hi Ilya,
>
> I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
> 3.2+ version.
> ovsdb(7) states that the recommended path for upgrade is to upgrade
> ovsdb-servers one-by-one
> and that after upgrade ovsdb-server should be started with option
> --disable-file-no-data-conversion
> and after whole cluster is upgradede, it is needed to enable no-data
> conversion via appctl.
>
> I’ve ran through code and did some upgrade tests, so my question is:
> Do I understand correctly, that if there is no necessity to call schema
> conversion after start
> and before end of cluster upgrade, so it is allowed just to restart
> ovsdb-servers without
> --disable-file-no-data-conversion option and this will not have any downsides?
Simply re-starting removing the option is enough. There is no need
to enable specifically via appctl in this case.
Also, we did actually backport parts of the format change to 3.1.
It should be in v3.1.2 release and newer. So, technically, if you're
performing upgrade from v3.1.2+, it should be safe to just upgrade as
usual. ovsdb-servr v3.1.2 understands new database format, though it
can't produce it on its own. See the following commit on branch-3.1:
9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
storage.")
It would be painful to document all the combinations of minor versions,
so it's not an officially supported upgrade path, but it is there if
you know what are you doing.
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
On 4/23/24 17:39, Gavin McKee wrote: > If you look at both traces (non working and working) the thing that > stands out to me is this > > At line 10 in the working file the following entry exists > ct_state NEW tcp (SYN_SENT) orig [172.27.16.11.38793 > > 172.27.31.189.9100] reply [172.27.31.189.9100 > 172.27.16.11.38793] > zone 195 > > his doesn't happen in the non working file - I just see the following > > 3904992932904126 [swapper/125] 0 [kr] queue_userspace_packet > #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 4 > if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > > 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP > (6) flags [S] seq 2266263186 win 42340 > upcall_enqueue (miss) (125/3904992932890052) q 3682874017 ret 0 > + 3904992932907247 [swapper/125] 0 [kr] ovs_dp_upcall > #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 5 > if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > > 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP > (6) flags [S] seq 2266263186 win 42340 > upcall_ret (125/3904992932890052) ret 0 > > I am wondering if a failure to track the ct_state SYN is causing the > returning ACK to drop ? > > + 3904992936344421 [swapper/125] 0 [tp] skb:kfree_skb > #ddf9204d21c48ff1d0b676c330c00 (skb 18382861792850913792) n 3 drop > (TC_INGRESS) > if 33 (genev_sys_6081) rxif 33 172.27.31.189.9100 > > 172.27.16.11.42303 ttl 64 tos 0x0 id 0 off 0 [DF] len 52 proto TCP (6) > flags [S.] seq 605271182 ack 2266263187 win 42340 > > On Mon, 22 Apr 2024 at 18:54, Gavin McKee wrote: >> >> Ok @Adrian Moreno @Flavio Leitner >> >> Two more detailed Retis traces attached. One is not working - the >> same session that I can't establish a TCP session to on port 9010 >> 172.27.16.11.42303 > 172.27.31.189.9100 >> >> Then I restart Open vSwtich and try again >> 172.27.16.11.38793 > 172.27.31.189.9100 (this works post restart) >> >> It looks to me in the non working example that we - >> SEND SYN -> exits the tunnel interface genev_sys_6081 via >> enp148s0f0np0 - exactly as expected >> RECV ACK -> tcp_gro_receive then -> net:netif_receive_skb where we hit >> drop (TC_INGRESS) >> >> After a restart things seem to be very different >> >> Any ideas where to look next ? You mentioned that you're using 5.14.0-362.8.1.el9_3.x86 kernel. RHEL 9.3 contains a large refactor for OVS connection tracking, but it doesn't contain at least one fix for this refactor: https://github.com/torvalds/linux/commit/e6345d2824a3f58aab82428d11645e0da861ac13 This may cause all sorts of incorrect packet processing in the kernel. I'd suggest trying the latest upstream v6.8.7 kernel that has all the known fixes or try the 9.4 beta kernel that should have the fix mentioned above. Downgrading to 9.2 may also be an option since 9.2 doesn't contain the refactoring, IIRC. I'd not recommend running with this bug present anyway. Updating your OVN 23.09.1 to 23.09.3 may also be worth trying. The fact that OVS restart fixes the issue may also indicate a problem with incremental processing in ovn-controller. Next time the issue happens try to force flow recompute with ovn-appctl -t ovn-controller recompute And see if that fixes the issue. If it does, it would be great to have OpenFlow dumps before and after the recompute for comparison. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovsdb-server fails to start and report one unreferenced file path doesn't exist
On 4/24/24 12:12, Ilya Maximets wrote: > On 4/24/24 05:32, Wang, Fei2 via discuss wrote: >> Hi all, >> >> >> >> We are hitting below error when trying to start ovsdb-server 3.3.0 on SLES15 >> SP5, I don’t tend to believe this issue is related to OS but not sure, we >> poked around and found similar issue >> https://bugzilla.redhat.com/show_bug.cgi?id=1785586 >> <https://bugzilla.redhat.com/show_bug.cgi?id=1785586>, since we are using >> OVS built from source code with dpdk enabled so cannot reference to it >> directly, any idea how to narrow down this issue further? Thanks a lot. >> >> *Our OVS build and startup steps:* >> >> ./configure --with-dpdk=static >> make >> make install >> >> mkdir -p /usr/local/etc/openvswitch >> rm -rf /usr/local/etc/openvswitch/conf.db >> ovsdb-tool create /usr/local/etc/openvswitch/conf.db >> /usr/local/share/openvswitch/vswitch.ovsschema >> mkdir -p /usr/local/var/run/openvswitch/ >> >> # Start ovsdb-server >> >> ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock >> --remote=db:Open_vSwitch,Open_vSwitch,manager_options >> --private-key=db:Open_vSwitch,SSL,private_key >> --certificate=db:Open_vSwitch,SSL,certificate >> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach > > For some reason the pidfile creation is happening in > /usr/var/run/openvswitch/ and not /usr/local/... . > Sounds strange. Also , make sure that you're running the same binary that you built. i.e. check the 'which ovsdb-server', it should be in /usr/local/bin. > > Try to run 'make distclean' and re-configure the build. > > Or use explicit configuration to set up the prefixes: > > ./configure --prefix=/usr/local \ > --localstatedir=/usr/local/var \ > --sysconfdir=/usr/local/etc > > Another option is to provide --pidfile option to ovsdb-server > specifying the desired location. > > Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovsdb-server fails to start and report one unreferenced file path doesn't exist
On 4/24/24 05:32, Wang, Fei2 via discuss wrote: > Hi all, > > > > We are hitting below error when trying to start ovsdb-server 3.3.0 on SLES15 > SP5, I don’t tend to believe this issue is related to OS but not sure, we > poked around and found similar issue > https://bugzilla.redhat.com/show_bug.cgi?id=1785586 > <https://bugzilla.redhat.com/show_bug.cgi?id=1785586>, since we are using OVS > built from source code with dpdk enabled so cannot reference to it directly, > any idea how to narrow down this issue further? Thanks a lot. > > *Our OVS build and startup steps:* > > ./configure --with-dpdk=static > make > make install > > mkdir -p /usr/local/etc/openvswitch > rm -rf /usr/local/etc/openvswitch/conf.db > ovsdb-tool create /usr/local/etc/openvswitch/conf.db > /usr/local/share/openvswitch/vswitch.ovsschema > mkdir -p /usr/local/var/run/openvswitch/ > > # Start ovsdb-server > > ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock > --remote=db:Open_vSwitch,Open_vSwitch,manager_options > --private-key=db:Open_vSwitch,SSL,private_key > --certificate=db:Open_vSwitch,SSL,certificate > --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach For some reason the pidfile creation is happening in /usr/var/run/openvswitch/ and not /usr/local/... . Sounds strange. Try to run 'make distclean' and re-configure the build. Or use explicit configuration to set up the prefixes: ./configure --prefix=/usr/local \ --localstatedir=/usr/local/var \ --sysconfdir=/usr/local/etc Another option is to provide --pidfile option to ovsdb-server specifying the desired location. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Question on listen backlog
On 4/4/24 18:07, Brian Haley wrote:
> Hi,
>
> On 4/4/24 6:12 AM, Ilya Maximets wrote:
>> On 4/3/24 22:15, Brian Haley via discuss wrote:
>>> Hi,
>>>
>>> I recently have been seeing issues in a large environment where the
>>> listen backlog of ovsdb-server, both NB and SB, is getting over-flowed,
>>> for example:
>>>
>>> 17842 times the listen queue of a socket overflowed
>>> 17842 SYNs to LISTEN sockets dropped
>>
>> Does this cause significant re-connection delays or is it just an
>> observation?
>
> It is just an observation at this point.
Ack.
>
>>> There is more on NB than SB, but I was surprised to see any. I can only
>>> guess at the moment it is happening when the leader changes and hundreds
>>> of nodes try and reconnect.
>>
>> This sounds a little strange. Do you have hundreds leader-only clients
>> for Northbound DB? In general, only write-heavy clients actually need
>> to be leader-only.
>
> There are a lot of leader-only clients due to the way the neutron API
> server runs - each worker thread has a connection, and they are scaled
> depending on processor count, so typically there are at least 32. Then
> multiply that by three since there is HA involved.
>
> Actually I had a look in a recent report and there were 61 NB/62 SB
> connections per system, so that would make ~185 for each server. I would
> think in a typical deployment there might be closer to 100.
>
>>> Looking at their sockets I can see the backlog is only set to 10:
>>>
>>> $ ss -ltm | grep 664
>>> LISTEN 0 10 0.0.0.0:66410.0.0.0:*
>>> LISTEN 0 10 0.0.0.0:66420.0.0.0:*
>>>
>>> Digging into the code, there is only two places where listen() is
>>> called, one being inet_open_passive():
>>>
>>> /* Listen. */
>>> if (style == SOCK_STREAM && listen(fd, 10) < 0) {
>>> error = sock_errno();
>>> VLOG_ERR("%s: listen: %s", target, sock_strerror(error));
>>> goto error;
>>> }
>>>
>>> There is no way to config around this to even test if increasing would
>>> help in a running environment.
>>>
>>> So my question is two-fold:
>>>
>>> 1) Should this be increased? 128, 256, 1024? I can send a patch.
>>>
>>> 2) Should this be configurable?
>>>
>>> Has anyone else seen this?
>>
>> I don't remember having any significant issues related to connection
>> timeouts as they usually getting resolved quickly. And if the server
>> doesn't accept the connection fast enough it means that the server is
>> busy and there may not be real benefit from having more connections
>> in the backlog. It may just hide the connection timeout warning while
>> service will not actually be available for the roughly the same amount
>> of time anyway. Having lower backlog may allow clients to re-connect
>> to a less loaded server faster.
>
> Understood, increasing the backlog might just hide the warnings and not
> fix the issue.
>
> I'll explain what seems to be happening, at least from looking at the
> logs I have. All the worker threads in question are happily connected to
> the leader. When the leader changes there is a bit of a stampede while
> they all try and re-connect to the new leader. But since they don't know
> which of the three (again, HA) systems are the leader, they just pick
> one of the other two. When they don't get the leader they disconnect and
> try another.
>
> It might be there is something we can do on the neutron side as well,
> the 10 backlog just seemed like the first place to start.
I believe I heard something about adjusting the number of connections
in neutron, but I don't have any specific pointers. Maybe Ihar knows
something about it?
>
>> Saying that, the original code clearly wasn't designed for a high
>> number of simultaneous connection attempts, so it makes sense to
>> increase the backlog to some higher value. I see Ihar re-posted his
>> patch doing that here:
>>
>> https://patchwork.ozlabs.org/project/openvswitch/patch/20240403211818.10023-1-ihrac...@redhat.com/
>> I'll take a look at it.
>
> Thanks, I plan on testing that as well.
>
>> One other thing that we could do is to accept more connections at a time.
>> Currently we accept one connection per event loop iteration. But we
>> need to be careful here as handling multiple initial monitor requests
>> for the database within a single ite
Re: [ovs-discuss] Question on listen backlog
On 4/3/24 22:15, Brian Haley via discuss wrote:
> Hi,
>
> I recently have been seeing issues in a large environment where the
> listen backlog of ovsdb-server, both NB and SB, is getting over-flowed,
> for example:
>
> 17842 times the listen queue of a socket overflowed
> 17842 SYNs to LISTEN sockets dropped
Does this cause significant re-connection delays or is it just an
observation?
>
> There is more on NB than SB, but I was surprised to see any. I can only
> guess at the moment it is happening when the leader changes and hundreds
> of nodes try and reconnect.
This sounds a little strange. Do you have hundreds leader-only clients
for Northbound DB? In general, only write-heavy clients actually need
to be leader-only.
>
> Looking at their sockets I can see the backlog is only set to 10:
>
> $ ss -ltm | grep 664
> LISTEN 0 10 0.0.0.0:66410.0.0.0:*
> LISTEN 0 10 0.0.0.0:66420.0.0.0:*
>
> Digging into the code, there is only two places where listen() is
> called, one being inet_open_passive():
>
> /* Listen. */
> if (style == SOCK_STREAM && listen(fd, 10) < 0) {
> error = sock_errno();
> VLOG_ERR("%s: listen: %s", target, sock_strerror(error));
> goto error;
> }
>
> There is no way to config around this to even test if increasing would
> help in a running environment.
>
> So my question is two-fold:
>
> 1) Should this be increased? 128, 256, 1024? I can send a patch.
>
> 2) Should this be configurable?
>
> Has anyone else seen this?
I don't remember having any significant issues related to connection
timeouts as they usually getting resolved quickly. And if the server
doesn't accept the connection fast enough it means that the server is
busy and there may not be real benefit from having more connections
in the backlog. It may just hide the connection timeout warning while
service will not actually be available for the roughly the same amount
of time anyway. Having lower backlog may allow clients to re-connect
to a less loaded server faster.
Saying that, the original code clearly wasn't designed for a high
number of simultaneous connection attempts, so it makes sense to
increase the backlog to some higher value. I see Ihar re-posted his
patch doing that here:
https://patchwork.ozlabs.org/project/openvswitch/patch/20240403211818.10023-1-ihrac...@redhat.com/
I'll take a look at it.
One other thing that we could do is to accept more connections at a time.
Currently we accept one connection per event loop iteration. But we
need to be careful here as handling multiple initial monitor requests
for the database within a single iteration may be costly and may reduce
overall responsiveness of the server. Needs some research.
Having hundreds leader-only clients for Nb still sounds a little strange
to me though.
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Cannot remove mirror (possibly) added by ovs-tcpdump
On 3/30/24 12:13, Giovanni Tirloni via discuss wrote: > Hello, > > I was playing with ovs-vsctl, adding and removing ports/interfaces and > checking some traffic with ovs-tcpdump. Later, I noticed the following > warning in the logs: > > ovs-vswitchd[156154]: ovs|02897|bridge|ERR|bridge ovs-default: > mirror m_bond0.10 does not specify output; ignoring > > I tried to remove it manually but got the following error: > > # ovs-vsctl remove mirror b20f3ce4-c5c2-453a-9fa4-99175be0a02a name > m_bond0.10 > ovs-vsctl: "remove" operation would put 0 values in column name of > table Mirror but the minimum number is 1 > > I'm not sure why the minimum would be 1 and how to make that warning go away. As the error message says, you're trying to remove the name from the mirror and not the mirror itself. Mirror must have a name, so the operation fails. And you will also not be able to remove the mirror record while bridge is referencing it. You need to remove the mirror from the bridge and other database tables will be cleaned up automatically. If you don't have other mirrors, the easiest way to remove this one will be: # ovs-vsctl clear bridge ovs-default mirrors If you only want to remove this particular mirror: # ovs-vsctl remove bridge ovs-default mirrors b20f3ce4-c5c2-453a-9fa4-99175be0a02a Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Huge Logical_DP_Groups; NorthD 100% CPU
On 3/28/24 14:24, Шагов Георгий via discuss wrote: > Hello everyone > > In our setup (*) we have encountered a problem with NorthD consuming 100% CPU > Looking into transaction log of SBDB it was found that some transactions from > NorthD take almost 2Mb in Json; these are not exception, rather regular > Looking deeply into the transaction we have encountered Huge Logical_DP_Group > Record, > More in details: 2Mb Json SBDB Transaction Record was parsed by json_pp to > json > file, that takes 157K Lines, from which 152K Lines – Logical_DP_Group > That Logical_DP_Group record contains of 10 Groups into it. Approx 4K dp per > every group > > Googling for a similar case I have found one, that looked close to our: > (link1) – > this is not our case – we checked that > Unfortunately, the basic information regarding Logical DB Groups is well > documented > into the code of NorthD > > Any clue/help/questions are extremely appreciated > > (*) Setup: > > - OVN 22.03.3 – we are in process of migrating to 24.03 > - OVS 2.17.7 – the same for 3.3 22.03 and 2.17 are not a particularly great choice for high-scale deployments. And they have numerous inefficiencies fixed in later versions. In general, I'd not recommend versions below 23.06 or at the very least 22.09. Databases in newer OVS releases are much more performant than 2.17 as well. I'd suggest you proceed with your migration/upgrade. I would expect at least 10x performance improvement from it, maybe even more. Best regards, Ilya Maximets. > - Routers – approx. 5K > > (link1): > https://github.com/ovn-org/ovn/commit/db4cb7098c890e974175d4d05dd70dc409fad91e > > Yours truly, George ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Enabling TSO functionality affects virtual machine TCP traffic
On 3/25/24 08:06, Felix Huettner wrote: >>> >>> Hi everyone, >>> >>> i have built a first version of this here and it does indeed solve our >>> issues. Not sure though if there is more. I would like to send it to >>> ovs-dev sometime next week once i added some tests. >>> In the mean time you can take a look here: >>> >>> https://github.com/openvswitch/ovs/compare/master...felixhuettner:ovs:fix_it >> >> Thanks, Felix! This is a possible route to take, however, there is >> still an issue with miniflow_extract() requesting UDP checksum offload >> not realizing that it requests it for an inner header. >> >> >> IMHO, the fact that flag can mean different things at different times >> is the root of many of the problems with offloading and we'll keep >> getting issues like this until API is fixed. >> >> >> I sent my version of the fix here: >> >> https://patchwork.ozlabs.org/project/openvswitch/patch/20240322144213.1073946-1-i.maximets%40ovn.org/ >> >> This should resolve the issue, but at a cost of dropping tunneled >> traffic when userspace-tso is enabled. >> >> Would be great if you can test it out. > > Hi Ilya, > > thanks for the fix. > It works great for me. Thanks for testing! Best regards, Ilya Maximets. > > Thanks > Felix > >> >> The other big issue is that these crashes are happening in the default >> configuration, i.e. do not actually require users to opt-in for >> userspace-tso. >> >> Best regards, Ilya Maximets. >> ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd running at 100% cpu
On 11/1/22 13:30, Donald Sharp wrote:
>
>
> On Mon, Oct 31, 2022 at 5:54 PM Ilya Maximets <mailto:i.maxim...@ovn.org>> wrote:
>
> On 10/31/22 17:25, Donald Sharp via discuss wrote:
> > Hi!
> >
> > I work on the FRRouting project (https://frrouting/org
> <https://frrouting/org> <https://frrouting/org <https://frrouting/org>> ) and
> am doing work with FRR and have noticed that when I have a full BGP feed on a
> system that is also running ovs-vswitchd that ovs-vswitchd sits at 100% cpu:
> >
> > top - 09:43:12 up 4 days, 22:53, 3 users, load average: 1.06, 1.08,
> 1.08
> > Tasks: 188 total, 3 running, 185 sleeping, 0 stopped, 0 zombie
> > %Cpu(s): 12.3 us, 14.7 sy, 0.0 ni, 72.8 id, 0.0 wa, 0.0 hi, 0.2 si,
> 0.0 st
> > MiB Mem : 7859.3 total, 2756.5 free, 2467.2 used, 2635.6
> buff/cache
> > MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 5101.9 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
> > 730 root 10 -10 146204 146048 11636 R 98.3 1.8 6998:13
> ovs-vswitchd
>
> > 169620 root 20 0 0 0 0 I 3.3 0.0 1:34.83
> kworker/0:3-events
>
> > 21 root 20 0 0 0 0 S 1.3 0.0 14:09.59
> ksoftirqd/1
>
> > 131734 frr 15 -5 2384292 609556 6612 S 1.0 7.6 21:57.51
> zebra
>
> > 131739 frr 15 -5 1301168 1.0g 7420 S 1.0 13.3 18:16.17
> bgpd
> >
> > When I turn off FRR ( or turn off the bgp feed ) ovs-vswitchd stops
> running at 100%:
> >
> > top - 09:48:12 up 4 days, 22:58, 3 users, load average: 0.08, 0.60,
> 0.89
> > Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie
> > %Cpu(s): 0.2 us, 0.4 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.1 si,
> 0.0 st
> > MiB Mem : 7859.3 total, 4560.6 free, 663.1 used, 2635.6
> buff/cache
> > MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 6906.1 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
> > 179064 sharpd 20 0 11852 3816 3172 R 1.0 0.0 0:00.09
> top
>
> > 1037 zerotie+ 20 0 291852 113180 7408 S 0.7 1.4 19:09.17
> zerotier-one
>
> > 1043 Debian-+ 20 0 34356 21988 7588 S 0.3 0.3 22:04.42
> snmpd
>
> > 178480 root 20 0 0 0 0 I 0.3 0.0 0:01.21
> kworker/1:2-events
>
> > 178622 sharpd 20 0 14020 6364 4872 S 0.3 0.1 0:00.10
> sshd
>
> > 1 root 20 0 169872 13140 8272 S 0.0 0.2 2:33.26
> systemd
>
> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.60
> kthreadd
> >
> > I do not have any particular ovs configuration on this box:
> > sharpd@janelle:~$ sudo ovs-vsctl show
> > c72d327c-61eb-4877-b4e7-dcf7e07e24fc
> > ovs_version: "2.13.8"
> >
> >
> > sharpd@janelle:~$ sudo ovs-vsctl list o .
> > _uuid : c72d327c-61eb-4877-b4e7-dcf7e07e24fc
> > bridges : []
> > cur_cfg : 0
> > datapath_types : [netdev, system]
> > datapaths : {}
> > db_version : "8.2.0"
> > dpdk_initialized : false
> > dpdk_version : none
> > external_ids : {hostname=janelle, rundir="/var/run/openvswitch",
> system-id="
Re: [ovs-discuss] Enabling TSO functionality affects virtual machine TCP traffic
On 3/22/24 15:30, Felix Huettner wrote: >>> >>> Thanks for the report and analysis! >>> >>> I agree, this is indeed a bug in recirculation + tunnel offload logic. >>> >>> To recap: >>> >>> 1. TCP packet enters OVS. >>> 2. TCP packet gets encapsulated into UDP tunnel. >>> 3. Recirculation happens. >>> 4. Packet is re-parsed after recirculation with miniflow_extract() >>>or similar function. >>> 5. Packet is marked for UDP checksumming because we parse the outermost set >>>of headers. But since it is tunneled, it means inner UDP checksumming. >>>And that makes no sense, because the inner packet is TCP. >> >> Hi everyone, >> >> we are actually seeing a similar issue, allthough in our case we are >> crashing ovs with the following assertion when sending pings: >> >> util(pmd-c07/id:9)|EMER|lib/dp-packet.h:1425: assertion ip failed in >> dp_packet_ip_set_header_csum() >> >> or a segmentation fault when sending tcp traffic (because >> packet_tcp_complete_csum has not asserts) >> >>> >>> I'm assuming that this re-parsing is also messing up all the offsets in the >>> dp-packet. We likely need fix all the checksums before recirculation and >>> clear all the flags. For TSO however that would mean that we will need to >>> drop the packet as we don't have GSO yet and we can't actually segment >>> traffic in the middle of pipeline anyway... >> >> So miniflow_extract resets inner_l3_ofs and inner_l4_ofs quite at the >> start of the function and has no logic to add these values back (at >> least as far as i saw that). >> >>> >>> Mike, any thoughts on this? >>> >>> >>> A workaround for this particular situation hwoever would be enabling support >>> for lb-output action on the bond: >>> >>> ovs-vsctl set Port bond0 other_config:lb-output-action=true >> >> This helps at least for our issue. >> >>> >>> This should avoid the recirculation in this particular scenario. >>> But we still need to find a solution for the recirculation issue. >> >> I currently have no real idea how we could recover inner_l3_ofs for >> geneve/vxlan packets. Since for UDP traffic we do not know the upper >> protocol it is basically impossible to find out for sure if a given >> communication is geneve/vxlan. Therefor recovering inner_l3_ofs for a >> given packet sounds to me either like guessing or not really possible. >> >> An alternative i have though about is not resetting the offsets in >> miniflow_extract for packets that are recirculating. In this case we >> should have started with the original packet at some point and only >> modified it by actions we defined. If all of these actions modify the >> offsets as appropriate is there any reason at all to recalculate them? > > Hi everyone, > > i have built a first version of this here and it does indeed solve our > issues. Not sure though if there is more. I would like to send it to > ovs-dev sometime next week once i added some tests. > In the mean time you can take a look here: > > https://github.com/openvswitch/ovs/compare/master...felixhuettner:ovs:fix_it Thanks, Felix! This is a possible route to take, however, there is still an issue with miniflow_extract() requesting UDP checksum offload not realizing that it requests it for an inner header. IMHO, the fact that flag can mean different things at different times is the root of many of the problems with offloading and we'll keep getting issues like this until API is fixed. I sent my version of the fix here: https://patchwork.ozlabs.org/project/openvswitch/patch/20240322144213.1073946-1-i.maxim...@ovn.org/ This should resolve the issue, but at a cost of dropping tunneled traffic when userspace-tso is enabled. Would be great if you can test it out. The other big issue is that these crashes are happening in the default configuration, i.e. do not actually require users to opt-in for userspace-tso. Best regards, Ilya Maximets. > > Thanks > Felix > >> >> The really ugly alternative that is probably also rather guesswork would >> be: >> <<< start of miniflow_extract >> tun_l3_ofs = inner_l3_ofs - l4_ofs; >> tun_l4_ofs = inner_l4_ofs - inner_l3_ofs; >> <<< main miniflow_extract code >> packet->inner_l3_ofs = packet->l4_ofs + tun_l3_ofs; >> packet->inner_l4_ofs = packet->inner_l3_ofs + tun_l4_ofs; >> <<< end of miniflow_extract >> >> But i do not think that is a good idea. >> >> Thanks >> Felix >> >> ___ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] kernel module build
On 3/22/24 00:20, Cpp Code via discuss wrote: > Hi, > > Now with kernel modules residing inside linux repository, I am unable > to figure out how to build them so they would match my kernel. I don't think it is generally possible. You should use the version that is provided with your kernel. If you want to make changes in the module, you need to get sources of your current running kernel, make changes in the openvswitch module provided in it and re-build that kernel or just a module from it. Kernel will normally refuse loading of the module that is built for different (even minor) version, so you can't use the module from a different set of sources. > > I have both: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > and > > https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy > > repositories and Ubuntu: > > $ uname -a > Linux virtual-machine 6.5.0-25-generic #25~22.04.1-Ubuntu SMP > PREEMPT_DYNAMIC Tue Feb 20 16:09:15 UTC 2 x86_64 x86_64 x86_64 > GNU/Linux > > What is the correct way to build openvswitch kernel module? So, in this case you should be able to make changes in the code in the ubuntu repo and re-build it. This module should be loadable into your current kernel as long as the version of the running kernel matches exactly the version in the git repo. If you can't find the exact commit in the ubuntu repo that corresponds to 6.5.0-25-generic build, then you'll need to get source rpm, make changes in it and re-build the RPM. An easier way might be to just build and install a new kernel altogether from an upstream repo. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Enabling TSO functionality affects virtual machine TCP traffic
On 3/21/24 08:48, Qiang Qiang45 Zhang via discuss wrote: > Problem: > > When the Geneve tunnel is the Bond port, TCP traffic between virtual machines > is not connected, and PING is allowed; When the Geneve tunnel is a single > physical port, both TCP and PING are normal. > > Version: > > ovs-vswitchd (Open vSwitch) 3.3.0 > DPDK 23.11.0 > > Main configuration: > > ovs-vsctl set op . other_config:userspace-tso-enable=true > ovs-vsctl add-br int-br -- set bridge int-br datapath_type=netdev > ovs-vsctl add-port int-br geneve0 -- set interface geneve0 type=geneve -- set > interface geneve0 options:csum=false -- set interface geneve0 > options:key=flow -- set interface geneve0 options:remote_ip="192.168.54.183" > ovs-vsctl add-port int-br vm0 -- set interface vm0 type=dpdkvhostuserclient > ovs-ofctl show int-br > ovs-ofctl del-flows int-br > ovs-ofctl add-flow int-br 'table=0,in_port=2,actions=output:1' > ovs-ofctl add-flow int-br 'table=0,in_port=1,actions=output:2' > > > > Topology: > > Key points for investigation: Comparison between ordinary physical and BOND > tunnel entrances > > The tunnel entrance is a single physical entrance: > > The tunnel entrance is a bond: > > The key issue is that there is an additional redirect (0x1) for the bond, > which > will cause the corresponding flag to be filled with outer Geneve information > when > setting mbuf flags. The actual GDB stack for virtual machine debugging is as > follows > (the inner layer is SSH TCP traffic, but it has set the csum_udp identifier. > This > analysis code reads the outer layer header, which is the UDP header of Geneve) Thanks for the report and analysis! I agree, this is indeed a bug in recirculation + tunnel offload logic. To recap: 1. TCP packet enters OVS. 2. TCP packet gets encapsulated into UDP tunnel. 3. Recirculation happens. 4. Packet is re-parsed after recirculation with miniflow_extract() or similar function. 5. Packet is marked for UDP checksumming because we parse the outermost set of headers. But since it is tunneled, it means inner UDP checksumming. And that makes no sense, because the inner packet is TCP. I'm assuming that this re-parsing is also messing up all the offsets in the dp-packet. We likely need fix all the checksums before recirculation and clear all the flags. For TSO however that would mean that we will need to drop the packet as we don't have GSO yet and we can't actually segment traffic in the middle of pipeline anyway... Mike, any thoughts on this? A workaround for this particular situation hwoever would be enabling support for lb-output action on the bond: ovs-vsctl set Port bond0 other_config:lb-output-action=true This should avoid the recirculation in this particular scenario. But we still need to find a solution for the recirculation issue. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [ovs-dev] [PATCH] Rename primary development branch as main.
-git a/Documentation/tutorials/faucet.rst
> b/Documentation/tutorials/faucet.rst
> index 6aa4d39aa8ab..33e4543e4023 100644
> --- a/Documentation/tutorials/faucet.rst
> +++ b/Documentation/tutorials/faucet.rst
> @@ -27,7 +27,7 @@ OVS Faucet Tutorial
>
> This tutorial demonstrates how Open vSwitch works with a general-purpose
> OpenFlow controller, using the Faucet controller as a simple way to get
> -started. It was tested with the "master" branch of Open vSwitch and version
> +started. It was tested with the "main" branch of Open vSwitch and version
> 1.6.15 of Faucet. It does not use advanced or recently added features in OVS
> or Faucet, so other versions of both pieces of software are likely to work
> equally well.
> @@ -68,7 +68,7 @@ approaches:
> $ git clone https://github.com/openvswitch/ovs.git
> $ cd ovs
>
> - The default checkout is the master branch. You will need to use the
> master
> + The default checkout is the main branch. You will need to use the main
> branch for this tutorial as it includes some functionality required for
> this
> tutorial.
>
> @@ -84,7 +84,7 @@ approaches:
>
> The default behaviour for some of the commands used in this tutorial
> changed in Open vSwitch versions 2.9.x and 2.10.x which breaks the
> - tutorial. We recommend following step 3 and building master from
> + tutorial. We recommend following step 3 and building main from
> source or using a system Open vSwitch that is version 2.8.x or older.
>
> If it is successful, you will find yourself in a subshell environment,
> which
> diff --git a/Documentation/tutorials/ovs-conntrack.rst
> b/Documentation/tutorials/ovs-conntrack.rst
> index e8a58c4eb298..6b0b73cd1173 100644
> --- a/Documentation/tutorials/ovs-conntrack.rst
> +++ b/Documentation/tutorials/ovs-conntrack.rst
> @@ -35,7 +35,7 @@ to match on the TCP segments from connection setup to
> connection tear down.
> It will use OVS with the Linux kernel module as the datapath for this
> tutorial. (The datapath that utilizes the openvswitch kernel module to do
> the packet processing in the Linux kernel)
> -It was tested with the "master" branch of Open vSwitch.
> +It was tested with the "main" branch of Open vSwitch.
>
> Definitions
> ---
> diff --git a/NEWS b/NEWS
> index c9e4064e67a7..9c3e59455d29 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -4,7 +4,9 @@ Post-v3.3.0
> * Conntrack now supports 'random' flag for selecting ports in a range
> while natting and 'persistent' flag for selection of the IP address
> from a range.
> -
Please, keep two empty lines between sections for different releases.
> + - The primary development branch has been renamed from 'master' to 'main'.
> +The OVS tree remains hosted on GitHub.
> +https://github.com/openvswitch/ovs.git
>
> v3.3.0 - 16 Feb 2024
>
> diff --git a/README.rst b/README.rst
> index a2c234f4d17c..ca9e386c2069 100644
> --- a/README.rst
> +++ b/README.rst
> @@ -8,7 +8,7 @@ Open vSwitch
>
> .. image::
> https://github.com/openvswitch/ovs/workflows/Build%20and%20Test/badge.svg
> :target: https://github.com/openvswitch/ovs/actions
> -.. image::
> https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch=master=true=true
> +.. image::
> https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch=main=true=true
> :target: https://ci.appveyor.com/project/blp/ovs/history
> .. image:: https://api.cirrus-ci.com/github/openvswitch/ovs.svg
> :target: https://cirrus-ci.com/github/openvswitch/ovs
> diff --git a/appveyor.yml b/appveyor.yml
> index 29cc44d6c6f6..65e29eb4e3be 100644
> --- a/appveyor.yml
> +++ b/appveyor.yml
> @@ -2,7 +2,7 @@ version: 1.0.{build}
> image: Visual Studio 2019
> branches:
>only:
> - - master
> + - main
Should be done in two stages as described above.
> configuration:
>- Debug
>- Release
> @@ -23,7 +23,7 @@ install:
> New-Item -ItemType Directory -Force -Path C:\ovs-build-downloads
>
> # Find and download the latest stable OpenSSl 3.0.
> -$URL =
> "https://raw.githubusercontent.com/slproweb/opensslhashes/master/win32_openssl_hashes.json;
> +$URL =
> "https://raw.githubusercontent.com/slproweb/opensslhashes/main/win32_openssl_hashes.json;
As mentioned in the other email, this is not a correct change
as it is not our repo.
> $webData = (Invoke-WebRequest -Uri $URL).content | ConvertFrom-Json
> $source = ($webData.files.PSObject.Properties | Where-Object {
> $_.Value.basever -match "3.0.*" -and
> @@ -74,6 +74,6 @@ build_script:
> c:\OpenvSwitch-$env:CONFIGURATION.msi
>
> after_build:
> -- ps: 7z a C:\ovs-master-$env:CONFIGURATION.zip C:\openvswitch
> -- ps: Push-AppveyorArtifact C:\ovs-master-$env:CONFIGURATION.zip
> +- ps: 7z a C:\ovs-main-$env:CONFIGURATION.zip C:\openvswitch
> +- ps: Push-AppveyorArtifact C:\ovs-main-$env:CONFIGURATION.zip
> - ps: Push-AppveyorArtifact C:\OpenvSwitch-$env:CONFIGURATION.msi
>
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] What version of ovsdbapp is compatible with the latest OVN LTS?
On 3/18/24 17:35, Brendan Doyle via discuss wrote: > Folks, > > > Here are the versions of OVN/OVS that we are currently using: > > > ovn v22.03.2 (released Dec 20, 2022) > > NB DB Schema 6.1.0 > SB DB Schema 20.21.0 > > ovs v2.17.7 (released Jun 27, 2023) > DB Schema 8.3.0 > > ovsdbapp v2.4.0 (released Jul 05, 2023) > > python_ovs library - 2.16.90 > > Is this the correct version of ovsdbapp/python_ovs library For the > OVN/OVS versions above? Hi, Brendan. x.x.90 is almost never a correct version to use as it means that you're using something that wasn't actually released. Also, 2.16 is kind of old at this point and not supported anymore. In general, python ovs library should be compatible with any version of database schemas, i.e. any release of OVS/OVN, since it's not bound to any specific database. On this topic I would just suggest to upgrade to a newer supported version, e.g. 2.17.9 or 3.3.0. Note that python-openvswitch distribution packages are preferred to a pip install versions, since they are much more performant due to support for JSON parsing via C bindings. ovsdbapp itself is using the schema knowledge, but it tries to be compatible with different versions as well, IIRC. CC: Terry, he may have some more details on this. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [External] : 01775|raft|INFO|current entry eid 1da31769-d05e-4d07-ad03-6eb680bdb67f does not match prerequisite eb3b4396-c15f-4fc9-911e-1ce8a4795ec8 in execute_command_request
On 3/15/24 20:10, Brendan Doyle via discuss wrote: > > > On 15/03/2024 19:07, Brendan Doyle wrote: >> I found this old email thread: >>> On Tue, Oct 20, 2020 at 11:55 AM Tony Liu >>> wrote: > > Hi, > > From ovnsb log, I see many of the following messages. > >>> What does it mean? Is that a concern? > > >>> 2020-10-20T18:52:50.483Z|00093|raft|INFO|current entry eid >>> 2ab3eff8-87e1-4e19-9a1f-d359ad56a9ad does not match prerequisite >>> c6ffd854-6f6e-4533-a6d8-b297acb542e0 in execute_command_request > > > >>> Thanks! > Tony This is expected in cluster mode when there are concurrent >>> transactions initiated from different cluster nodes. It is recommended to >>> connect to the leader for write-heavy clients. Occasional conflict is ok >>> and the transaction will be resubmitted. Thanks, Han >> >> But we are seeing 10's of 100's of these messages! > > 10's of 1000's !! Han's old reply still stands. You're likely sending a lot of write transactions from multiple clients connected to different followers. They'll all be re-tried, but it's better to connect to the leader instead for write-heavy operations. In the past ovn-kubernetes used a version of libovsdb that wouldn't re-connect to a new leader and would keep sending transactions through the follower causing a lot of such logs. But it was Northbound, not Southbound. You need to find which processes are sending many transactions to Sb followers. Best regards, Ilya Maximets. >> >> On 15/03/2024 18:41, Brendan Doyle via discuss wrote: >>> Hi, >>> >>> >>> We run OVN central as a RAFT cluster of 3 in a k8 DaemonSet. We recently >>> upgraded to the latest >>> LTS, and started see loads of these >>> >>> 2024-03-14T20:17:44.677Z|01779|raft|INFO|current entry eid >>> e857968d-5da5-4b8a-9a17-f06075e67703 does not match prerequisite >>> 025d7ea8-2c1d-4177-986f-1b6617ff5bed in execute_command_request >>> 2024-03-14T20:17:47.315Z|01780|raft|INFO|Dropped 171 log messages in last 3 >>> seconds (most recently, 3 seconds ago) due to excessive rate >>> >>> In ovsdb-server-sb.log >>> >>> Has anybody got an idea what they mean? >>> >>> >>> Thanks >>> >>> Brendan >>> ___ >>> discuss mailing list >>> disc...@openvswitch.org >>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!J2ZmXngj-e7hJHejBcN5FYYihte0Ng8x9wRXbbuVIUH1DEGVoxAmCB_KM9gYpMLhbDuQhQBlec-ZI8SiuUGkeAvUpfDdMg$ >>> >> > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] A use-after-free defect was discovered at line 1251 of the file /ovs/ovsdb/transaction.c.
On 3/11/24 03:58, 尹麓鸣 via discuss wrote: > Dear OpenvSwitch Developers, > > I hope this email finds you well. I am writing to report a potential > vulnerability found in the /ovs/ovsdb/transaction.c file. > > Upon investigation, it has been discovered that there exists a use-after-free > defect at line 1251 of the mentioned file. For detailed information regarding > this defect, please refer to the following link: > https://github.com/LuMingYinDetect/openvswitch_defects/blob/main/openvswitch_detect_1.md > > <https://github.com/LuMingYinDetect/openvswitch_defects/blob/main/openvswitch_detect_1.md>. > > As a responsible member of the community, I believe it is crucial to promptly > address such security concerns to ensure the integrity and reliability of the > Open vSwitch project. > > Thank you for your attention to this matter. Please let me know if you > require any further information or assistance from my end. > > Best regards, > LuMingYin Hi, LuMingYin. Thanks for the report. See my reply on the github issue: https://github.com/openvswitch/ovs-issues/issues/322 This is not a security issue, because the code in question is not reachable, however, for the future, please report security issues to ovs-secur...@openvswitch.org instead of public forums. Thanks! If you want to make a cosmetic change removing the incorrect ovsdb_transaction_abort() call, feel free to post a patch to ovs-...@openvswitch.org. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Mirror: ovs-tcpdump cannot capture vlan packets on the port with tag
On 3/7/24 04:50, Zhangweiwei wrote:
> I have configured vlan-limit to 0, but the packets still are not delivered to
> mitapVm72.
> Here is the track information:
>
> [root@localhost ~]# ovs-vsctl set o . other_config:vlan-limit=0
> [root@localhost ~]#
> [root@localhost ~]# ovs-appctl ofproto/trace vds1-br in_port=tapVm72
> 52540067d5615254009abfed81640800455458ad4000400151f502024602020246010800dde3067a0078e030e96588c00200101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
> Flow:
> icmp,in_port=6,dl_vlan=100,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=52:54:00:9a:bf:ed,dl_dst=52:54:00:67:d5:61,nw_src=2.2.70.2,nw_dst=2.2.70.1,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
>
> bridge("vds1-br")
> -
> 0. in_port=6, priority 32768
> >>>> dropping VLAN 100 tagged packet received on port tapVm72 configured
> as VLAN 0 access port <<<<
> output:3
> >>>> dropping VLAN 100 tagged packet received on port tapVm71 configured
> as VLAN 0 access port <<<<
>
> Final flow: unchanged
> Megaflow: recirc_id=0,eth,ip,in_port=6,nw_frag=no
> Datapath actions: 6
>
>
> Packets are mirrored in this function. input_vid_is_valid() checked failed
> when a packet with vlan ingress on tapVm72 , because tapVm72 is an access
> port.
I see. That makes sense. The issue is caused by the mix of the NORMAL
pipeline processing and direct forwarding. There are two ways you can
fix the issue:
1. Remove the tag 0 from the ports, so they are not access ports anymore.
Is there a reason for them to be access ports if you're not using the
NORMAL action? Also, the tag 0 is a special value that can be ignored,
so I'd advise to not use it.
2. Change the type from access to dot1q-tunnel. This way double tagging
will be allowed and the packets should not be dropped anymore.
I don't think the vlan check can be easily removed from the mirroring
code, because we should not mirror a packet if it is going to be dropped
by a NORMAL pipleline later.
Best regrads, Ilya Maximets.
> mirror_packet(struct xlate_ctx *ctx, struct xbundle *xbundle,
> mirror_mask_t mirrors)
> {
> struct xvlan in_xvlan;
> struct xvlan xvlan;
>
> /* Figure out what VLAN the packet is in (because mirrors can select
> * packets on basis of VLAN). */
> xvlan_extract(>xin->flow, _xvlan);
> if (!input_vid_is_valid(ctx, in_xvlan.v[0].vid, xbundle)) {
> return;
> }
>
> -邮件原件-
> 发件人: Ilya Maximets [mailto:i.maxim...@ovn.org]
> 发送时间: 2024年3月6日 19:43
> 收件人: zhangweiwei (RD) ; ovs-discuss@openvswitch.org
> 抄送: i.maxim...@ovn.org
> 主题: Re: [ovs-discuss] Mirror: ovs-tcpdump cannot capture vlan packets on the
> port with tag
>
> On 3/6/24 08:54, Zhangweiwei via discuss wrote:
>> Hi,
>>
>> I set tag 0 on port tapVm72and tapVm71, and then send ping packets
>> with vlan
>> 100 from tapVm72to tapVm71. But ovs-tcpdump cannot capture any packets
>> with vlan on tapVm72. It seems that vlan check is failed in
>> mirror_packet(), because
>> tapVm72 is an access port and the vlan packets are dropped. This is
>> not reasonable because OVS does not use NORMAL forward. When using
>> custom OpenFlow tables, mirror action should not consider tag configuration.
>
> I don't think that is related to tapVm72 being an access port.
> Could you, please, run ovs-appctl ofproto/trace on a packet arriving from
> tapVm72 ?
>
> Note that since mitapVm72 is not in vlan 0, mirrored traffic will have both
> vlan tags pushed to the packet. For this to work the vlan-limit
> configuration should be 2 or 0 (unlimited). Default value is 1 and that may
> be one reason why packets are not delivered to mitapVm72. ofproto/trace
> should be able to confirm if that is the case.
>
> Best regards, Ilya Maximets.
>
>>
>> 1、ovs version: 3.2.1
>> 2、Bridge
>> [root@localhost openvswitch-3.2.1]# ovs-vsctl show
>> Bridge vds1-br
>> Controller "tcp:172.20.66.228:6633"
>> is_connected: true
>> Controller "tcp:172.20.66.229:6633"
>> is_connected: true
>> fail_mode: secure
>> datapath_type: netdev
>> Port vxlan_vds1-br
>> Interface vxlan_vds1-br
>> type: vxlan
>> options: {key=flow, local_ip="3.3.3.70",
>> remote_ip=flow, tos=inherit}
>> Port tapVm72
>> tag: 0
>> Interface tapVm72
>> type: dpdkvhostuserclient
>> options
Re: [ovs-discuss] Mirror: ovs-tcpdump cannot capture vlan packets on the port with tag
On 3/6/24 08:54, Zhangweiwei via discuss wrote:
> Hi,
>
> I set tag 0 on port tapVm72and tapVm71, and then send ping packets with vlan
> 100 from tapVm72to tapVm71. But ovs-tcpdump cannot capture any packets with
> vlan on tapVm72. It seems that vlan check is failed in mirror_packet(),
> because
> tapVm72 is an access port and the vlan packets are dropped. This is not
> reasonable
> because OVS does not use NORMAL forward. When using custom OpenFlow tables,
> mirror
> action should not consider tag configuration.
I don't think that is related to tapVm72 being an access port.
Could you, please, run ovs-appctl ofproto/trace on a packet
arriving from tapVm72 ?
Note that since mitapVm72 is not in vlan 0, mirrored traffic will
have both vlan tags pushed to the packet. For this to work the
vlan-limit configuration should be 2 or 0 (unlimited). Default
value is 1 and that may be one reason why packets are not delivered
to mitapVm72. ofproto/trace should be able to confirm if that is
the case.
Best regards, Ilya Maximets.
>
> 1、ovs version: 3.2.1
> 2、Bridge
> [root@localhost openvswitch-3.2.1]# ovs-vsctl show
> Bridge vds1-br
> Controller "tcp:172.20.66.228:6633"
> is_connected: true
> Controller "tcp:172.20.66.229:6633"
> is_connected: true
> fail_mode: secure
> datapath_type: netdev
> Port vxlan_vds1-br
> Interface vxlan_vds1-br
> type: vxlan
> options: {key=flow, local_ip="3.3.3.70", remote_ip=flow,
> tos=inherit}
> Port tapVm72
> tag: 0
> Interface tapVm72
> type: dpdkvhostuserclient
> options: {vhost-server-path="/var/run/openvswitch/tapVm72"}
> Port mitapVm72
> Interface mitapVm72
> Port tapVm71
> tag: 0
> Interface tapVm71
> type: dpdkvhostuserclient
> options: {vhost-server-path="/var/run/openvswitch/tapVm71"}
> Port vds1-br
> Interface vds1-br
> type: internal
> ovs_version: "3.2.1"
>
> 3、dpcls:
>
> [[root@localhost openvswitch-3.2.1]# ovs-appctl dpctl/dump-flows -m | grep tap
>
> ufid:21fadb70-e3c1-4a2c-a0db-a042daa051c4,
> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(tapVm71),skb_mark(0/0),ct_state(0/0x30),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0,id=0),eth(src=52:54:00:67:d5:61,dst=52:54:00:9a:bf:ed),eth_type(0x8100),vlan(vid=100,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=2.2.70.1,dst=2.2.70.2/255.255.192.0,proto=1,tos=0/0,ttl=64/0,frag=no),icmp(type=0/0,code=0/0)),
> packets:4388, bytes:447576, used:0.420s, dp:ovs, actions:tapVm72,
> dp-extra-info:miniflow_bits(5,2)
>
> ufid:83a55534-3c62-4415-9aa7-bd8486675c68,
> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(tapVm72),skb_mark(0/0),ct_state(0/0x30),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0,id=0),eth(src=52:54:00:9a:bf:ed,dst=52:54:00:67:d5:61),eth_type(0x8100),vlan(vid=100,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=2.2.70.2,dst=2.2.70.1/255.255.192.0,proto=1,tos=0/0,ttl=64/0,frag=no),icmp(type=8/0,code=0/0)),
> packets:4388, bytes:447576, used:0.420s, dp:ovs, actions:tapVm71,
> dp-extra-info:miniflow_bits(5,2)
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 2/26/24 17:40, Mikhail Sennikovsky wrote: > Hi Ilya and everyone, > > I'm interested in obtaining further insight into the technical > rationale behind the automatic removal of ingress qdiscs for OVS > internal ports. > I.e. it's no doubt that ovs internal ports are managed by ovs, however > since internal ports are something different than external ports, I > would expect they could/should be managed differently. > > From the "outside the ovs" standpoint how would the the ovs internal > port network interface differ from a veth pair network interface whose > peer is enslaved in the ovs (via external port)? > > I.e. why should qdisc configuration for the ovs internal port network > interface have some additional limitations compared to that of the > veth interface whose peer is enslaved in ovs? OVS removes qdiscs from veth interfaces the same way it removes them from internal ports. There is no special treatment, AFAIK. That was the point of my reply. Ingress qdiscs will be removed from any port attached to OVS. Egress qdiscs will also be removed, unless linux-noop QoS type is configured. Best regards, Ilya Maximets. > > Kind regards, > Mikhail > > >> On 2/22/24 10:03, Reshma Sreekumar wrote: >>> Hello, >>> >>> I see similar behaviour for *ovs-internal* ports as well. My understanding >>> is that, >>> ovs-internal ports aren't /managed/ by Openvswitch (Please correct me if >>> I'm wrong) ? >> >> It would be strange if ovs-internal ports weren't managed by OVS. >> So, they are managed. >> >>> >>> In this case, is it expected for the *ingress qdisc on ovs-internal ports* >>> as well >>> to be managed by ovs ? >> >> It is expected for any ports attached to OVS. >> >> Basically, if you see an interface in the output of 'ovs-vsctl show', >> OVS will manage qdiscs on that interface. >> >> Bets regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd core at revalidator_sweep__
On 2/26/24 11:20, Eelco Chaudron wrote:
>
>
> On 26 Feb 2024, at 11:10, LIU Yulong wrote:
>
>> Hi Eelco,
>>
>> Thank you for the quick response.
>>
>> I did not add those logs, because in order to reproduce the issue, we
>> have to send lots of packets to the host.
>> So there are too many ukeys created/deleted to do logging.
>
> Maybe a circular buffer with all alloc/free (+ 1ukey address, and timestamp),
> 1 or 2 Mb of memory can hold a lot.
>
>> And can we ensure that this [1] is the only place for ovs to free the ukey?
>>
>> [1]
>> https://github.com/openvswitch/ovs/blob/v2.17.8/ofproto/ofproto-dpif-upcall.c#L2084
>
> Yes, this should be the only place, and should always be done through an RCU
> delayed delete.
>
>> For last mail, can this issue be concurrent read-and-update/delete?
>> The revalidator_sweep__ is trying to lock the ukey->mutex, while
>> another thread is updating the ukey->mutex to NULL and free ukey.
>
> This should not happen as the delete should happen by the delayed RCU delete,
> and if the ukey is still in the cmap after the delayed delete (quiescent
> state) something is wrong.
I agree with Eelco and I don't see any abvious issues with the current
implementation.
However, the ususal suspect for RCU problems is entering quiescent state
while iterating RCU-protected structure. Though I'm not sure how that can
happen in the revalidator, usually such issues are hiding somewhere way
down the call stack. I made a small patch that can help to be sure that
this doesn't actually happen in your setup:
https://github.com/igsilya/ovs/commit/8268347a159b5afa884f5b3008897878b5b520f5
Could you try it?
The change will log an error message and abort the process if we happen
to enter quiescent state while iterating over the hash map. Core dump
will point to a problematic call.
Best regards, Ilya Maximets.
>
>> LIU Yulong
>>
>>
>> On Mon, Feb 26, 2024 at 5:41 PM Eelco Chaudron wrote:
>>>
>>>
>>>
>>> On 26 Feb 2024, at 9:33, LIU Yulong wrote:
>>>
>>>> Hi,
>>>>
>>>> I have read the code by comparing the call stack of the core files
>>>> carefully, and found
>>>> a potential race condition. Please confirm whether the following 3 threads
>>>> have a race condition. Just did some code trace, can such
>>>> race condition happen?
>>>>
>>>> * PMD thread1 ===:
>>>> -> pmd_thread_main
>>>> -> dp_netdev_process_rxq_port
>>>> -> dp_netdev_input
>>>> -> dp_netdev_input__
>>>> -> handle_packet_upcall
>>>> -> dp_netdev_upcall
>>>> -> upcall_cb
>>>> -> ukey_install
>>>> -> ukey_install__
>>>> -> try_ukey_replace:
>>>> ovs_mutex_lock(_ukey->mutex);
>>>> <-- the CMAP_FOR_EACH loop in the revalidator_sweep__ run a
>>>> bit earlier than the cmap_replace next line, so the old_ukey can be
>>>> iterated. [1]
>>>> cmap_replace(>cmap, _ukey->cmap_node,
>>>> _ukey->cmap_node, new_ukey->hash);
>>>> ovsrcu_postpone(ukey_delete__, old_ukey);
>>>> < delete the ukey asynchronously. [2]
>>>> transition_ukey(old_ukey, UKEY_DELETED); <
>>>> transition the ukey state to UKEY_DELETED, most core files show that
>>>> the ukey last state change was at this line. [3]
>>>> transition_ukey(new_ukey, UKEY_VISIBLE);
>>>>
>>>> [1]
>>>> https://github.com/openvswitch/ovs/blob/v2.17.8/ofproto/ofproto-dpif-upcall.c#L1892
>>>> [2]
>>>> https://github.com/openvswitch/ovs/blob/v2.17.8/ofproto/ofproto-dpif-upcall.c#L1896
>>>> [3]
>>>> https://github.com/openvswitch/ovs/blob/v2.17.8/ofproto/ofproto-dpif-upcall.c#L1897
>>>>
>>>> This function try_ukey_replace was called many times, because the
>>>> `counter_upcall_ukey_replace` is not zero.
>>>> For instance:
>>>> {
>>>> name = 0x55ba9755206b "upcall_ukey_replace",
>>>> count = 0x55ba971c7610 ,
>>>> total = 2287997,
>>>> last_total = 2287997,
>>>> min = {221, 247, 444, 278, 324, 570, 379, 464, 283, 280, 0, 427},
>>>> hr = {3300, 4378, 3557, 4554, 3748, 3710, 4340, 3559, 4296, 3759,
>>>> 3522, 4136, 3660, 4428, 3802, 3652, 3880, 3375, 4806, 4221, 415
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 2/22/24 10:03, Reshma Sreekumar wrote: > Hello, > > I see similar behaviour for *ovs-internal* ports as well. My understanding is > that, > ovs-internal ports aren't /managed/ by Openvswitch (Please correct me if I'm > wrong) ? It would be strange if ovs-internal ports weren't managed by OVS. So, they are managed. > > In this case, is it expected for the *ingress qdisc on ovs-internal ports* as > well > to be managed by ovs ? It is expected for any ports attached to OVS. Basically, if you see an interface in the output of 'ovs-vsctl show', OVS will manage qdiscs on that interface. Bets regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 2/19/24 17:19, Ilya Maximets wrote:
> On 2/7/24 03:21, Lim, Derrick wrote:
>> Hi Ilya Maximets,
>>
>> From the tcpdump, with or without the rewrite, the link-local address was
>> used.
>>
>> ===
>> $ ovs-tcpdump -nn -i exit_p0
>> 11:10:26.323938 IP6 fe80::dc03:37ff:fee2:1fef.51513 >
>> 2403:400:31da:::18:3.4789: VXLAN, flags [I] (0x08), vni 1
>> IP 100.87.18.60 > 192.168.1.33: ICMP echo request, id 70, seq 1, length 64
>> 11:10:27.326875 IP6 fe80::dc03:37ff:fee2:1fef.51513 >
>> 2403:400:31da:::18:3.4789: VXLAN, flags [I] (0x08), vni 1
>> IP 100.87.18.60 > 192.168.1.33: ICMP echo request, id 70, seq 2, length 64
>> ===
>>
>> Here is the output of the trace without the rewrite.
>> ===
>> $ ovs-appctl ofproto/trace --names br-int 'in_port=dpdk-vm101,
>> eth_src=52:54:00:3d:cd:0c,eth_dst=00:00:00:00:00:01,eth_type=0x0800,
>> nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_proto=1,nw_ttl=64,nw_frag=no,
>> icmp_type=8,icmp_code=0'
>> Flow:
>> icmp,in_port="dpdk-vm101",vlan_tci=0x,dl_src=52:54:00:3d:cd:0c,dl_dst=00:00:00:00:00:01,nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
>>
>> bridge("br-int")
>> --
>> 0. in_port="dpdk-vm101", priority 32768
>> output:vxlan0
>> -> output to native tunnel
>> -> tunneling to fe80::920a:84ff:fe9e:9570 via br-phy
>> -> tunneling from de:03:37:e2:1f:ef fe80::dc03:37ff:fee2:1fef to
>> 90:0a:84:9e:95:70 fe80::920a:84ff:fe9e:9570
>>
>> bridge("br-phy")
>> ---
>> 0. priority 10
>> NORMAL
>> -> forwarding to learned port
>>
>> Final flow: unchanged
>> Megaflow: recirc_id=0,eth,ip,in_port="dpdk-vm101",nw_ecn=0,nw_frag=no
>> Datapath actions:
>> tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4,eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd),ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)),push_vlan(vid=304,pcp=0),exit_p0
>> ===
>>
>> The "tunneling to fe80::920a:84ff:fe9e:9570 via br-phy" looks a bit curious.
>> I'm not sure why this was picked instead of the `remote_ip` specified in the
>> tunnel configuration. But then the final datapath actions shows the correct
>> `dst` address.
>
> Hi. Sorry for the late reply, was caught up in the releases.
>
> The 'tunneling to' message may be a little misleading, it prints out the
> result of a route lookup, and we only use the device name from it while
> building a tunnel header. The correct remote ip will be taken from a tunnel
> configuration, not the IP from a route lookup. Maybe the wording in the
> trace needs some adjustment.
On a second look, it does seem a little strange. The likley cause of
having fe80::920a:84ff:fe9e:9570 instead of the configured remote_ip
is that OVS found a route to 2403:400:31da:::18:3 via a gateway
fe80::920a:84ff:fe9e:9570. But in one of the previous route lookups
you provided the fe80::920a:84ff:fe9e:9570 was indeed a gatewey IP,
so it checks out. The correct remote_ip is used in the actions because
though we're sending the packet via gateway, we're not send it to the
gateway. The gateway IP is only needed to get the destination MAC.
'tunneling to fe80::920a:84ff:fe9e:9570 via br-phy' should probbaly be
'tunneling via fe80::920a:84ff:fe9e:9570 and br-phy' in this case.
>
>> Why is the `local_ip` specified in the VXLAN tunnel options
>> not considered?
> I see there is a bug in the tunnel lookup code that doesn't take into
> account IPv6 local ip. It only checks for IPv4 one. The following
> change should fix it:
>
> diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
> index 1cf4d5f7c..89f183182 100644
> --- a/ofproto/ofproto-dpif-xlate.c
> +++ b/ofproto/ofproto-dpif-xlate.c
> @@ -3815,6 +3815,8 @@ native_tunnel_output(struct xlate_ctx *ctx, const
> struct xport *xport,
>
> if (flow->tunnel.ip_src) {
> in6_addr_set_mapped_ipv4(_ip6, flow->tunnel.ip_src);
> +} else if (ipv6_addr_is_set(>tunnel.ipv6_src)) {
> +s_ip6 = flow->tunnel.ipv6_src;
> }
>
> err = tnl_route_lookup_flow(ctx, flow, _ip6, _ip6, _dev);
> ---
>
> Could you try it in your setup?
>
> Without this change the route lookup is performed without taking the
> local_ip into account and later the local_ip is not used for the packet
> header.
>
> I'll work on a proper patch for this
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 2/7/24 03:21, Lim, Derrick wrote:
> Hi Ilya Maximets,
>
> From the tcpdump, with or without the rewrite, the link-local address was
> used.
>
> ===
> $ ovs-tcpdump -nn -i exit_p0
> 11:10:26.323938 IP6 fe80::dc03:37ff:fee2:1fef.51513 >
> 2403:400:31da:::18:3.4789: VXLAN, flags [I] (0x08), vni 1
> IP 100.87.18.60 > 192.168.1.33: ICMP echo request, id 70, seq 1, length 64
> 11:10:27.326875 IP6 fe80::dc03:37ff:fee2:1fef.51513 >
> 2403:400:31da:::18:3.4789: VXLAN, flags [I] (0x08), vni 1
> IP 100.87.18.60 > 192.168.1.33: ICMP echo request, id 70, seq 2, length 64
> ===
>
> Here is the output of the trace without the rewrite.
> ===
> $ ovs-appctl ofproto/trace --names br-int 'in_port=dpdk-vm101,
> eth_src=52:54:00:3d:cd:0c,eth_dst=00:00:00:00:00:01,eth_type=0x0800,
> nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_proto=1,nw_ttl=64,nw_frag=no,
> icmp_type=8,icmp_code=0'
> Flow:
> icmp,in_port="dpdk-vm101",vlan_tci=0x,dl_src=52:54:00:3d:cd:0c,dl_dst=00:00:00:00:00:01,nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
>
> bridge("br-int")
> --
> 0. in_port="dpdk-vm101", priority 32768
> output:vxlan0
> -> output to native tunnel
> -> tunneling to fe80::920a:84ff:fe9e:9570 via br-phy
> -> tunneling from de:03:37:e2:1f:ef fe80::dc03:37ff:fee2:1fef to
> 90:0a:84:9e:95:70 fe80::920a:84ff:fe9e:9570
>
> bridge("br-phy")
> ---
> 0. priority 10
> NORMAL
> -> forwarding to learned port
>
> Final flow: unchanged
> Megaflow: recirc_id=0,eth,ip,in_port="dpdk-vm101",nw_ecn=0,nw_frag=no
> Datapath actions:
> tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4,eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd),ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)),push_vlan(vid=304,pcp=0),exit_p0
> ===
>
> The "tunneling to fe80::920a:84ff:fe9e:9570 via br-phy" looks a bit curious.
> I'm not sure why this was picked instead of the `remote_ip` specified in the
> tunnel configuration. But then the final datapath actions shows the correct
> `dst` address.
Hi. Sorry for the late reply, was caught up in the releases.
The 'tunneling to' message may be a little misleading, it prints out the
result of a route lookup, and we only use the device name from it while
building a tunnel header. The correct remote ip will be taken from a tunnel
configuration, not the IP from a route lookup. Maybe the wording in the
trace needs some adjustment.
> Why is the `local_ip` specified in the VXLAN tunnel options
> not considered?
I see there is a bug in the tunnel lookup code that doesn't take into
account IPv6 local ip. It only checks for IPv4 one. The following
change should fix it:
diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
index 1cf4d5f7c..89f183182 100644
--- a/ofproto/ofproto-dpif-xlate.c
+++ b/ofproto/ofproto-dpif-xlate.c
@@ -3815,6 +3815,8 @@ native_tunnel_output(struct xlate_ctx *ctx, const struct
xport *xport,
if (flow->tunnel.ip_src) {
in6_addr_set_mapped_ipv4(_ip6, flow->tunnel.ip_src);
+} else if (ipv6_addr_is_set(>tunnel.ipv6_src)) {
+s_ip6 = flow->tunnel.ipv6_src;
}
err = tnl_route_lookup_flow(ctx, flow, _ip6, _ip6, _dev);
---
Could you try it in your setup?
Without this change the route lookup is performed without taking the
local_ip into account and later the local_ip is not used for the packet
header.
I'll work on a proper patch for this.
Best regards, Ilya Maximets.
>
> Here is the out of the trace with the rewrite. It seems the flow entry was
> matched but the rewrite didn't happen.
>
> ===
> $ ovs-appctl ofproto/trace --names br-int 'in_port=dpdk-vm101,
> eth_src=52:54:00:3d:cd:0c,eth_dst=00:00:00:00:00:01,eth_type=0x0800,
> nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_proto=1,nw_ttl=64,nw_frag=no,
> icmp_type=8,icmp_code=0'
> Flow:
> icmp,in_port="dpdk-vm101",vlan_tci=0x,dl_src=52:54:00:3d:cd:0c,dl_dst=00:00:00:00:00:01,nw_src=100.87.18.60,nw_dst=192.168.1.33,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
>
> bridge("br-int")
> --
> 0. in_port="dpdk-vm101", priority 32768
> output:vxlan0
> -> output to native tunnel
> -> tunneling to fe80::920a:84ff:fe9e:9570 via br-phy
> -> tunneling from de:03:37:e2:1f:ef fe80::dc03:37ff:fee2:1fef to
> 90:0a:84:9e:95:70 fe80::920a:84ff:fe9e:9570
>
> bridge("br-phy")
> ---
> 0. ipv6,ipv6_dst=2403:40
Re: [ovs-discuss] ovs-vswitchd core at revalidator_sweep__
On 2/19/24 11:14, Eelco Chaudron wrote: > > > On 19 Feb 2024, at 10:34, LIU Yulong wrote: > >> Hi OVS experts, >> >> Our ovs-vswitchd runs to core at the ovs_mutex_trylock(>mutex) in the >> function revalidator_sweep__. >> >> I've sent the mail before but have no response. >> https://mail.openvswitch.org/pipermail/ovs-discuss/2023-August/052604.html >> >> So I'm trying to send this mail again. And I may apologize in advance because >> I would like to post as much useful information as possible to help identify >> potential issues. So this mail will have a really long text. >> >> Compared to the mail 2023-August/052604.html, we upgrade the OVS to 2.17.8 >> and DPDK to 22.11 to pray for good luck that maybe the community has >> potential >> fixes for this issue. But unfortunately, the ovs-vswitchd still runs to core. > > As you mentioned it looks like some memory corruption, which I have not seen > before. > > Have you tried this without rte offload? This is the only feature I never > used. > There is a 2.17.9 with DPDK 22.11.6 you could try. OVS 2.17 is not supposed to work with DPDK 22.11, it's supposed to work with 21.11. See the compatibility table here: https://docs.openvswitch.org/en/latest/faq/releases/ Though it's hard to tell if DPDK version is anyhow related to the issue. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] [ADVISORY] CVE-2023-5366: Open vSwitch: OpenFlow match on Neighbor Discovery Target may be ignored
Description === In multiple versions of Open vSwitch, if OpenFlow rules on a switch contain a match on a Target Address (nd_target) of Neighbor Discovery IPv6 packets (Neighbor Solicitation or Neighbor Advertisement) without also matching on ICMPv6 Code (icmp_code or icmpv6_code) field being zero, the match on the Target Address can be ignored and the specified actions may be executed for a packet with a different Target Address. This constitutes vulnerability if such OpenFlow rules are used in order to provide Neighbor Discovery anti-spoofing protection. For example, the following set of rules may allow packets with any nd_target, even though it should only allow packets with the 2001::1 Target: priority=10 icmp6,icmpv6_type=136,nd_target=2001::1 actions= priority=0 icmp6 actions=drop The issue is caused by the difference between the OpenFlow specification that only lists ICMPV6 TYPE=135 or ICMPV6 TYPE=136 as a prerequisite for the IPV6_ND_TARGET and datapath implementations that treat ICMPV6_CODE=0 as a requirement for a packet to have the Target Address option. This leads to creation of an overly broad datapath flow that matches packets regardless of the Target Address value. Triggering the issue depends on the order in which packets are seen by the switch. Open vSwitch versions 2.1 and newer are affected. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned CVE-2023-5366 identifier to this issue. Mitigation == For any affected version of Open vSwitch, adding an icmpv6_code=0 match to an OpenFlow rule will mitigate the issue. For example: priority=10 icmp6,icmpv6_type=136,icmpv6_code=0,nd_target=2001::1 \ actions= priority=0 icmp6 actions=drop 'icmpv6_code' corresponds to OXM_OF_ICMPV6_CODE staring with OF1.2 or NXM_OF_ICMPV6_CODE Nicira Extension before that. Fix === Patches to fix this vulnerability in Open vSwitch 2.17 and newer has been applied to the appropriate branches: * 3.2.x: https://github.com/openvswitch/ovs/commit/7570744c5add3a91b468c4ffa5bc73ef1f5bb18a * 3.1.x: https://github.com/openvswitch/ovs/commit/d3f9eab1abbb6a11c2a166472c184f54fd740bf1 * 3.0.x: https://github.com/openvswitch/ovs/commit/a6c0a3deb268f34faef0062e2c05ece563d50ecb * 2.17.x: https://github.com/openvswitch/ovs/commit/e235a421fbdb0c70176e8a3bef13bf7e2056cbc1 The original patch is located at: https://mail.openvswitch.org/pipermail/ovs-dev/2024-February/411701.html Note for those performing selective patch backports: The following change is a prerequisite for the fix and must also be applied for the fix to work correctly: https://github.com/openvswitch/ovs/commit/489553b1c21692063931a9f50b6849b23128443c Recommendation == We recommend that users of Open vSwitch apply the respective patches, or upgrade to a known patched version of Open vSwitch. These include: * 3.2.2 * 3.1.4 * 3.0.6 * 2.17.9 However, while patches are enough to fix the issue, checking both the ICMPv6 Code and Type is *the correct way* to validate Neighbor Discovery packets according to RFC 4861. With this in mind, we recommend that users follow the mitigation strategy even with the patches applied, i.e. add an extra match criteria on 'icmpv6_code' field. Acknowledgments === The Open vSwitch team wishes to thank the reporters: * Alex Katz * Slawomir Kaplonski OpenPGP_0xB9F7EC77C829BF96.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] [ADVISORY] CVE-2023-3966: Open vSwitch: Invalid memory access in Geneve with HW offload.
Description === Multiple versions of Open vSwitch are vulnerable to crafted Geneve packets causing invalid memory accesses and potential denial of service. Triggering the vulnerability requires that Open vSwitch has flow hardware offload with Linux TC flower enabled (other_config:hw-offload=true). It is not enabled by default. The issue is caused by insufficient validation of Geneve metadata fields in the offload path. Open vSwitch versions 2.12 and newer are affected. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned CVE-2023-3966 identifier to this issue. Mitigation == For any version of Open vSwitch, disabling flow hardware offload will prevent the issue (setting other_config:hw-offload=false and reboot the system). We do not recommend attempting to mitigate the vulnerability this way because it may impact the overall system throughput. By default, flow hardware offload support is not enabled. Fix === Patches to fix this vulnerability in Open vSwitch 2.17 and newer: * 3.2.x: https://github.com/openvswitch/ovs/commit/2cfbcd5247ed0fd941c1ebb9f4adb952b67fe13a * 3.1.x: https://github.com/openvswitch/ovs/commit/91e621bd5abab19954bec09c7d27c59acdf607b1 * 3.0.x: https://github.com/openvswitch/ovs/commit/935cd1d574c6f432a451df8941374ffb36d767d9 * 2.17.x: https://github.com/openvswitch/ovs/commit/b8657dada9641fbd2bd3a3f882e0862448d60910 Original patch is located at: https://mail.openvswitch.org/pipermail/ovs-dev/2024-February/411702.html Recommendation == We recommend that users of Open vSwitch apply the linked patches, or upgrade to a known patched version of Open vSwitch. These include: * v3.2.2 * v3.1.4 * v3.0.6 * v2.17.9 Acknowledgments === The Open vSwitch team wishes to thank the reporters: Timothy Redaelli Haresh Khandelwal OpenPGP_0xB9F7EC77C829BF96.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] RSTP cannot be configured on both bridges that are connected by patch ports.
On 2/1/24 04:33, Huangzhidong via discuss wrote: > Hi > > I have two bridges, vds1-br and vds1-br-ex, connected by patch ports > with OVS 3.2. When I configure RSTP on both bridges, it causes the > OVS process to crash and restart repeatedly. > > [root@localhost ~]# ovs-vsctl set Bridge vds1-br-ex rstp_enable=true > [root@localhost ~]# ovs-vsctl set Bridge vds1-br rstp_enable=true ->at this > time, ovs crash and ovsdb poll > > 2024-02-01T02:35:00Z|2|fatal_signal|WARN|terminating with signal 2 > (Interrupt) > > [root@localhost ~]# ps aux | grep ovs > root 4173731 0.0 0.0 13712 2572 ? S ovsdb-server: monitoring pid 4173732 (healthy) > root 4173732 0.0 0.0 14844 8080 ? S< 10:07 0:00 > ovsdb-server /usr/local/etc/openvswitch/conf.db -vconsole:emer -vsyslog:err > -vfile:info --remote=punix:/usr/local/var/run/openvswitch/db.sock > --private-key=db:Open_vSwitch,SSL,private_key > --certificate=db:Open_vSwitch,SSL,certificate > --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir > --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log > --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid --detach --monitor > root 4173747 0.0 0.0 50476 10240 ? S ovs-vswitchd: monitoring pid 4176348 (7 crashes: pid 4176261 died, killed > (Aborted), core dumped) > root 4176348 103 0.0 286086580 184520 ? D ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer > -vsyslog:err -vfile:info --mlockall --no-chdir > --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log > --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid --detach –monitor > > From core , I can get: > #0 rstp_port_received_bpdu (rp=0x562c718d0ed0, bpdu=0x562c718dbaa1, > bpdu_size=36) at lib/rstp.c:238 > #1 0x562c703aa415 in rstp_process_packet (packet=, > xport=) at ./lib/dp-packet.h:619 > #2 0x562c703aad22 in process_special (ctx=ctx@entry=0x7fff448169c0, > xport=xport@entry=0x562c719192f0) at ofproto/ofproto-dpif-xlate.c:3443 > #3 0x562c703ad4b3 in patch_port_output (ctx=ctx@entry=0x7fff448169c0, > in_dev=in_dev@entry=0x562c7191c190, out_dev=0x562c719192f0, > is_last_action=is_last_action@entry=true) at > ofproto/ofproto-dpif-xlate.c:3951 > #4 0x562c703af17a in compose_output_action__ > (ctx=ctx@entry=0x7fff448169c0, ofp_port=1, xr=xr@entry=0x0, > check_stp=check_stp@entry=true, > is_last_action=, truncate=truncate@entry=false) at > ofproto/ofproto-dpif-xlate.c:4274 > #5 0x562c703b0f61 in compose_output_action (truncate=false, > is_last_action=, xr=0x0, ofp_port=, > ctx=0x7fff448169c0) > at ofproto/ofproto-dpif-xlate.c:5374 > #17 0x562c704d27ae in rstp_port_set_mac_operational (p=0x562c718df850, > new_mac_operational=) at lib/rstp.c:1042 > #18 0x562c703778d7 in ofproto_port_set_rstp (ofproto=0x562c718974c0, > ofp_port=1, s=s@entry=0x7fff44818450) at ofproto/ofproto.c:1271 > > It caused by rstp_mutex set to PTHREAD_MUTEX_INITIALIZER, it does not allow > RECURSIVE. > > From stp_mutex, it says: > > /* We need a recursive mutex because stp_send_bpdu() could loop back > * into the stp module through a patch port. This happens > * intentionally as part of the unit tests. Ideally we'd ditch > * the call back function, but for now this is what we have. */ > ovs_mutex_init_recursive(); > > > And the early version of RSTP also used recursive mutex locks. > > With commit * lib/rstp: Remove lock recursion.( > SHA-1:6b90bc57e7a23b89a594ceb857f8267c8b4026df , > Change the RSTP send_bpdu interface so that a recursive mutex is not > needed.), RECURSIVE lock > has been canceled. > > May I still need a RECURSIVE lock for rstp? Thanks for the report! Indeed, the mutex has to be recursive in the current version of the code for the same reason STP mutex is recursive. I can prepare a patch to fix this. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 2/2/24 08:58, Lim, Derrick via discuss wrote: > Hi Ilya Maximets, > >> The rules look mostly fine. I think the main problem you have is priority. >> Default priority for OF rules (if not specified) is 32768, so your new rules >> with priority 50 are too low in a priority list and not getting hit. > > I tried this again with the default flow at priority 50 and mine at 499 but I > still couldn't get the flow to hit. > > However, I observed that if the source address is set to anything other than > `2403:400:31da:::18:6`, which is an address that exists on the phy > `br-phy` > interface, the lookup is a hit and the action taken. > > Is there anything that prevents the address from being set to that of > something > that is already configured on an interface? > > For example, > > $ ip addr > 35: br-phy: mtu 1500 qdisc fq_codel > state UNKNOWN group default qlen 1000 > link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff > inet6 2403:400:31da:::18:6/128 scope global > valid_lft forever preferred_lft forever > inet6 fe80::dc03:37ff:fee2:1fef/64 scope link > valid_lft forever preferred_lft forever > > Set the source address to `2403:400:31da:::18:5`. In the flow entry, > `set(ipv6(src=2403:400:31da:::18:5))` is applied in actions. > > === > > $ /usr/bin/ovs-ofctl add-flow br-phy \ > > "priority=499,ipv6,ipv6_dst=2403:400:31da:::18:3,actions=set_field:2403:400:31da:::18:5->ipv6_src,normal" > > $ /usr/bin/ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev | grep > 192.168.1.33 > > ufid:acc4b3bc-4958-412d-90c2-9bc4b3fbfac7, > > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0), > recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0), > > eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=00:00:00:00:00:01/00:00:00:00:00:00),eth_type(0x0800), > > ipv4(src=100.87.18.60/0.0.0.0,dst=192.168.1.33/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no), > icmp(type=8/0,code=0/0), packets:407, bytes:39886, used:0.661s, dp:ovs, > actions: > tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4, > eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd), > > ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64), > > udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)), > set(ipv6(src=2403:400:31da:::18:5)), > push_vlan(vid=304,pcp=0), > exit_p0, > dp-extra-info:miniflow_bits(4,1) > > === > > > Set the source address to `2403:400:31da:::18:6`. This is a configured > address on `br-phy`. The `set(ipv6(src=))` part is no longer applied. > > === > > $ /usr/bin/ovs-ofctl add-flow br-phy \ > > "priority=499,ipv6,ipv6_dst=2403:400:31da:::18:3,actions=set_field:2403:400:31da:::18:6->ipv6_src,normal" > > $ /usr/bin/ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev | grep > 192.168.1.33 > > ufid:acc4b3bc-4958-412d-90c2-9bc4b3fbfac7, > > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0), > recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0), > > eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=00:00:00:00:00:01/00:00:00:00:00:00),eth_type(0x0800), > > ipv4(src=100.87.18.60/0.0.0.0,dst=192.168.1.33/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no), > icmp(type=8/0,code=0/0), packets:423, bytes:41454, used:0.803s, dp:ovs, > actions: > tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4, > eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd), > > ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64), > > udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)), > push_vlan(vid=304,pcp=0), > exit_p0, > dp-extra-info:miniflow_bits(4,1) > > $ /usr/bin/ovs-ofctl dump-flows br-phy > > cookie=0x0, duration=170.787s, table=0, n_packets=251, n_bytes=40328, > priority=499,ipv6,ipv6_dst=2403:400:31da:::18:3 > > actions=load:0x180006->NXM_NX_IPV6_SRC[0..63],load:0x2403040031da->NXM_NX_IPV6_SRC[64..127],NORMAL > > cookie=0x0, duration=1136.132s, table=0, n_packets=10175, n_bytes=1116852, > priority=10 actions=NORMAL > > === Hmm. This is interesting. We can see that some packets do actually hit the OpenFlow rule (n_packets=251). The decision to not include the set(ipv6(src=)) action is likely made because it is the same as one already in the packet.
Re: [ovs-discuss] bond: bond/show next balance time is a negative value.
On 2/1/24 04:42, Huangzhidong via discuss wrote:
> Hi
>
> When I use bond/show to get next balance time, it sometimes get a negative
> value.
>
> It can easily reproduce by run shell scripts:
>
> while true; do
> ovs-appctl bond/show | grep next
> done
>
> and it can be easily fixed by:
>
> ofproto/bond.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> diff --git a/ofproto/bond.c b/ofproto/bond.c
> index cfdf44f85..384ffdb08 100644
> --- a/ofproto/bond.c
> +++ b/ofproto/bond.c
> @@ -1539,7 +1539,7 @@ bond_print_details(struct ds *ds, const struct bond
> *bond)
> if (bond_is_balanced(bond)) {
> ds_put_format(ds, "next rebalance: %lld ms\n",
> - bond->next_rebalance - time_msec());
> + (bond->next_rebalance + bond->rebalance_interval -
> time_msec()) % bond->rebalance_interval);
> }
> ds_put_format(ds, "lacp_status: %s\n",
Hi,
My understanding is that we print out negative value because rebalancing
is already overdue, i.e. we should have been rebalanced X ms ago. And
that indicates that rebalancing will be performed as soon as possible.
Your suggested change will make the value positive, but it will no longer
be correct in this case. We could print out a zero, I guess, instead of
a negative value, but, I think, the negative value is somewhat useful,
because we can tell how far behind OVS is on the rebalancing.
Does that make sense?
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] bond: bond entry statistics overflow
On 1/31/24 10:29, Zhangweiwei via discuss wrote:
> Hi,
>
> I encountered an issue while using OVS bond in balance-tcp mode.
> After performing down and up operations on bond members, the bond
> entry statistics displayed by ovs-appctl bond/show occured overflow.
> In addition to the statistics values issue, this also led to longer
> load balancing time for bond members.
>
> 1、information：
> ovs version:2.17.2
> bond mode: balance-tcp
> openflow: cookie=0x0, duration=7027.270s, table=0, n_packets=15169077,
> n_bytes=9334457220, priority=0 actions=NORMAL
> datapath_type: netdev
>
> 2、ovs-appctl bond/show print:
>
> [root@localhost zzz]# ovs-appctl bond/show
> eobond
> bond_mode: balance-tcp
> bond may use recirculation: yes, Recirc-ID : 1
> bond-hash-basis: 0
> lb_output action: disabled, bond-id: -1
> updelay: 0 ms
> downdelay: 0 ms
> next rebalance: 9673 ms
> lacp_status: negotiated
> lacp_fallback_ab: false
> active-backup primary:
> active member mac: 98:a9:2d:c5:00:69(u0)
>
> member u0: enabled
> active member
> may_enable: true
> hash 89: 9007199254740413 kB load
> hash 219: 9007199254740991 kB load
>
> member u1: enabled
> may_enable: true
> hash 141: 9007199254520657 kB load
>
> 3、analysis：
>
> After performing down and up operations on bond members, recirc rules are
> changed，bond_entry_account( ) function updates bond entry statistics through
> recirc rules. rule_tx_bytes < entry->pr_tx_bytes , so delta occurs overflow.
So, the main issue here seems to be that the statistics on the rule
itself jumps back for some reason. There were a few patches in the
past year or so that fix several occurrences of similar statistics
issues during flow dumps. Can you reproduce the issue on the latest
v2.17.8 release? 2.17.2 is fairly old and coent contain many important
bug fixes. Newer releases like 3.1.2+ also have enhanced logging
around statistics mishaps, so they are easier to debug.
Best regards, Ilya Maximets.
>
> static void bond_entry_account (struct bond_entry *entry, uint64_t
> rule_tx_bytes)
> OVS_REQ_WRLOCK(rwlock) {
> if (entry->member) {
> uint64_t delta;
> delta = rule_tx_bytes - entry->pr_tx_bytes; // delta occurs
> overflow
> entry->tx_bytes += delta;
> entry->pr_tx_bytes = rule_tx_bytes;
> }
> }
>
> 4、solution
>
> I try to add last_pr_rule in struct bond_entry to solve this problem. When
> then recirc rule changes, delta = rule_tx_bytes, and entry->tx_bytes +=
> rule_tx_bytes.
> But I’m not sure whether the value of entry->tx_bytes is correct after the
> modification.
>
> index ddc96a4..7b14d53 100644
> --- a/openvswitch-2.17.2/ofproto/bond.c
> +++ b/openvswitch-test/ofproto/bond.c
> @@ -71,6 +71,7 @@ struct bond_entry {
> * 'pr_tx_bytes' is the most recently seen statistics for 'pr_rule',
> which
> * is used to determine delta (applied to 'tx_bytes' above.) */
> struct rule *pr_rule;
> + struct rule *last_pr_rule;
> uint64_t pr_tx_bytes OVS_GUARDED_BY(rwlock);
> };
>
> @@ -990,8 +991,12 @@ bond_entry_account(struct bond_entry *entry, uint64_t
> rule_tx_bytes)
> if (entry->member) {
> uint64_t delta;
>
> - delta = rule_tx_bytes - entry->pr_tx_bytes;
> - entry->tx_bytes += delta;
> + if (entry->last_pr_rule != entry->pr_rule) {
> + entry->tx_bytes += rule_tx_bytes;
> + } else {
> + delta = rule_tx_bytes - entry->pr_tx_bytes;
> + entry->tx_bytes += delta;
> + }
> entry->pr_tx_bytes = rule_tx_bytes;
> }
> }
> @@ -1027,6 +1032,7 @@ bond_recirculation_account(struct bond *bond)
> continue;
> }
> bond_entry_account(entry, stats.n_bytes);
> + entry->last_pr_rule=rule;
> }
> }
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS 3.2.0 crashing setting port QOS
On 1/29/24 20:13, Daryl Wang via discuss wrote:
> After upgrading from Open vSwitch 3.1.2-4 to 3.2.0-2 we've been consistently
> seeing new OVS crashes when setting QoS on ports. Both packages were taken
> from the Debian distribution
> (https://packages.debian.org/source/sid/openvswitch
> we're running on. From the core dump we're seeing the following backtrace:
>
> # gdb --batch -ex bt /usr/sbin/ovs-vswitchd /core
> [NewLWP 67669]
> [NewLWP 67682]
> [NewLWP 67681]
> [NewLWP 67671]
> [NewLWP 67679]
> [NewLWP 67680]
> [Threaddebugging usinglibthread_db enabled]
> Usinghost libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Corewas generated by`ovs-vswitchd unix:/tmp/ovs/db.sock -vconsole:emer
> -vsyslog:err -vfile:info --ml'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> [Current thread is 1 (Thread 0x7fcacf4cfa80 (LWP 67669))]
> #0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x7fcacec6d472 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x7fcacec574b2 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x560787952c7e in ovs_abort_valist (err_no=,
> format=, args=args@entry=0x7ffd14c3bce0) at ../lib/util.c:447
> #4 0x560787952d14 in ovs_abort (err_no=,
> format=format@entry=0x5607879f3ec7 "%s: pthread_%s_%s failed") at
> ../lib/util.c:439
> #5 0x56078791ee11 in ovs_mutex_lock_at (l_=l_@entry=0x56078934c6c8,
> where=where@entry=0x5607879f864b "../lib/netdev-linux.c:2575") at
> ../lib/ovs-thread.c:76
> #6 0x56078796d03d in netdev_linux_get_speed (netdev_=0x56078934c640,
> current=0x7ffd14c3be64, max=0x7ffd14c3be1c) at ../lib/netdev-linux.c:2575
> #7 0x5607878c04f3 in netdev_get_speed
> (netdev=netdev@entry=0x56078934c640, current=current@entry=0x7ffd14c3be64,
> max=0x7ffd14c3be1c, max@entry=0x0) at ../lib/netdev.c:1175
> #8 0x560787968d67 in htb_parse_qdisc_details__
> (netdev=netdev@entry=0x56078934c640, details=details@entry=0x56078934a880,
> hc=hc@entry=0x7ffd14c3beb0) at ../lib/netdev-linux.c:4804
> #9 0x5607879755da in htb_tc_install (details=0x56078934a880,
> netdev=0x56078934c640) at ../lib/netdev-linux.c:4883
> #10 htb_tc_install (netdev=0x56078934c640, details=0x56078934a880) at
> ../lib/netdev-linux.c:4875
> #11 0x560787974937 in netdev_linux_set_qos (netdev_=0x56078934c640,
> type=, details=0x56078934a880) at ../lib/netdev-linux.c:3054
> #12 0x560787814ea5 in iface_configure_qos (qos=0x56078934a780,
> iface=0x560789349fd0) at ../vswitchd/bridge.c:4845
> #13 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x5607892dbf90) at
> ../vswitchd/bridge.c:928
> #14 0x560787817f7d in bridge_run () at ../vswitchd/bridge.c:3321
> #15 0x56078780d205 in main (argc=, argv=)
> at ../vswitchd/ovs-vswitchd.c:130
>
> A shell script to reproduce the issue is:
>
> #!/bin/sh
>
> apt-get install openvswitch-{common,switch}{,-dbgsym}
>
> # Don't need it running on the system
> systemctl stop openvswitch-switch
>
> set-e
>
> cleanup(){
> ip link delveth0
> rm /tmp/ovs/conf.db
> }
>
> trap cleanup EXIT
>
> # Setup our environment
>
> ip link add veth0 type veth peer veth1
>
> mkdir -p /tmp/ovs
>
> exportOVS_RUNDIR=/tmp/ovs
> exportOVS_LOGDIR=/tmp/ovs
> exportOVS_DBDIR=/tmp/ovs
>
> /usr/share/openvswitch/scripts/ovs-ctl start
> ovs-vsctl add-br demo
> ovs-vsctl add-port demo veth1
>
> # Make it crash
>
> ovs-vsctl setPortveth1 qos=@qos\
> -- --id=@qoscreate QoStype=linux-htb queues:1=@highqueues:2=@low\
> -- --id=@highcreate Queueother-config:priority=1other-config:min-rate=0.1\
> -- --id=@low create Queueother-config:priority=6other-config:min-rate=0.05
>
> We built the reproduction script based on speculation that
> https://github.com/openvswitch/ovs/commit/6240c0b4c80ea3d8dd1bf77526b04b55742de2ce
> is related to the crash. Notably we don't seem to run into the problem when we
> pass in a specific maximum bandwidth instead of relying on the interface's
> maximum
> bandwidth.
Hi, Daryl. Thanks for the report!
Looking at the stack trace, the root cause seems to be the following commit:
https://github.com/openvswitch/ovs/commit/b8f8fad8643518551cf742056ae8728c936674c6
It introduced the netdev_get_speed() call in QoS functions.
These functions are running under netdev mutex and the netdev_get_speed()
function is trying to take the same mutex for a second time. That fails
with 'deadlock avoided' or something like that.
From the commit I linked it's clear why it's not happening if the max-rate
is specified. The code just doesn't go that route.
To fi
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 1/29/24 10:23, Lim, Derrick wrote:
> Hi Ilya Maximets,
>
> I did some further testing on my end. Just to make sure it's an address
> family issue, I tried to configure all VXLAN interfaces with IPv6, but
> I ran into an issue with the source IP address selection.
>
> I specified the `local_ip` of the tunnel as `2403:400:31da:::18:6`,
> which is also added on the bridge interface, but it picks the link-local
> address of `fe80::dc03:37ff:fee2:1fef` instead. This gets dropped by other
> devices along the way.
>
> ```
> $ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev | grep 192.168.1.33
> [abbreviated]
> ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
> skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
> packets:1, bytes:42, used:0.063s, dp:ovs,
> actions:tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4,eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd),ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)),push_vlan(vid=304,pcp=0),exit_p0,
> dp-extra-info:miniflow_bits(4,0)
> ```
>
> ```
> $ ovs-vsctl show
> Bridge br-int
> datapath_type: netdev
> Port dpdk-vm101
> Interface dpdk-vm101
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
> Port vxlan0
> Interface vxlan0
> type: vxlan
> options: {dst_port="4789", key="1",
> local_ip="2403:400:31da:::18:6", remote_ip="2403:400:31da:::18:3"}
> ```
>
> The bridge interface has the `local_ip` addresses.
>
> ```
> $ ip a
> [abbreviated]
> 27: br-phy: mtu 1500 qdisc fq_codel
> state UP group default qlen 1000
> link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
> inet 100.87.18.6/32 scope global br-phy
> valid_lft forever preferred_lft forever
> inet6 2403:400:31da:::18:6/128 scope global
> valid_lft forever preferred_lft forever
> inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
> valid_lft forever preferred_lft forever
> ```
>
> The kernel routing table show has a `src` of `2403:400:31da:::18:6`
>
> ```
> $ ip -6 route
> [abbreviated]
> ::1 dev lo proto kernel metric 256 pref medium
> 2403:400:31da:::18:3 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy
> proto bgp src 2403:400:31da:::18:6 metric 20 pref medium
> 2403:400:31da:::18:6 dev br-phy proto kernel metric 256 pref medium
> 2403:400:31da::/48 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy proto bgp
> src 2403:400:31da:::18:6 metric 20 pref medium
> ```
> However, in OVS's route table, the link-local address of
> `fe80::dc03:37ff:fee2:1fef` is picked as the the SRC instead.
>
> ```
> $ ovs-appctl ovs/route/show
> Route Table:
> [abbreviated]
> Cached: 2403:400:31da:::18:3/128 dev br-phy GW fe80::920a:84ff:fe9e:9570
> SRC fe80::dc03:37ff:fee2:1fef
> Cached: 2403:400:31da:::18:6/128 dev br-phy SRC 2403:400:31da:::18:6
> ```
>
> I tried to rewrite the source address on bridge br-phy but this not
> seem to have any effect(no packet hits). My idea was to match udp
> packets of port 4789 and rewrite the IPv6 source address. Is my way
> of rewriting it correct?
>
> ```
>
> $ ovs-ofctl add-flow br-phy
> "priority=50,dl_type=0x86dd,ipv6_src=fe80::dc03:37ff:fee2:1fef,nw_proto=17,tp_dst=4789,actions=set_field:2403:400:31da:::18:6->ipv6_src,normal"
>
> $ ovs-ofctl dump-flows br-phy
> cookie=0x0, duration=653.933s, table=0, n_packets=0, n_bytes=0,
> priority=50,udp6,ipv6_src=fe80::dc03:37ff:fee2:1fef,tp_dst=4789
> actions=load:0x180006->NXM_NX_IPV6_SRC[0..63],load:0x2403040031da->NXM_NX_IPV6_SRC[64..127],NORMAL
> cookie=0x0, duration=275973.900s, table=0, n_packets=1638167,
> n_bytes=152347460, priority=0 actions=NORMAL
> ```
>
> What did work was that if I add a static route for destination with> the
> correct source address and add an ARP entry for the source. So if
> the source address was picked up correctly by OVS, the tunnel works
> correctly.
>
> ```
> $ ovs-appctl ovs/route/add 2403:400:31da:::18:3 br-phy
> 2403:400:31da:::18:6
> OK
> $ ovs-appctl tn
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 1/26/24 09:37, Lim, Derrick wrote:
> Hi Ilya Maximets,
>
> Thank you for explanation. I've create the two bridges, but the packet
> seems to be dropped looking at the `ovs-appctl dpctl/dump-flows` output.
> I didn’t receive it on the remote host either.
>
> In my setup, the two physical hosts are separated by a L3 network
> (local=100.87.18.6/32, remote=100.87.18.3/32). The routes are learnt by
> a routing agent and exported to the kernel. The kernel has the correct
> routes, but this information does not seem to be synced to OVS.
>
> $ ip route get 100.87.18.3
> 100.87.18.3 via inet6 fe80::920a:84ff:fe9e:9570 dev br-phy src 100.87.18.6
> uid 0
> cache
>
> $ ovs-appctl ovs/route/lookup 100.87.18.3
> src 100.87.2.168
> gateway 100.87.2.129
> dev ens3f1v1
>
> Perhaps the problem is that because I'm using BGP unnumbered, so the IPv4
> destination has an IPv6 next-hop. I tried adding the route statically but
> it seems not to be accepted.
>
> $ ovs-appctl ovs/route/add 100.87.18.3/32 br-phy fe80::920a:84ff:fe9e:9570
> Invalid pkt_mark or gateway
> ovs-appctl: ovs-vswitchd: server returned an error
>
> I've included some additional outputs from my setup below if you find them
> helpful.
>
> Is the routing where I'm going wrong or do you have any other advice about my
> setup?
Hmm. Interesting. I looked through the code and I see that OVS
router module that is responsible for syncing routes from the
kernel to userspace doesn't expect routes with different families.
Such routes are ignored.
The manual ovs/route/add command also expects the next hop to be of
the same IP family, so it refuses to add an v4-via-v6 static route.
So, unfortunately, the setup would work fine with the kernel datapath,
since kernel does all the routing in that case, but it will not work
with userspace.
We need to add support for v4 via v6 routing to ovs-vswitchd in order
to make the tunnels work.
If you're interested in making the change, I could point you to
the right places in the code. :)
Otherwise, maybe someone else from the community will pick it up.
Best regards, Ilya Maximets.
>
> $ ovs-vsctl show
> [abbreviated]
> Bridge br-int
> datapath_type: netdev
> Port vxlan0
> Interface vxlan0
> type: vxlan
> options: {dst_port="4789", key="1", local_ip="100.87.18.6",
> remote_ip="100.87.18.3"}
> Port dpdk-vm101
> Interface dpdk-vm101
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
>
> Bridge br-phy
> fail_mode: standalone
> datapath_type: netdev
> Port br-phy
> tag: 304
> Interface br-phy
> type: internal
> Port exit_p0
> Interface exit_p0
> type: dpdk
> options: {dpdk-devargs=":c4:01.0"}
> ovs_version: "3.1.1"
>
> $ ip addr show
> [abbreviated]
> 27: br-phy: mtu 1500 qdisc fq_codel
> state UP group default qlen 1000
> link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
> inet 100.87.18.6/32 scope global br-phy
> valid_lft forever preferred_lft forever
> inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
> valid_lft forever preferred_lft forever
>
> $ ovs-ofctl dump-flows br-int
> cookie=0x0, duration=221.063s, table=0, n_packets=24, n_bytes=1008,
> priority=50,in_port="dpdk-vm101" actions=output:vxlan0
>
> $ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev
> [abbreviated]
> ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
> skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
> packets:5, bytes:210, used:0.447s, dp:ovs, actions:drop,
> dp-extra-info:miniflow_bits(4,0)
>
> Thank you,
> Derrick
>
> *From: *Ilya Maximets
> *Date: *Thursday, January 25, 2024 at 20:16
> *To: *Lim, Derrick | Derrick | CMD ,
> ovs-discuss@openvswitch.org
> *Cc: *i.maxim...@ovn.org
> *Subject: *Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
>
> [EXTERNAL] This message comes from an external organization.
>
> On 1/25/24 10:42, Lim, Derrick via discuss wrote:
>> Hey all,
>>
>> Is there a way I can encapsulate a packet with VXLAN, and then resubmit
>> it through OVS a
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 1/25/24 10:42, Lim, Derrick via discuss wrote: > Hey all, > > Is there a way I can encapsulate a packet with VXLAN, and then resubmit > it through OVS again to run other flow actions based on this encapsulated > packet? > > Currently, I have a OVS-DPDK setup where in the final step, before a packet > leaves the host, the group action is used to pick between multiple physical > ports, and then rewrite the mac address (mod_dl_dst) to that of the > destination's, as well as apply the appropriate vlan tag (mod_vlan_vid). > > I would like the encapsulate action to take place before the step mentioned > above. I created a tunnel port (eg. vxlan0). But if I set the action to this > port, the packet basically leaves OVS and I can't resubmit it. > > In the userspace tunneling example, two bridges are used so that information > from the kernel can be used for routing and ARP resolution. Is there a way I > can populate these fields through various flow actions if I already know what > they should be without going through the kernel? Or is going through the > kernel > absolutely required to create the data structure for encapsulation? The output to a tunnel is an 'output' action, i.e. the packet always leaves the bridge. And so, it requires routing after encapsulation in order to identify where it should go next. For routing we need IP addresses and a routing table. This information is normally synced from the kernel. You can add static routes via ovs-appctl ovs/route/add, but you still need IP addresses configured on bridges. Normally the problem of applying actions after encapsulation is solved by having a tunnel interface in one bridge (br-int) and the egress interfaces in the other bridge (br-phy). The br-phy should have an IP address from the tunnel subnet, so after encapsulation the packet is getting routed to br-phy. In br-phy the packet can be matched with OF rules and actions can be executed before sending it to the egress interface. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Flow not registered in OVSDB flow table
On 1/18/24 11:00, Mansuroglu, Zekeriya via discuss wrote: > Hi all, > > I’m using ovs through mininet with OpendayLight. The ovs version is 2.13.8 > and the mininet version is 2.3.0.dev6. I create a network with a single > switch and two hosts. After that I create a flow through the nortboubd > interface of opendaylight that works fine. I can also verify with ovs-ofctl > that the flow is accepted by the switch. I expected that the flow is also > registered in a table in ovsdb, but when I dump the database the flow doesn’t > appear anywhere in the datastore. > > Shouldn’t the flow be registered in one of the table Flow_Table, sFlow or > NetFlow? > > I aimed to use the ovsdb-client to monitor for changes of the flow table > configurations. As I understood ovsdb-client’s monitor command this should be > doable when changes of the flow configurations lead to changes in ovsdb. Is > this approach realistic? Hi, Zekeriya. OpenFlow rules are not stored in the database, so you can't do that. What you can do is to use OpenFlow monitor instead. ovs-ofctl has a correcponding command. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] 答复: dpif-netlink: ovs flow put failed (File exist) to datapath
On 1/17/24 08:38, David Zhang (张同剑)-浪潮信息 wrote: > Hi Ilya： > I have merged the path fixed by this kernel and tested my bug. The steps > are as follows, but the problem still persists. Adding a flow table failed. > step 1 : ovs-appctl dpctl/add-dp system@test > step 2: ovs-appctl dpctl/add-flow system@test > "in_port(1),eth_type(0x0800),ipv4(src=172.16.0.0/255.252,0,0,dst=172.30.48.48,frag=no)" > 2 > step 3: ovs-appctl dpctl/add-flow system@test > "in_port(1),eth_type(0x0800),ipv4(src=172.16.0.0/255.248,0,0,dst=172.30.48.48,frag=no)" > 2 > > out err info ： > ovs-vswitchd: updating flow table (File exists) > ovs-appctl: ovs-vswitchd:server returned an error This is expected, because you're trying to install a flow with exactly the same key, but different mask. You need to change the key in some bits that are not in the mask for the flow to be installed. The assumption here is that packet with ipv4(src=172.16.0.0,dst=172.30.48.48) will always match on the first flow, so there will be no upcall. But if you send a packet with src=172.20.0.0, it will not match the first flow and go to upcall. However, later we will fail to install a wider mask flow matching on src=172.20.0.0/255.248.0.0. This is because 16 & 252 == 16 is the same result as 20 & 248 == 16. This behavior was fixed in the kernel patch I mentioned by comparing unmasked keys during flow installation. Since 16 != 20, the second flow will be successfully installed with the fix applied. > > > > -邮件原件- > 发件人: Ilya Maximets > 发送时间: 2024年1月16日, 星期二 20:53 > 收件人: David Zhang (张同剑)-浪潮信息 ; b...@openvswitch.org > 抄送: i.maxim...@ovn.org > 主题: Re: [ovs-discuss] dpif-netlink: ovs flow put failed (File exist) to > datapath > > On 1/15/24 12:58, David Zhang (张同剑)-浪潮信息 via discuss wrote: >> Hi: >> >>ovs flows fail to create, with file exists error, Have you met this one? >> >>The user state flow table cannot be correctly distributed to the >> datapath, resulting in the rule datapath flow table not matching correctly. >> Upcall is required. >> >> OVS LOG: >> >> >> 2024-01-12T07:12:51.657Z|00080|dpif(handler51)|DBG|system@ovs-system: >> failed to put[create] (File exists) >> ufid:5df6220d-e2d2-48f8-a6a4-5c47ccc869c4 >> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(3),skb_mark(0/0),c >> t_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),eth(src=c4:e >> 2:87:01:84:00,dst=fa:16:3e:25:b1:3b),eth_type(0x8100),vlan(vid=80,pcp= >> 0),encap(eth_type(0x0800), >> >> ipv4(src=172.20.226.17/255.248.0.0,dst=172.30.48.48,proto=6,tos=0/0,tt >> l=63,frag=no),tcp(src=5201/0,dst=57674/0),tcp_flags(0/0)), >> actions:ct_clear,ct_clear,set(eth(src=fa:16:3e:8f:73:57,dst=fa:16:3e:5 >> 9:e8:49)),set(ipv4(dst=10.147.7.12,ttl=62)),pop_vlan,ct(zone=1),recirc >> (0x13b6c8) >> > > It's hard to tell for sure what is going on here, but the issue might be > caused by a kernel issue fixed in: > > https://lore.kernel.org/netdev/167111551443.359845.7122827280135116424.stgit@ebuild/ > > You may need to upgrade your kernel, if it doesn't include the commit above. > >> Datapath Flow : >> >> >> recirc_id(0),in_port(3),ct_state(-new-est-rel-rpl-inv-trk),ct_label(0/ >> 0x1),eth(src=5c:e7:47:05:34:00,dst=fa:16:3e:25:b1:3b),eth_type(0x8100) >> ,vlan(vid=80,pcp=0),encap(eth_type(0x0800), >> >> ipv4(src=172.16.0.0/255.252.0.0,dst=172.30.48.48,proto=6,ttl=63,frag=n >> o)), packets:2355652448, bytes:11500879337271, used:0.220s, >> flags:SFPR., >> actions:ct_clear,ct_clear,set(eth(src=fa:16:3e:8f:73:57,dst=fa:16:3e:5 >> 9:e8:49)),set(ipv4(dst=10.147.7.12,ttl=62)),pop_vlan,ct(zone=1),recirc >> (0x13b6c8) >> >> ovs version: 2.15.2 > > Note: 2.15 is a fairly old release now and it is no longer supported. > You should upgrade to at least 2.17. > > Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 1/16/24 13:09, Reshma Sreekumar wrote: > So generally the IFB netdev is used to enforce QoS on egress when the > traffic is ingress in reality i.e. one can redirect ingressing traffic > (or even egress for that matter) to an IFB netdev and it will hit the > egress qdisc on IFB, where we can configure the egress QoS. However, > another usecase is to have consolidated traffic arrive on IFB for QoS . > i.e. ingress traffic from different netdevs redirected to IFB netdev for > a combined QoS. And my use-case is similar, where I want traffic arriving > on several ports, including an ovs-enslaved port to be redirected to IFB > for QoS. So, in that case, I believe, enslaving IFB into ovs will not work > for me. If ovs supports some sort of ingress redirection, then that might > help! So, you don't want traffic to return to OVS? If so, why you need that interface to be in OVS at all? You may create another veth pair and add one side of it to OVS, then add an OpenFlow rule to direct all traffic from veth1 to that new port. Traffic will appear on the other side of a veth. Though I'm not sure why OVS is in the picture in the first place. > > Thanks & Regards, > RS > > On Mon, Jan 15, 2024 at 10:23 PM Ilya Maximets <mailto:i.maxim...@ovn.org>> wrote: > > On 1/15/24 21:05, Reshma Sreekumar via discuss wrote: > > I see..thanks for the explanation. My use-case is to have a traffic > redirection > > from an ovs-enslaved port to IFB netdevice on ingress Qdisc. > Considering the > > given mode of operation,may be it's better if I opt for some other > methods of > > redirection to IFB netdevice? > > I never worked with IFB interfaces, so I'm not 100% sure that will work, > but you may remove your port from OVS and add you IFB interface instead. > Egress qdisc on IFB will be in use, you may protect it from being removed > by using linux-noop QoS type. > > I suppose, the traffic will enter OVS after IFB. So, you'll have your > veth1 (ingress qdisc) --> ifb (egress qdisc) --> OVS. Should be fine > if you're going to redirect all the traffic to ifb. > > Let us know if that works. > > Best regards, Ilya Maximets. > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] dpif-netlink: ovs flow put failed (File exist) to datapath
On 1/15/24 12:58, David Zhang (张同剑)-浪潮信息 via discuss wrote: > Hi: > > ovs flows fail to create, with file exists error, Have you met this one? > > The user state flow table cannot be correctly distributed to the datapath, > resulting in the rule datapath flow table not matching correctly. Upcall is > required. > > OVS LOG: > > 2024-01-12T07:12:51.657Z|00080|dpif(handler51)|DBG|system@ovs-system: > failed to put[create] (File exists) ufid:5df6220d-e2d2-48f8-a6a4-5c47ccc869c4 > recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(3),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),eth(src=c4:e2:87:01:84:00,dst=fa:16:3e:25:b1:3b),eth_type(0x8100),vlan(vid=80,pcp=0),encap(eth_type(0x0800), > > ipv4(src=172.20.226.17/255.248.0.0,dst=172.30.48.48,proto=6,tos=0/0,ttl=63,frag=no),tcp(src=5201/0,dst=57674/0),tcp_flags(0/0)), > > actions:ct_clear,ct_clear,set(eth(src=fa:16:3e:8f:73:57,dst=fa:16:3e:59:e8:49)),set(ipv4(dst=10.147.7.12,ttl=62)),pop_vlan,ct(zone=1),recirc(0x13b6c8) > It's hard to tell for sure what is going on here, but the issue might be caused by a kernel issue fixed in: https://lore.kernel.org/netdev/167111551443.359845.7122827280135116424.stgit@ebuild/ You may need to upgrade your kernel, if it doesn't include the commit above. > Datapath Flow : > > > recirc_id(0),in_port(3),ct_state(-new-est-rel-rpl-inv-trk),ct_label(0/0x1),eth(src=5c:e7:47:05:34:00,dst=fa:16:3e:25:b1:3b),eth_type(0x8100),vlan(vid=80,pcp=0),encap(eth_type(0x0800), > > ipv4(src=172.16.0.0/255.252.0.0,dst=172.30.48.48,proto=6,ttl=63,frag=no)), > packets:2355652448, bytes:11500879337271, used:0.220s, flags:SFPR., > actions:ct_clear,ct_clear,set(eth(src=fa:16:3e:8f:73:57,dst=fa:16:3e:59:e8:49)),set(ipv4(dst=10.147.7.12,ttl=62)),pop_vlan,ct(zone=1),recirc(0x13b6c8) > > ovs version: 2.15.2 Note: 2.15 is a fairly old release now and it is no longer supported. You should upgrade to at least 2.17. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 1/15/24 21:05, Reshma Sreekumar via discuss wrote: > I see..thanks for the explanation. My use-case is to have a traffic > redirection > from an ovs-enslaved port to IFB netdevice on ingress Qdisc. Considering the > given mode of operation,may be it's better if I opt for some other methods of > redirection to IFB netdevice? I never worked with IFB interfaces, so I'm not 100% sure that will work, but you may remove your port from OVS and add you IFB interface instead. Egress qdisc on IFB will be in use, you may protect it from being removed by using linux-noop QoS type. I suppose, the traffic will enter OVS after IFB. So, you'll have your veth1 (ingress qdisc) --> ifb (egress qdisc) --> OVS. Should be fine if you're going to redirect all the traffic to ifb. Let us know if that works. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 1/12/24 15:31, Reshma Sreekumar wrote: > Thanks for your suggestion. > > However, this is what I observe: > > ~$ *sudo ovs-vsctl add-port br0 veth1 -- set port veth1 qos=@myqos -- > --id=@myqos create qos type=linux-noop* > ac47866d-6812-47a0-a5e2-ec00cfa0e849 > ~$ sudo ovs-vsctl show > f96fba96-cc42-4d96-8eba-c8b4e3b4af2f > Bridge br0 > Port veth1 > Interface veth1 > Port br0 > Interface br0 > type: internal > ovs_version: "2.17.1" > ~$ sudo tc qdisc add dev veth1 ingress > ~$ sudo tc -p -s -d qdisc show dev veth1 > qdisc noqueue 0: root refcnt 2 > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > qdisc ingress : parent :fff1 > ingress qdisc present > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > > > ~$ sudo tcpdump -i veth1 > tcpdump: verbose output suppressed, use -v[v]... for full protocol decode > listening on veth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes > ^C > 0 packets captured > 0 packets received by filter > 0 packets dropped by kernel > > > ~$ sudo tc -p -s -d qdisc show dev veth1 > qdisc noqueue 0: root refcnt 2 > ingress qdisc lost > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > ~$ Hmm, yeah, I missed the fact that you're creating an ingress qdisc. Ingress qdiscs are controlled by OVS as well via ingress_policing_* options. These default to 0 that means no policing. And OVS ensures that there is no policing by removing any ingress qdiscs form the interface. There is no work around for that. linux-noop only allows non-ingress qdiscs. Ingress qdiscs may also be used for hardware offloading purposes, so the externally created ones will also be removed in this case. If you want to use ingress qdiscs for the purpose of ingress traffic policing, you should use the ingress_policing_* configuration options instead. If you have some other use case for external ingress qdiscs, OVS will have to be changed to be aware of it. Best regards, Ilya Maximets. > > Regards, > RS > > On Thu, Jan 11, 2024 at 5:42 PM Ilya Maximets <mailto:i.maxim...@ovn.org>> wrote: > > (trying to re-send as gmail decided to not deliver the previous email) > > On 1/11/24 17:38, Ilya Maximets wrote: > > On 1/11/24 17:25, Reshma Sreekumar via discuss wrote: > >> Hello all, > >> > >> An ingress qdisc configured on a port that is enslaved into an > OpenvSwitch > >> bridge gets automatically deleted when tcpdump is run on that port! > > > > OVS owns QoS configuration of ports attached to it, so it is expected. > > If you want OVS to avoid removing qdiscs configured externally, you > > may configure a "linux-noop" QoS class for this interface. E.g.: > > > > $ ovs-vsctl set port veth1 qos=@myqos -- --id=@myqos create qos > type=linux-noop > > > > Best regards, Ilya Maximets. > > > >> > >> *OVS version :* > >> > >> ovs-vsctl (Open vSwitch) 2.17.1 > >> DB Schema 8.3.0 > >> * > >> * > >> *OVS bridge config* > >> > >> Bridge br0 > >> Port br0 > >> Interface br0 > >> type: internal > >> Port L3port > >> Interface L3port > >> type: internal > >> Port veth1 > >> Interface veth1 > >> > >> *Steps to reproduce:* > >> > >> 1. sudo tc qdisc add dev veth1 ingress > >> 2. sudo tc filter add dev veth1 parent : protocol all u32 match > u32 0 0 action mirred ingress redirect dev ifb0 > >> 3. sudo tcpdump -i veth1 > >> > >> *And by running `tcpdump`, the ingress qdisc is lost!* > >> > >> Thanks & Regards, > >> Reshma Sreekumar > > > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
(trying to re-send as gmail decided to not deliver the previous email) On 1/11/24 17:38, Ilya Maximets wrote: > On 1/11/24 17:25, Reshma Sreekumar via discuss wrote: >> Hello all, >> >> An ingress qdisc configured on a port that is enslaved into an OpenvSwitch >> bridge gets automatically deleted when tcpdump is run on that port! > > OVS owns QoS configuration of ports attached to it, so it is expected. > If you want OVS to avoid removing qdiscs configured externally, you > may configure a "linux-noop" QoS class for this interface. E.g.: > > $ ovs-vsctl set port veth1 qos=@myqos -- --id=@myqos create qos > type=linux-noop > > Best regards, Ilya Maximets. > >> >> *OVS version :* >> >> ovs-vsctl (Open vSwitch) 2.17.1 >> DB Schema 8.3.0 >> * >> * >> *OVS bridge config* >> >> Bridge br0 >> Port br0 >> Interface br0 >> type: internal >> Port L3port >> Interface L3port >> type: internal >> Port veth1 >> Interface veth1 >> >> *Steps to reproduce:* >> >> 1. sudo tc qdisc add dev veth1 ingress >> 2. sudo tc filter add dev veth1 parent : protocol all u32 match u32 0 0 >> action mirred ingress redirect dev ifb0 >> 3. sudo tcpdump -i veth1 >> >> *And by running `tcpdump`, the ingress qdisc is lost!* >> >> Thanks & Regards, >> Reshma Sreekumar > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Ingress qdisc on ovs-bridge-enslaved port get automatically removed
On 1/11/24 17:25, Reshma Sreekumar via discuss wrote: > Hello all, > > An ingress qdisc configured on a port that is enslaved into an OpenvSwitch > bridge gets automatically deleted when tcpdump is run on that port! OVS owns QoS configuration of ports attached to it, so it is expected. If you want OVS to avoid removing qdiscs configured externally, you may configure a "linux-noop" QoS class for this interface. E.g.: $ ovs-vsctl set port veth1 qos=@myqos -- --id=@myqos create qos type=linux-noop Best regards, Ilya Maximets. > > *OVS version :* > > ovs-vsctl (Open vSwitch) 2.17.1 > DB Schema 8.3.0 > * > * > *OVS bridge config* > > Bridge br0 > Port br0 > Interface br0 > type: internal > Port L3port > Interface L3port > type: internal > Port veth1 > Interface veth1 > > *Steps to reproduce:* > > 1. sudo tc qdisc add dev veth1 ingress > 2. sudo tc filter add dev veth1 parent : protocol all u32 match u32 0 0 > action mirred ingress redirect dev ifb0 > 3. sudo tcpdump -i veth1 > > *And by running `tcpdump`, the ingress qdisc is lost!* > > Thanks & Regards, > Reshma Sreekumar ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Match on header rewrite fields
On 1/4/24 07:35, Eli Britstein via discuss wrote: > > >> -Original Message----- >> From: Ilya Maximets >> Sent: Wednesday, 3 January 2024 19:12 >> To: Eli Britstein ; ovs-discuss@openvswitch.org >> Cc: Maor Dickman ; i.maxim...@ovn.org >> Subject: Re: [ovs-discuss] Match on header rewrite fields >> >> External email: Use caution opening links or attachments >> >> >> On 12/19/23 17:00, Eli Britstein via discuss wrote: >>> Hello, >>> >>> In case there is a header rewrite action, OVS implicitly adds match on the >> corresponding fields. >>> >>> For example: >>> ovs-ofctl add-flow br-int0 >> 'in_port=enp8s0f0,ip,actions=mod_dl_dst=00:11:22:33:44:55,enp8s0f0_0' >>> >>> The datapath flow will have an exact match on dl_dst field. >>> >>> Does someone know what is the logic behind this? >> >> Hi, Eli. The main reason for this, AFAIK, is that OVS is actually using the >> field >> value while trying to optimize the action later. If the action doesn't >> really >> change the value, the action will be omitted. In this case, if we do not >> have a >> match on the value, the action will not be executed for other packets as >> well. >> And that is a problem. It's also hard to debug such issues, because it >> depend >> on which packet hits OVS first. >> >> E.g. if the packet with dl_dst=00:11:22:33:44:55 hits OVS, OVS will install >> the >> following datapath flow: >> >> in_port(enp8s0f0),eth(dst=00:11:22:33:44:55), actions:enp8s0f0_0 >> >> For a packet with dl_dst=00:11:22:33:44:77, it will be: >> >> in_port(enp8s0f0),eth(dst=00:11:22:33:44:77), >> actions:set(eth(dst=00:11:22:33:44:55)),enp8s0f0_0 >> >> If we remove the match, the first flow will turn into: >> >> in_port(enp8s0f0), actions:enp8s0f0_0 >> >> swallowing all the traffic without modifying the fields, if it happens to be >> installed first. >> >> There might be a room for optimization, e.g. by making sure the flow that >> doesn't have an action has a match, but the one that has an action, doesn't >> have a match. In this case we will end up with overlapping flows in the >> detapath. These are fine as long as the wider one has all the actions. >> However, this would need a very careful consideration, as it's hard to tell >> right >> away if the set of flows will be correct in the end or not. >> >> Let's see what will happen with a basic flow that changes 2 >> fields: >> >> ip, actions:set_field:A->eth_src,set_field:B->eth_dst >> >> Packet eth(A,C) results in datapath flow: >> 1. eth(src=A), actions:set(eth(dst=B)) >> >> Packet eth(C,B) results in datapath flow: >> 2. eth(dst=B), actions:set(eth(src=A)) >> >> Packet eth(A,B) results in datapath flow: >> 3. eth(src=A,dst=B), actions: >> >> Packet eth(C,D) results in datapath flow: >> 4. , actions:set(eth(src=A,dst=B)) >> >> After all the flows installed: >> Packet: >> eth(A,C) -> matches #1 -> set(eth(dst=B))-> eth(A,B) >> matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) >> >> eth(A,B) -> matches all 4, ether way result will be eth(A,B) >> >> eth(C,B) -> matches #2 -> set(eth(src=A))-> eth(A,B) >> matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) >> >> eth(C,D) -> matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) >> >> So, this might work, assuming datapath correctly handles overlapping flows, >> which it should. But it also becomes a bit more complex to process and adds >> a >> potential unnecessary overwrite of the fields that don't need it. The action >> execution performance is likely not a problem here. I'm not sure if having >> too >> many overlapping flows may cause performance issues for the datapath's >> packet classifier though. As seen above, we can have a lot of them with >> increasing number of fields. >> >> The action commit code also changed quite a bit since the logic was >> introduced, so there might be problems with this approach that I'm not aware >> of. >> >> Best regards, Ilya Maximets. > > Hi Ilya, > > Thanks for your input. > The optimization to avoid rewrite of the same value was done at [1]. Before > that #3 dp-flow created in your example was like this: > eth(src=A,dst=B), actions:set(eth(src=A,dst=B)) Not really. Commit [1] is only responsible for different fields within th
Re: [ovs-discuss] Match on header rewrite fields
On 12/19/23 17:00, Eli Britstein via discuss wrote: > Hello, > > In case there is a header rewrite action, OVS implicitly adds match on the > corresponding fields. > > For example: > ovs-ofctl add-flow br-int0 > 'in_port=enp8s0f0,ip,actions=mod_dl_dst=00:11:22:33:44:55,enp8s0f0_0' > > The datapath flow will have an exact match on dl_dst field. > > Does someone know what is the logic behind this? Hi, Eli. The main reason for this, AFAIK, is that OVS is actually using the field value while trying to optimize the action later. If the action doesn't really change the value, the action will be omitted. In this case, if we do not have a match on the value, the action will not be executed for other packets as well. And that is a problem. It's also hard to debug such issues, because it depend on which packet hits OVS first. E.g. if the packet with dl_dst=00:11:22:33:44:55 hits OVS, OVS will install the following datapath flow: in_port(enp8s0f0),eth(dst=00:11:22:33:44:55), actions:enp8s0f0_0 For a packet with dl_dst=00:11:22:33:44:77, it will be: in_port(enp8s0f0),eth(dst=00:11:22:33:44:77), actions:set(eth(dst=00:11:22:33:44:55)),enp8s0f0_0 If we remove the match, the first flow will turn into: in_port(enp8s0f0), actions:enp8s0f0_0 swallowing all the traffic without modifying the fields, if it happens to be installed first. There might be a room for optimization, e.g. by making sure the flow that doesn't have an action has a match, but the one that has an action, doesn't have a match. In this case we will end up with overlapping flows in the detapath. These are fine as long as the wider one has all the actions. However, this would need a very careful consideration, as it's hard to tell right away if the set of flows will be correct in the end or not. Let's see what will happen with a basic flow that changes 2 fields: ip, actions:set_field:A->eth_src,set_field:B->eth_dst Packet eth(A,C) results in datapath flow: 1. eth(src=A), actions:set(eth(dst=B)) Packet eth(C,B) results in datapath flow: 2. eth(dst=B), actions:set(eth(src=A)) Packet eth(A,B) results in datapath flow: 3. eth(src=A,dst=B), actions: Packet eth(C,D) results in datapath flow: 4. , actions:set(eth(src=A,dst=B)) After all the flows installed: Packet: eth(A,C) -> matches #1 -> set(eth(dst=B))-> eth(A,B) matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) eth(A,B) -> matches all 4, ether way result will be eth(A,B) eth(C,B) -> matches #2 -> set(eth(src=A))-> eth(A,B) matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) eth(C,D) -> matches #4 -> set(eth(src=A,dst=B)) -> eth(A,B) So, this might work, assuming datapath correctly handles overlapping flows, which it should. But it also becomes a bit more complex to process and adds a potential unnecessary overwrite of the fields that don't need it. The action execution performance is likely not a problem here. I'm not sure if having too many overlapping flows may cause performance issues for the datapath's packet classifier though. As seen above, we can have a lot of them with increasing number of fields. The action commit code also changed quite a bit since the logic was introduced, so there might be problems with this approach that I'm not aware of. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Default selection method
On 12/22/23 17:33, Celyn Birkinshaw via discuss wrote:
> Hi,
>
> I've made a topology of four OVS (version 2.17.8) switches in a diamond,
> two endpoints, and I'm using a Ryu controller. Installed in br1 is a group
> and a flow entry to direct packets out via buckets, toward br2 and br3.
> The group is set via the Ryu API in an OpenFlow v1.5 OFPGroupMod message.
> The problem is that I cannot trace packets through the buckets since there
> is 'no live bucket' in the group. However, if I check the group using
> ovs-ofctl, it looks ok.
It is an issue of the trace output, it should say something like
"selection method in use: dp_hash, recirculating" instead of
"no live bucket". You can see the actual datapath actions at the
end of all your traces:
Datapath actions: hash(l4(0)),recirc(0x1b)
OpenFlow layer doesn't know what the hash will be, so it can't
know to which bucket the packet will be directed to. It instructs
the datapath to calculate the hash and recirculate the packet,
once the hash is known the packet will be passed through the
OpenFlow tables again and the bucket will be selected.
You may add dp_hash=0xabcd to the flow definition of the trace
command to see to which bucket the packet with a particular hash
will go. But ovs-vswitchd in general doesn't know what the hash
value will be, because it depends on datapath implementation.
Actual traffic should work fine, it's only a trace output that is
misleading.
> Also, if I remove it and reinstall it via ovs-ofctl,
> I can trace a packet through a bucket but I have to set the group selection
> method to 'hash'.
'hash' is showing the trace because in this case the hash is
calculated in userspace, so OpenFlow layer can calculate the
hash right away and figure out which bucket it will go.
> I'm looking at how to include the selection method in the
> properties of an OFPGroupMod message, or how to set the default selection
> method in the switches, or any other suggestions.
If you still want to use the 'selection_method' property, it is a
Netronome extension NTRT_SELECTION_METHOD. See more details here:
https://raw.githubusercontent.com/openvswitch/ovs/master/Documentation/group-selection-method-property.txt
Best regards, Ilya Maximets.
> Here is a set of commands to show what I'm seeing in OVS (initial group
> installed via Ryu's API):
>
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ ./groups.sh
> br1
> OFPST_GROUP_DESC reply (OF1.5) (xid=0x2):
>
> group_id=1,type=select,bucket=bucket_id:0,weight:10,actions=output:"vp1",bucket=bucket_id:1,weight:10,actions=output:"vp2"
>
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ ./flows.sh
> br1
> cookie=0x1, duration=8.464s, table=0, n_packets=0, n_bytes=0, idle_age=8,
> priority=1,in_port="fd4352db8bf24_l",dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12
> actions=group:1
>
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ sudo ovs-appctl ofproto/trace br1
> in_port=3,tcp,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,nw_src=10.2.0.2,nw_dst=10.2.0.3
> Flow:
> tcp,in_port=3,vlan_tci=0x,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,nw_src=10.2.0.2,nw_dst=10.2.0.3,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no,tp_src=0,tp_dst=0,tcp_flags=0
>
> bridge("br1")
> -
> 0. in_port=3,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12, priority 1,
> cookie 0x1
> group:1
> -> no live bucket
>
> Final flow: unchanged
> Megaflow:
> recirc_id=0,eth,ip,in_port=3,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,nw_src=0.0.0.0/2,nw_dst=0.0.0.0/2,nw_frag=no
> <http://0.0.0.0/2,nw_dst=0.0.0.0/2,nw_frag=no>
> Datapath actions: hash(l4(0)),recirc(0x1b)
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ ovs-ofctl del-groups br1
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ ovs-ofctl add-group br1
> group_id=1,type=select,bucket=bucket_id:0,weight:10,actions=output:"vp1",bucket=bucket_id:1,weight:10,actions=output:"vp2"
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ ovs-ofctl add-flow br1
> in_port=3,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,actions=group:1
> ubuntu@ip-172-31-18-197:~/sdn/ovs_scripts$ sudo ovs-appctl ofproto/trace br1
> in_port=3,tcp,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,nw_src=10.2.0.2,nw_dst=10.2.0.3
> Flow:
> tcp,in_port=3,vlan_tci=0x,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12,nw_src=10.2.0.2,nw_dst=10.2.0.3,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no,tp_src=0,tp_dst=0,tcp_flags=0
>
> bridge("br1")
> -
> 0. in_port=3,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:2b:12, priority
> 32768
> group:1
> -> no live bucket
>
> Final flow: unchanged
> Megaflow:
> recirc_id=0,eth,ip,in_port=3,dl_src=de:9c:43:99:7c:46,dl_dst=ce:5c:1e:cd:
Re: [ovs-discuss] bugzilla.kernel.org 218039
On 11/1/23 17:05, Chuck Lever III wrote: > > >> On Nov 1, 2023, at 3:18 AM, Ilya Maximets wrote: >> >> On 10/31/23 22:00, Chuck Lever III via discuss wrote: >>> Hi- >>> >>> I recently made some changes to tmpfs/shmemfs and someone has reported >>> an occasional memory leak, possibly when using ovs_vswitch.service. Can >>> I get someone to have a look at this report and perhaps make some >>> suggestions (or shoot down my working theory) ? >> >> Hi, Chuck. >> >> I looked at the bug, but the ovs-ctl script doesn't really do anything >> exceptional with tmpfs. It does use it though in a following way: >> >> 1. Create a couple of files with mktemp: >> >> https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L629 >> >> 2. Write some commands into these files, e.g.: >> >> https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L588 >> >> 3. Make them executable: >> >> https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L663 >> >> 4. Execute: >> >> https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L617 >> >> 5. Files are removed with 'rm -f' by an exit trap: >> >> https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L632 >> >> Also, ovsdb-server process is using the most basic version of a >> temporary file in its runtime, created by tmpfile(3). But this >> process is not even restarted by the ovs-vswitchd restart. > > Ilya, your response is very much appreciated. Would you mind > if I copied it to 218039 ? Sure. No problem. The list is actually public: https://mail.openvswitch.org/pipermail/ovs-discuss/2023-November/052783.html (bugs@ is an alias for ovs-discuss@) Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] bugzilla.kernel.org 218039
On 10/31/23 22:00, Chuck Lever III via discuss wrote: > Hi- > > I recently made some changes to tmpfs/shmemfs and someone has reported > an occasional memory leak, possibly when using ovs_vswitch.service. Can > I get someone to have a look at this report and perhaps make some > suggestions (or shoot down my working theory) ? Hi, Chuck. I looked at the bug, but the ovs-ctl script doesn't really do anything exceptional with tmpfs. It does use it though in a following way: 1. Create a couple of files with mktemp: https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L629 2. Write some commands into these files, e.g.: https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L588 3. Make them executable: https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L663 4. Execute: https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L617 5. Files are removed with 'rm -f' by an exit trap: https://github.com/openvswitch/ovs/blob/fdbf0bb2aed53e70b455eb1adcfda8d8278ea690/utilities/ovs-lib.in#L632 Also, ovsdb-server process is using the most basic version of a temporary file in its runtime, created by tmpfile(3). But this process is not even restarted by the ovs-vswitchd restart. HTH. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Restarting the network triggers the deletion of one ovs port
On 10/31/23 03:59, Liqi An wrote: > Hi , > Ok , I will continue to consult SUSE support on this issue . > > About the new workaround, I hope bond1 work as before , > > A> Old solution(bond1 in bridge: br-oam will lost when restarting network of > host): > cluster12-b:/etc/sysconfig/network # cat ifcfg-bond1 > DEVICE='bond1' > BONDING_MASTER='yes' > BONDING_MODULE_OPTS='mode=active-backup miimon=100 use_carrier=0' > BONDING_SLAVE0='eth1' > BONDING_SLAVE1='eth5' > BOOTPROTO='static' > BORADCAST='' > ETHTOOL_OPTIONS='' > IPADDR='' > MTU='' > NAME='' > NETMASK='' > NETWORK='' > REMOTE_IPADDR='' > STARTMODE='auto' > USERCONTROL='no' > BONDING_SKIP_REMOVE_WORKAROUND='yes' > ZONE=public > > # /usr/bin/ovs-vsctl add-port br-oam bond1 trunk=3932,3933 > > Bridge br-oam > Port "2.11-SC-2-eth1" > tag: 3932 > Interface "2.11-SC-2-eth1" > Port bond1 > trunks: [3932, 3933] > Interface bond1 > Port "2.11-SC-2-eth2" > tag: 3933 > Interface "2.11-SC-2-eth2" > Port br-oam > Interface br-oam > type: internal > > > > > B> New solution: > # ovs-vsctl add-bond br-oam bond1 eth1 eth5 trunk=3932,3933 > >Bridge br-oam > Port br-oam > Interface br-oam > type: internal > Port bond1 > trunks: [3932, 3933] > Interface eth5 > Interface eth1 > Port "2.11-SC-2-eth2" > tag: 3933 > Interface "2.11-SC-2-eth2" > Port "2.11-SC-2-eth1" > tag: 3932 > Interface "2.11-SC-2-eth1" > > Would bond1 in solution B> work in the same way as bond1 in solution A> > , Especially the work of eth1 > as ' BONDING_MODULE_OPTS='mode=active-backup miimon=100 use_carrier=0''? Or > more parameter configuration is required in > command"# ovs-vsctl add-bond br-oam bond1 eth1 eth5 trunk=3932,3933" ? By default the bonding mode is active-backup, but the default failure detection mode is carrier, so you'll need to change that in order to match the previous configuration. For example: ovs-vsctl set port bond1 other_config:bond-detect-mode=miimon \ other_config:bond-miimon-interval=100 > > > //An > > -Original Message- > From: Ilya Maximets > Sent: Monday, October 30, 2023 7:09 PM > To: Liqi An ; ovs-discuss@openvswitch.org > Cc: i.maxim...@ovn.org; Cheng Chi ; Jonas Yi > ; Yawei Lu > Subject: Re: [ovs-discuss] Restarting the network triggers the deletion of > one ovs port > > On 10/30/23 10:28, Liqi An wrote: >> Hi , >> Is there any update about this issue , It's been bothering me for >> over two weeks,thx~ > > Hi. As you saw in the log, something is calling ovs-vsctl to remove the port > from OVS: > > 2023-10-16T13:07:17.668420+08:00 cluster12-b ovs-vsctl: > ovs|1|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port br-oam bond1 > > OVS is not doing that on its own. You need to find what is calling this > command in order to fix the problem. Likely candidates are network-scripts, > NetworkManager or something similar. > > Best regards, Ilya Maximets. > >> >> //An >> >> -Original Message- >> From: Liqi An >> Sent: Wednesday, October 18, 2023 5:13 PM >> To: Ilya Maximets ; ovs-discuss@openvswitch.org >> Cc: Cheng Chi ; Jonas Yi >> ; Yawei Lu ; IPW AQUA >> team (CBC) >> Subject: RE: [ovs-discuss] Restarting the network triggers the >> deletion of one ovs port >> >> Hi , >> We added bond1 to br-oam with below command before: >> >> # ovs-vsctl add-port br-oam bond1 trunk=3932,3933 >> >> In my opinion , , once it add bond1 to the virtual switch >> (openvswtich) successfully , this configuration should be saved into ovs db >> , What I'm wondering is why the network management on the host actively >> triggers the deletion, That's not supposed to be the job of SUSE15-SP5 , The >> ovs-vsctl command is used to manually add or delete these configurations >> before. >> >> >> By the way , I tried another way , delete bond1 configuration of >> previous host , and add eth1 as bond1 to openvswitch directly: >> WA: >> cluster12-b:/etc/sysconfig/network # ip link set bond1 down >> cluster12-b:/etc/sysconfig/network # rm -f >> /etc/sysconfig/network/ifcfg-bond1 >> cluster12-b:
Re: [ovs-discuss] OVS & OVN HWOL with Nvidia ConnectX-6 Dx - Kernel flower acknowledgment does not match request
On 10/26/23 14:05, Odintsov Vladislav wrote:
> Hi,
>
>> On 19 Oct 2023, at 17:06, Vladislav Odintsov via discuss
>> wrote:
>>
>>
>>
>>> On 18 Oct 2023, at 18:43, Ilya Maximets via discuss
>>> wrote:
>>>
>>> On 10/18/23 16:24, Vladislav Odintsov wrote:
>>>> Hi Ilya,
>>>>
>>>> thanks for your response!
>>>>
>>>>> On 18 Oct 2023, at 15:59, Ilya Maximets via discuss
>>>>> wrote:
>>>>>
>>>>> On 10/17/23 16:30, Vladislav Odintsov via discuss wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I’m testing OVS hardware offload with tc flower with Mellanox/NVidia
>>>>>> ConnectX-6 Dx smartnic and see next warning in ovs-vswitchd log:
>>>>>>
>>>>>> 2023-10-17T14:23:15.116Z|00386|tc(handler20)|WARN|Kernel flower
>>>>>> acknowledgment does not match request! Set dpif_netlink to dbg to see
>>>>>> which rule caused this error.
>>>>>>
>>>>>> With dpif_netlink debug logs enabled, after this message appears two
>>>>>> additional lines:
>>>>>>
>>>>>> 2023-10-17T14:23:15.117Z|00387|dpif_netlink(handler20)|DBG|added flow
>>>>>> 2023-10-17T14:23:15.117Z|00388|dpif_netlink(handler20)|DBG|system@ovs-system:
>>>>>> put[create] ufid:d8a3ab6d-77d1-4574-8bbf-634b01a116f3
>>>>>> recirc_id(0),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0x10,src=10.1.0.105,dst=10.1.0.109,ttl=64/0,tp_src=59507/0,tp_dst=6081/0,geneve({class=0x102,type=0x80,len=4,0x60002}),flags(-df+csum+key)),in_port(4),skb_mark(0/0),ct_state(0/0x2f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x3),eth(src=00:00:ba:a4:6e:ad,dst=00:01:ba:a4:6e:ad),eth_type(0x0800),ipv4(src=172.32.2.4/0.0.0.0,dst=172.32.1.4/0.0.0.0,proto=1,tos=0/0x3,ttl=63/0,frag=no),icmp(type=8/0,code=0/0),
>>>>>>
>>>>>> actions:set(tunnel(tun_id=0xff0011,src=10.1.0.109,dst=10.1.1.18,ttl=64,tp_src=59507,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x18000b}),flags(df|csum|key))),4
>>>>>>
>>>>>
>>>>> Could you also enable debug logs for 'tc' module in OVS?
>>>>> It shoudl give more infomation about where exactly is the
>>>>> difference between what OVS asked for and what the kenrel
>>>>> reported back.
>>>>>
>>>>> In general this warning typically signifies a kernel bug,
>>>>> but it could be that OVS doesn't format something correctly
>>>>> as well.
>>>>
>>>> With enabled tc logs I see mismatches in expected/real keys and actions:
>>>>
>>>> 2023-10-18T13:33:35.882Z|00118|tc(handler21)|DBG|tc flower compare failed
>>>> action compare
>>>> Expected Mask:
>>>> ff ff 00 00 ff ff ff ff-ff ff ff ff ff ff ff ff
>>>> 0030 00 00 2f 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 0040 03 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 0050 00 00 00 00 ff ff ff ff-00 00 00 00 00 00 00 00
>>>> 0060 00 00 00 00 ff 00 00 00-00 00 00 00 00 00 00 00
>>>> 0090 00 00 00 00 00 00 00 00-ff ff ff ff ff ff ff ff
>>>> 00c0 ff 00 00 00 ff ff 00 00-ff ff ff ff ff ff ff ff
>>>> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 00e0 ff ff ff 01 ff ff ff ff-00 00 00 00 00 00 00 00
>>>>
>>>> Received Mask:
>>>> ff ff 00 00 ff ff ff ff-ff ff ff ff ff ff ff ff
>>>> 0030 00 00 2f 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 0040 03 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 0050 00 00 00 00 ff ff ff ff-00 00 00 00 00 00 00 00
>>>> 0060 00 00 00 00 ff 00 00 00-00 00 00 00 00 00 00 00
>>>> 0090 00 00 00 00 00 00 00 00-ff ff ff ff ff ff ff ff
>>>> 00c0 ff 00 00 00 ff ff 00 00-ff ff ff ff ff ff ff ff
>>>> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>>>> 00e0 ff ff ff 01 ff ff ff ff-00 00 00 00 00 00 00 00
>>>>
>>>> Expected Key:
>>>> 08 06 00 00 ff ff ff ff-ff ff 00 00 ba a4 6e ad
>>>> 0050 a9 fe 64 01 a9 fe 64 03-00 00 ba a4 6e ad 00 00 <— mismatch in
>>>> this line
>>>> 0060 00 00 00 00 01 00 00 00-00 00 00 00 00 00 00 00
>>>> 0090 00 00 00 00 00 00 00 00-0a 01 00 68 0a 01 00 6d
>>>> 00c0 00 40 c0 5b 17 c1 00 00-00 00 00 00 00 00 00 10 <— mi
Re: [ovs-discuss] Restarting the network triggers the deletion of one ovs port
On 10/30/23 10:28, Liqi An wrote: > Hi , > Is there any update about this issue , It's been bothering me for over > two weeks,thx~ Hi. As you saw in the log, something is calling ovs-vsctl to remove the port from OVS: 2023-10-16T13:07:17.668420+08:00 cluster12-b ovs-vsctl: ovs|1|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port br-oam bond1 OVS is not doing that on its own. You need to find what is calling this command in order to fix the problem. Likely candidates are network-scripts, NetworkManager or something similar. Best regards, Ilya Maximets. > > //An > > -Original Message- > From: Liqi An > Sent: Wednesday, October 18, 2023 5:13 PM > To: Ilya Maximets ; ovs-discuss@openvswitch.org > Cc: Cheng Chi ; Jonas Yi ; > Yawei Lu ; IPW AQUA team (CBC) > > Subject: RE: [ovs-discuss] Restarting the network triggers the deletion of > one ovs port > > Hi , > We added bond1 to br-oam with below command before: > > # ovs-vsctl add-port br-oam bond1 trunk=3932,3933 > > In my opinion , , once it add bond1 to the virtual switch > (openvswtich) successfully , this configuration should be saved into ovs db , > What I'm wondering is why the network management on the host actively > triggers the deletion, That's not supposed to be the job of SUSE15-SP5 , The > ovs-vsctl command is used to manually add or delete these configurations > before. > > > By the way , I tried another way , delete bond1 configuration of > previous host , and add eth1 as bond1 to openvswitch directly: > WA: > cluster12-b:/etc/sysconfig/network # ip link set bond1 down > cluster12-b:/etc/sysconfig/network # rm -f /etc/sysconfig/network/ifcfg-bond1 > cluster12-b:/etc/sysconfig/network # service network restart > cluster12-b:/etc/sysconfig/network # ip link set eth1 up > cluster12-b:/etc/sysconfig/network # ip link set eth5 up > > cluster12-b:/etc/sysconfig/network # ovs-vsctl add-bond br-oam bond1 eth1 > eth5 trunk=3932,3933 cluster12-b:/etc/sysconfig/network # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > Bridge br-oam > Port br-oam > Interface br-oam > type: internal > Port bond1 > trunks: [3932, 3933] > Interface eth1 > Interface eth5 > ovs_version: "2.14.2" > cluster12-b:/etc/sysconfig/network # service network restart > cluster12-b:/etc/sysconfig/network # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > Bridge br-oam > Port br-oam > Interface br-oam > type: internal > Port bond1 > trunks: [3932, 3933] > Interface eth1 > Interface eth5 > ovs_version: "2.14.2" > > After initial testing, this scheme works temporarily and does not lose the > relevant network configuration after restarting the network and host ,But I > not sure how eth1 works in this configuration , I hope them work as > before: > > cluster12-b:~ # cat /etc/sysconfig/network/ifcfg-bond1 > DEVICE='bond1' > BORADCAST='' > NETWORK='' > USERCONTROL='no' > BONDING_SKIP_REMOVE_WORKAROUND='yes' > BOOTPROTO='static' > STARTMODE='auto' > ZONE='public' > BONDING_MASTER='yes' > BONDING_SLAVE0='eth5' > BONDING_SLAVE1='eth1' > BONDING_MODULE_OPTS='mode=active-backup miimon=100 use_carrier=0' > > do you have any idea about this solution or give some advice, thx ~ > > //An > > -Original Message- > From: Ilya Maximets > Sent: Tuesday, October 17, 2023 7:34 PM > To: Liqi An ; ovs-discuss@openvswitch.org > Cc: i.maxim...@ovn.org; Cheng Chi ; Jonas Yi > ; Yawei Lu > Subject: Re: [ovs-discuss] Restarting the network triggers the deletion of > one ovs port > > On 10/17/23 07:53, Liqi An wrote: >> Hi experts , >> I simplified the issue repetition steps , >> >> cluster12-b: # cat ovs-network.xml >> >> >> 2.11-ovs-network >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> cluster12-b: # virsh list --all >> Id Name State >> >> >> cluster12-b: # virsh net-list --all >> Name State Autostart Persistent >> >> >> cluster12-b: # virsh net-define ovs-network.xml Network >> 2.11-ovs-network defined from ovs-networ
Re: [ovs-discuss] ovsdb relay server active connection probe interval do not work
On 10/18/23 17:14, Vladislav Odintsov wrote: > Hi Ilya, Terry, > >> On 7 Mar 2023, at 14:03, Ilya Maximets wrote: >> >> On 3/7/23 00:15, Vladislav Odintsov wrote: >>> Hi Ilya, >>> >>> I’m wondering whether there are possible configuration parameters for ovsdb >>> relay -> main ovsdb server inactivity probe timer. >>> My cluster experiencing issues where relay disconnects from main cluster >>> due to 5 sec. inactivity probe timeout. >>> Main cluster has quite big database and a bunch of daemons, which connects >>> to it and it makes difficult to maintain connections in time. >>> >>> For ovsdb relay as a remote I use in-db configuration (to provide >>> inactivity probe and rbac configuration for ovn-controllers). >>> For ovsdb-server, which serves SB, I just set --remote=pssl:. >>> >>> I’d like to configure remote for ovsdb cluster via DB to set inactivity >>> probe setting, but I’m not sure about the correct way for that. >>> >>> For now I see only two options: >>> 1. Setup custom database scheme with connection table, serve it in same SB >>> cluster and specify this connection when start ovsdb sb server. >> >> There is a ovsdb/local-config.ovsschema shipped with OVS that can be >> used for that purpose. But you'll need to craft transactions for it >> manually with ovsdb-client. >> >> There is a control tool prepared by Terry: >> >> https://patchwork.ozlabs.org/project/openvswitch/patch/20220713030250.2634491-1-twil...@redhat.com/ >> >> But it's not in the repo yet (I need to get back to reviews on that >> topic at some point). The tool itself should be fine, but maybe name >> will change. > > I want to step back to this thread. > The mentioned patch is archived with "Changes Requested" state, but there is > no review comments in this patch. > If there is no ongoing work with it, I can take it over to finalise. > For now it needs a small rebase, so I can do it and resend, but before want > to hear your thoughts on this. > > Internally we use this patch to work with Local_Config DB for almost 6 months > and it works fine. > On each OVS update we have to re-apply it and sometimes solve conflicts, so > would be nice to have this patch in upstream. Hi, I'm currently in the middle of re-working the ovsdb-server configuration for a different approach that will replace command-line and appctl configs with a config file (cmdline and appctls will be preserved for backward compatibility, but there will be a new way of setting things up). This should be much more flexible and user-friendly than working with a local-config database. That should also address most of the concerns raised by Terry regarding usability of local-config (having way too many ways of configuring the same thing mainly, and requirement to use special tools to modify the configuration). I'm planning to post the first version of the change relatively soon. I can Cc you on the patches. Best regards, Ilya Maximets. > >> >>> 2. Setup second connection in ovn sb database to be used for ovsdb cluster >>> and deploy cluster separately from ovsdb relay, because they both start >>> same connections and conflict on ports. (I don’t use docker here, so I need >>> a separate server for that). >> >> That's an easy option available right now, true. If they are deployed >> on different nodes, you may even use the same connection record. >> >>> >>> Anyway, if I configure ovsdb remote for ovsdb cluster with specified >>> inactivity probe (say, to 60k), I guess it’s still not enough to have ovsdb >>> pings every 60 seconds. Inactivity probe must be the same from both ends - >>> right? From the ovsdb relay process. >> >> Inactivity probes don't need to be the same. They are separate for each >> side of a connection and so configured separately. >> >> You can set up inactivity probe for the server side of the connection via >> database. So, server will probe the relay every 60 seconds, but today >> it's not possible to set inactivity probe for the relay-to-server direction. >> So, relay will probe the server every 5 seconds. >> >> The way out from this situation is to allow configuration of relays via >> database as well, e.g. relay:db:Local_Config,Config,relays. This will >> require addition of a new table to the Local_Config database and allowing >> relay config to be parsed from the database in the code. That wasn't >> implemented yet. >> >>> I saw your talk on last ovscon about this topic, and the solutio
Re: [ovs-discuss] OVS & OVN HWOL with Nvidia ConnectX-6 Dx - Kernel flower acknowledgment does not match request
On 10/18/23 16:24, Vladislav Odintsov wrote:
> Hi Ilya,
>
> thanks for your response!
>
>> On 18 Oct 2023, at 15:59, Ilya Maximets via discuss
>> wrote:
>>
>> On 10/17/23 16:30, Vladislav Odintsov via discuss wrote:
>>> Hi,
>>>
>>> I’m testing OVS hardware offload with tc flower with Mellanox/NVidia
>>> ConnectX-6 Dx smartnic and see next warning in ovs-vswitchd log:
>>>
>>> 2023-10-17T14:23:15.116Z|00386|tc(handler20)|WARN|Kernel flower
>>> acknowledgment does not match request! Set dpif_netlink to dbg to see
>>> which rule caused this error.
>>>
>>> With dpif_netlink debug logs enabled, after this message appears two
>>> additional lines:
>>>
>>> 2023-10-17T14:23:15.117Z|00387|dpif_netlink(handler20)|DBG|added flow
>>> 2023-10-17T14:23:15.117Z|00388|dpif_netlink(handler20)|DBG|system@ovs-system:
>>> put[create] ufid:d8a3ab6d-77d1-4574-8bbf-634b01a116f3
>>> recirc_id(0),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0x10,src=10.1.0.105,dst=10.1.0.109,ttl=64/0,tp_src=59507/0,tp_dst=6081/0,geneve({class=0x102,type=0x80,len=4,0x60002}),flags(-df+csum+key)),in_port(4),skb_mark(0/0),ct_state(0/0x2f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x3),eth(src=00:00:ba:a4:6e:ad,dst=00:01:ba:a4:6e:ad),eth_type(0x0800),ipv4(src=172.32.2.4/0.0.0.0,dst=172.32.1.4/0.0.0.0,proto=1,tos=0/0x3,ttl=63/0,frag=no),icmp(type=8/0,code=0/0),
>>>
>>> actions:set(tunnel(tun_id=0xff0011,src=10.1.0.109,dst=10.1.1.18,ttl=64,tp_src=59507,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x18000b}),flags(df|csum|key))),4
>>>
>>
>> Could you also enable debug logs for 'tc' module in OVS?
>> It shoudl give more infomation about where exactly is the
>> difference between what OVS asked for and what the kenrel
>> reported back.
>>
>> In general this warning typically signifies a kernel bug,
>> but it could be that OVS doesn't format something correctly
>> as well.
>
> With enabled tc logs I see mismatches in expected/real keys and actions:
>
> 2023-10-18T13:33:35.882Z|00118|tc(handler21)|DBG|tc flower compare failed
> action compare
> Expected Mask:
> ff ff 00 00 ff ff ff ff-ff ff ff ff ff ff ff ff
> 0030 00 00 2f 00 00 00 00 00-00 00 00 00 00 00 00 00
> 0040 03 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 0050 00 00 00 00 ff ff ff ff-00 00 00 00 00 00 00 00
> 0060 00 00 00 00 ff 00 00 00-00 00 00 00 00 00 00 00
> 0090 00 00 00 00 00 00 00 00-ff ff ff ff ff ff ff ff
> 00c0 ff 00 00 00 ff ff 00 00-ff ff ff ff ff ff ff ff
> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 00e0 ff ff ff 01 ff ff ff ff-00 00 00 00 00 00 00 00
>
> Received Mask:
> ff ff 00 00 ff ff ff ff-ff ff ff ff ff ff ff ff
> 0030 00 00 2f 00 00 00 00 00-00 00 00 00 00 00 00 00
> 0040 03 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 0050 00 00 00 00 ff ff ff ff-00 00 00 00 00 00 00 00
> 0060 00 00 00 00 ff 00 00 00-00 00 00 00 00 00 00 00
> 0090 00 00 00 00 00 00 00 00-ff ff ff ff ff ff ff ff
> 00c0 ff 00 00 00 ff ff 00 00-ff ff ff ff ff ff ff ff
> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 00e0 ff ff ff 01 ff ff ff ff-00 00 00 00 00 00 00 00
>
> Expected Key:
> 08 06 00 00 ff ff ff ff-ff ff 00 00 ba a4 6e ad
> 0050 a9 fe 64 01 a9 fe 64 03-00 00 ba a4 6e ad 00 00 <— mismatch in
> this line
> 0060 00 00 00 00 01 00 00 00-00 00 00 00 00 00 00 00
> 0090 00 00 00 00 00 00 00 00-0a 01 00 68 0a 01 00 6d
> 00c0 00 40 c0 5b 17 c1 00 00-00 00 00 00 00 00 00 10 <— mismatch in
> this line
> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 00e0 01 02 80 01 00 03 00 02-00 00 00 00 00 00 00 00
>
> Received Key:
> 08 06 00 00 ff ff ff ff-ff ff 00 00 ba a4 6e ad
> 0050 00 00 00 00 a9 fe 64 03-00 00 00 00 00 00 00 00 <— mismatch in
> this line
> 0060 00 00 00 00 01 00 00 00-00 00 00 00 00 00 00 00
> 0090 00 00 00 00 00 00 00 00-0a 01 00 68 0a 01 00 6d
> 00c0 00 00 00 00 17 c1 00 00-00 00 00 00 00 00 00 10 <— mismatch in
> this line
> 00d0 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> 00e0 01 02 80 01 00 03 00 02-00 00 00 00 00 00 00 00
These are not very important, it is expected that the kernel clears out
fields that are not coverd by a mask. We do not have the difference
in the masks and we do not have a diference in the masked keys, so that
is fine.
>
> Expected Masked Key:
> 08 06 00 00 ff ff ff ff-ff ff 00 00 ba a4 6e ad
> 0050 00 00 00 00 a9 fe 64 03-00 00 00 00 00 00 00 00
> 0060 00 00 00 00 01 00 00 00-00 00 00 0
Re: [ovs-discuss] OVS & OVN HWOL with Nvidia ConnectX-6 Dx - Kernel flower acknowledgment does not match request
On 10/17/23 16:30, Vladislav Odintsov via discuss wrote:
> Hi,
>
> I’m testing OVS hardware offload with tc flower with Mellanox/NVidia
> ConnectX-6 Dx smartnic and see next warning in ovs-vswitchd log:
>
> 2023-10-17T14:23:15.116Z|00386|tc(handler20)|WARN|Kernel flower
> acknowledgment does not match request! Set dpif_netlink to dbg to see which
> rule caused this error.
>
> With dpif_netlink debug logs enabled, after this message appears two
> additional lines:
>
> 2023-10-17T14:23:15.117Z|00387|dpif_netlink(handler20)|DBG|added flow
> 2023-10-17T14:23:15.117Z|00388|dpif_netlink(handler20)|DBG|system@ovs-system:
> put[create] ufid:d8a3ab6d-77d1-4574-8bbf-634b01a116f3
> recirc_id(0),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0x10,src=10.1.0.105,dst=10.1.0.109,ttl=64/0,tp_src=59507/0,tp_dst=6081/0,geneve({class=0x102,type=0x80,len=4,0x60002}),flags(-df+csum+key)),in_port(4),skb_mark(0/0),ct_state(0/0x2f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x3),eth(src=00:00:ba:a4:6e:ad,dst=00:01:ba:a4:6e:ad),eth_type(0x0800),ipv4(src=172.32.2.4/0.0.0.0,dst=172.32.1.4/0.0.0.0,proto=1,tos=0/0x3,ttl=63/0,frag=no),icmp(type=8/0,code=0/0),
>
> actions:set(tunnel(tun_id=0xff0011,src=10.1.0.109,dst=10.1.1.18,ttl=64,tp_src=59507,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x18000b}),flags(df|csum|key))),4
>
Could you also enable debug logs for 'tc' module in OVS?
It shoudl give more infomation about where exactly is the
difference between what OVS asked for and what the kenrel
reported back.
In general this warning typically signifies a kernel bug,
but it could be that OVS doesn't format something correctly
as well.
> The test system is a CentOS 8.4 with installed elrepo mainline kernel 6.5.5,
> OVS 3.1.1 and OVN 22.09.1.
3.1.1 contains some known bugs in TC offloading code, so
you may want to try the latest 3.1.3. Though it's unlikely
to be related ot the issue you're facing here.
> The workload I’m testing is a L3 Gateway for OVN IC (cross-az traffic).
>
> tc monitor at the same moment outputs next:
>
> replaced filter dev genev_sys_6081 ingress protocol ip pref 2 flower chain 0
> handle 0x3
> dst_mac 00:01:ba:a4:6e:ad
> src_mac 00:00:ba:a4:6e:ad
> eth_type ipv4
> ip_proto icmp
> ip_tos 0/0x3
> enc_dst_ip 10.1.0.109
> enc_src_ip 10.1.0.105
> enc_key_id 16
> enc_dst_port 6081
> enc_tos 0
> geneve_opts 0102:80:00060002/:ff:
> ip_flags nofrag
> ct_state -trk-new-est
> ct_label /03
> in_hw in_hw_count 2
> action order 1: tunnel_key unset pipe
> index 5 ref 1 bind 1
> no_percpu
> used_hw_stats delayed
>
> action order 2: tunnel_key set
> src_ip 10.1.0.109
> dst_ip 10.1.1.18
> key_id 16711697
> dst_port 6081
> geneve_opts 0102:80:0018000b
> csum
> ttl 64 pipe
> index 6 ref 1 bind 1
> no_percpu
> used_hw_stats delayed
>
> action order 3: mirred (Egress Redirect to device genev_sys_6081) stolen
> index 3 ref 1 bind 1
> cookie 6daba3d87445d1774b63bf8bf316a101
> no_percpu
> used_hw_stats delayed
>
>
> Despite of these warnings, the flow is finally offloaded and the traffic
> traverses this gw node well, only first packets of an ICMP sequence reach CPU
> (seen in tcpdump):
The warning is a warning. It doesn't prevent the flow to be installed.
Though the installed flow may be incorrect and the traffic may be
handled in the wrong way. Enabling debug logs for tc should show what
exacltly is wrong.
>
> # ovs-appctl dpctl/dump-flows type=offloaded
> tunnel(tun_id=0x10,src=10.1.0.107,dst=10.1.0.109,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x50002}),flags(+key)),ct_state(-new-est-rel-rpl-trk),ct_label(0/0x3),recirc_id(0),in_port(4),eth(src=00:00:ba:a4:6e:ad,dst=00:01:ba:a4:6e:ad),eth_type(0x0800),ipv4(proto=1,tos=0/0x3,frag=no),
> packets:3192, bytes:312816, used:1.240s,
> actions:set(tunnel(tun_id=0xff0011,src=10.1.0.109,dst=10.1.1.18,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x18000b}),flags(csum|key))),4
> tunnel(tun_id=0xff0011,src=10.1.1.18,dst=10.1.0.109,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0xb0018}),flags(+key)),ct_state(-new-est-rel-rpl-trk),ct_label(0/0x3),recirc_id(0),in_port(4),eth(src=00:01:ba:a4:6e:ad,dst=00:00:ba:a4:6e:ad),eth_type(0x0800),ipv4(src=172.32.1.0/255.255.255.0,dst=172.32.0.4,proto=1,tos=0/0x3,ttl=63,frag=no),
> packets:3192, bytes:312816, used:1.240s,
> actions:set(tunnel(tun_id=0x11,src=10.1.0.109,dst=10.1.0.107,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x10002}),flags(csum|key))),set(eth(src=d0:fe:00:00:00:1d,dst=0a:00:66:ec:f7:40)),set(ipv4(ttl=62)),4
> tunnel(tun_id=0x10,src=10.1.0.105,dst=10.1.0.109,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x60002}),flags(+key)),ct_state(-new-est-rel-rpl-trk),ct_label(0/0x3),recirc_id(0),in_port(4),eth(src=00:00:ba:a4:6e:ad,dst=00:01:ba:a4:6e:ad),eth_type(0x0800),ipv4(proto=1,tos=0/0x3,frag=no),
> packets:293, bytes:28714, used:1.240s,
>
Re: [ovs-discuss] Restarting the network triggers the deletion of one ovs port
On 10/17/23 07:53, Liqi An wrote: > Hi experts , > I simplified the issue repetition steps , > > cluster12-b: # cat ovs-network.xml > > > 2.11-ovs-network > > > > > > > > > > > > > > > > > > > > > > > cluster12-b: # virsh list --all > Id Name State > > > cluster12-b: # virsh net-list --all > Name State Autostart Persistent > > > cluster12-b: # virsh net-define ovs-network.xml > Network 2.11-ovs-network defined from ovs-network.xml > > cluster12-b: # virsh net-list --all > Name State Autostart Persistent > --- > 2.11-ovs-network inactive no yes > > cluster12-b: # virsh net-start 2.11-ovs-network > Network 2.11-ovs-network started > > cluster12-b: # virsh net-list --all > Name StateAutostart Persistent > - > 2.11-ovs-network active no yes > > cluster12-b: # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > ovs_version: "2.14.2" > cluster12-b: # ovs-vsctl add-br br-oam > cluster12-b: # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > Bridge br-oam > Port br-oam > Interface br-oam > type: internal > ovs_version: "2.14.2" > cluster12-b: # ovs-vsctl add-port br-oam bond1 trunk=3932,3933 > cluster12-b: # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > Bridge br-oam > Port br-oam > Interface br-oam > type: internal > Port bond1 > trunks: [3932, 3933] > Interface bond1 > ovs_version: "2.14.2" > cluster12-b: # date > Tue Oct 17 13:47:02 CST 2023 > cluster12-b: # service network restart > cluster12-b: # ovs-vsctl show > 2e9bf291-50ac-4c3a-ac55-2d590df1880d > Bridge br-oam > Port br-oam > Interface br-oam > type: internal > ovs_version: "2.14.2" > cluster12-b: # > > it seems like a common issue . You have in your log "kernel: [340557.486796][ T2447] bonding: bond1 is being deleted", it means something is destroying a bonding device in the kernel. And it makes some sense because you're restarting the network. Does this bond1 interface exist after the network restart? What creates it back? network-scripts? If so, you should check these network-scripts. And the port-add command should be somewhere in the same network scripts. And the bridge should likely be created there. Otherwise, the configuration you created is not going to be persistent. > > //An > > -Original Message- > From: Ilya Maximets > Sent: Monday, October 16, 2023 6:54 PM > To: Liqi An ; ovs-discuss@openvswitch.org > Cc: Cheng Chi ; Jonas Yi ; > Yawei Lu ; i.maxim...@ovn.org > Subject: Re: [ovs-discuss] Restarting the network triggers the deletion of > one ovs port > > On 10/16/23 07:38, Liqi An via discuss wrote: >> Hi experts, >> >> I am having a very strange problem with matching virtual machines >> installations with openvswitch. My operating system is suse15-sp4; >> >> cluster12-b:~ # cat /etc/os-release >> /NAME="SLES"/ >> /VERSION="15-SP4"/ >> /VERSION_ID="15.4"/ >> /PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"/ /ID="sles"/ >> /ID_LIKE="suse"/ /ANSI_COLOR="0;32"/ >> /CPE_NAME="cpe:/o:suse:sles:15:sp4"/ >> /DOCUMENTATION_URL=https://documentation.suse.com/ >> <https://documentation.suse.com/>/ >> >> cluster12-b:~ # rpm -qa |grep openvswitch >> /openvswitch-2.14.2-150400.22.23.x86_64/ >> >> cluster12-b:~ # virsh net-list --all >> >> /Name State Autostart Persistent/ >> /-/ >> /2.11-ovs-network active yes yes/ >> >> bond1 was used by the VMs: >> … >> Bridge br-oam >> Port bond1 >> trunks: [3932, 3933] >> Interface bond1 >> Port "2.11-SC-2-eth1" >> tag: 3932 >> Interface "2.11-SC-2-eth1" >> Port br-oam >> Interface br-oam >> type: internal >>
Re: [ovs-discuss] [ovs-dpdk] Performance drops after 3-4 minutes
On 10/6/23 20:10, Алексей Кашавкин via discuss wrote: > Hello! > > I am using OVS with DPDK in OpenStack. This is RDO+TripleO deployment with > the Train release. I am trying to measure the performance of the DPDK compute > node. I have created two VMs [1], one as a DUT with DPDK and one as a traffic > generator with SR-IOV [2]. Both of them are using Pktgen. > > What happens is the following: for the first 3-4 minutes I see 2.6Gbit [3] > reception in DUT, after that the speed always drops to 400Mbit [4]. At the > same time in the output of `pmd-rxq-show` command I always see one of the > interfaces in the bond loaded [5], but it happens that after flapping of the > active interface the speed in DUT increases up to 5Gbit and in the output of > `pmd-rxq-show` command I start to see the load on two interfaces [6]. But at > the same time after 3-4 minutes the speed drops to 700Mbit and I continue to > see the same load on the two interfaces in the bond in the `pmd-rxq-show` > command. In the logs I see nothing but flapping [7] of the interfaces in bond > and the flapping has no effect on the speed drop after 3-4 minutes of test. > After the speed drop from the DUT itself I run traffic towards the traffic > generator [8] for a while and stop, then the speed on the DUT is restored to > 2.6Gbit again with traffic going through one interface or 5Gbit with traffic > going through two interfaces, but this again is only for 3-4 minutes. If I do > a test with a traffic generator with a 2.5 Gbit or 1 Gbit speed limit, the > speed also drops to DUT after 4-5 minutes. I've put logging in debug for > bond, dpdk, netdev_dpdk, dpif_netdev, but haven't seen anything that > clarifies what's going on, and also it's not clear that sometimes after > flapping the active interface traffic starts going through both interfaces in > bond, but this happens rarely, not in every test. Since rate is restored after you sending some traffic in the backward direction, I'd say you have MAC learning somewhere on the path and it is getting expired. For example, if you use NORMAL action in one of the bridges, once the MAC is expired, the bridge will start flooding packets to all ports of the bridge, which is very slow. You may look at datapath flow dump to confirm which actions are getting executed on your packets: ovs-appctl dpctl/dump-flows. In general, you should always continuously send some traffic back for learned MAC addresses to not expire. I'm not sure if Pktgen is doing that these days, but it wasn't a very robust piece of software in the past. > > [4] The flapping of the interface through which traffic is going to the DUT > VM is probably due to the fact that it is heavily loaded alone in the bond > and there are no LACP PDU packets going to or from it. The log shows that it > is down for 30 seconds because the LACP rate is set to slow mode. Dropped LACP packets can cause bond flapping indeed. The only way to fix that in older versions of OVS is to reduce the load. With OVS 3.2 you may try experimental 'rx-steering' configuration that was designed exactly for this scenario and should ensure that PDU packets are not dropped. Also, balancing depends on packet hashes, so you need to send many different traffic flows in order to get consistent balancing. > > I have done DUT on different OS, with different versions of DPDK and Pktgen. > But always the same thing happens, after 3-4 minutes the speed drops. > Only on the DPDK compute node I didn't change anything. The compute node has > Intel E810 network card with 25Gbit ports and Intel Xeon Gold 6230R CPU. The > PMD threads uses cores 11, 21, 63, 73 on numa 0 and 36, 44, 88, 96 on numa 1. All in all, 2.6Gbps seems like a small number for the type of a system you have. You might have some other configuration issues. > > In addition: > [9] ovs-vsctl show > [10] OVSDB dump > [11] pmd-stats-show > [12] bond info with ovs-appctl > > For compute nodes, I use Rocky Linux 8.5, Open vSwitch 2.15.5, and DPDK > 20.11.1. FWIW, OVS 2.15 reached EOL ~1.5 years ago. Best regards, Ilya Maximets. > > > What could be the cause of this behavior? I don't understand where I should > look to find out exactly what is going on. > > > 1. https://that.guru/blog/pktgen-between-two-openstack-guests > <https://that.guru/blog/pktgen-between-two-openstack-guests> > 2. https://freeimage.host/i/J206p8Q <https://freeimage.host/i/J206p8Q> > 3. https://freeimage.host/i/J20Po9p <https://freeimage.host/i/J20Po9p> > 4. https://freeimage.host/i/J20PRPs <https://freeimage.host/i/J20PRPs> > 5. https://pastebin.com/rpaggexZ <https://pastebin.com/rpaggexZ> > 6. https://pastebin.com/Zhm779vT <https://pastebin.com/Zhm779vT> > 7. https://pastebin.com/Vt5P
Re: [ovs-discuss] Restarting the network triggers the deletion of one ovs port
On 10/16/23 07:38, Liqi An via discuss wrote: > Hi experts, > > I am having a very strange problem with matching virtual machines > installations with openvswitch. My operating system is suse15-sp4; > > cluster12-b:~ # cat /etc/os-release > /NAME="SLES"/ > /VERSION="15-SP4"/ > /VERSION_ID="15.4"/ > /PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"/ > /ID="sles"/ > /ID_LIKE="suse"/ > /ANSI_COLOR="0;32"/ > /CPE_NAME="cpe:/o:suse:sles:15:sp4"/ > /DOCUMENTATION_URL=https://documentation.suse.com/ > <https://documentation.suse.com/>/ > > cluster12-b:~ # rpm -qa |grep openvswitch > /openvswitch-2.14.2-150400.22.23.x86_64/ > > cluster12-b:~ # virsh net-list --all > > /Name State Autostart Persistent/ > /-/ > /2.11-ovs-network active yes yes/ > > bond1 was used by the VMs: > … > Bridge br-oam > Port bond1 > trunks: [3932, 3933] > Interface bond1 > Port "2.11-SC-2-eth1" > tag: 3932 > Interface "2.11-SC-2-eth1" > Port br-oam > Interface br-oam > type: internal > Port "2.11-SC-2-eth2" > tag: 3933 > Interface "2.11-SC-2-eth2" > > But when I restarted the network service by command: # service network > restart , this port bond1 lost in the bridge br-oam , > and there are some abnormal log in systemlog, Detailed operation logs are > attached > > … > /25302 2023-10-16T13:07:12.708071+08:00 cluster12-b kernel: [340552.475586][ > T2447] device eth1 left promiscuous mode/ > /25303 2023-10-16T13:07:12.824022+08:00 cluster12-b kernel: [340552.593298][ > T2447] bonding: bond0 is being deleted.../ > /25304 2023-10-16T13:07:12.824045+08:00 cluster12-b kernel: [340552.593393][ > T2447] bond0 (unregistering): Released all slaves/ > /25305 2023-10-16T13:07:12.881576+08:00 cluster12-b systemd[1]: Starting > Generate issue file for login session.../ > /25306 2023-10-16T13:07:12.905589+08:00 cluster12-b systemd[1]: > issue-generator.service: Deactivated successfully./ > /25307 2023-10-16T13:07:12.905662+08:00 cluster12-b systemd[1]: Finished > Generate issue file for login session./ > /25308 2023-10-16T13:07:17.668420+08:00 cluster12-b ovs-vsctl: > ovs|1|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port br-oam bond1/ > /25309 2023-10-16T13:07:17.676015+08:00 cluster12-b kernel: [340557.444150][ > T2261] device bond1 left promiscuous mode/ > /25310 2023-10-16T13:07:17.720080+08:00 cluster12-b kernel: [340557.486796][ > T2447] bonding: bond1 is being deleted.../ > /25311 2023-10-16T13:07:17.720097+08:00 cluster12-b kernel: [340557.486891][ > T2447] bond1 (unregistering): Released all slaves/ IIUC, the 'bond1' is some sort of a kernel bonding device configured outside of OVS. And it is getting removed. When you restart the network, the system will execute whatever network configuration is in your system settings, e.g. stuff from /etc/sysconfig/network-scripts, maybe NetworkManager is going to re-apply its configuration or netplan, I don't really know what SUSE is using. So, you should look in these places for things that manage the bond1 interface. Best regards, Ilya Maximets. > > It seemed that Restarting the host's network service automatically triggered > behavior: /as /usr/bin/ovs-vsctl del-port br-oam bond1/ > > Also, I restart host which causes the same issue, would you please help check > and give some advice, thx~ > > //An ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Bonding configuration for KVM guests
On 10/13/23 14:22, Xuo Guoto via discuss wrote: > --- Original Message --- > On Friday, October 13th, 2023 at 4:51 PM, Ilya Maximets > wrote: > >> I'd suggest you disable the lacp-fallback-ab, so the bond doesn't >> fall back to active-backup mode on failures. At least while you're >> troubleshooting. > > Executed > > # ovs-vsctl set port bond0 bond_mode=balance-tcp lacp=active > other-config:lacp-fallback-ab=false other-config:lacp-time=slow > > Now all the interfaces are disabled This is expected. They were enabled before, because of ab fallback. > > bond0 > bond_mode: balance-tcp > bond may use recirculation: yes, Recirc-ID : 1 > bond-hash-basis: 0 > lb_output action: disabled, bond-id: -1 > updelay: 0 ms > downdelay: 0 ms > next rebalance: 9560 ms > lacp_status: configured > lacp_fallback_ab: false > active-backup primary: > active member mac: 00:00:00:00:00:00(none) > > member vnet74: disabled > may_enable: false > > member vnet75: disabled > may_enable: false > > member vnet76: disabled > may_enable: false > > member vnet77: disabled > may_enable: false > > member vnet78: disabled > may_enable: false > > member vnet79: disabled > may_enable: false > > >> Also, how does the flow table look like? i.e. what's in the output >> of 'ovs-ofctl dump-flows ovsbr-lacp0' ? In case there are rules that >> would drop LACP traffic. > > # ovs-ofctl dump-flows ovsbr-lacp0 > cookie=0x0, duration=103674.235s, table=0, n_packets=6798, n_bytes=2111314, > priority=0 actions=NORMAL > > I guess this is fine. Yep. Looks fine. > > I did some more digging and when running tcpdump on both the ovs and on > Ubuntu I get this packet repeatedly at both interfaces: > > 12:23:17.081894 LACPv1, length 110 > Actor Information TLV (0x01), length 20 > System a6:a0:01:43:3a:41, System Priority 65534, Key 15, Port 17, > Port Priority 65535 > State Flags [Activity, Aggregation, Collecting, Distributing, > Default] > Partner Information TLV (0x02), length 20 > System 00:00:00:00:00:00, System Priority 0, Key 0, Port 0, Port > Priority 0 > State Flags [none] > Collector Information TLV (0x03), length 16 > Max Delay 0 > Terminator TLV (0x00), length 0 > > I am assuming this is a packet originated from a6:a0:01:43:3a:41 which > happens to be the MAC of ovsbr-lacp0 > > # ovs-vsctl -f table --columns=name,mac_in_use list interface | grep > a6:a0:01:43:3a:41 > ovsbr-lacp0 "a6:a0:01:43:3a:41" > > Am I correct in assuming that ovs is sending LACP packets correctly while > Ubuntu is not responding and not sending its own LACP packets? Yes, looks like it. I'm not an expert in configuring native linux bonds, so I'm not sure how to troubleshoot that. Are the ports including the bond port UP in the guest? Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Bonding configuration for KVM guests
On 10/13/23 12:55, Xuo Guoto wrote: > > --- Original Message --- > On Friday, October 13th, 2023 at 3:51 PM, Ilya Maximets > wrote: > > Thanks again for taking time to answer my questions! > > >> Here is your problem. The bond is in active-backup mode. This mode doesn't >> support LACP. You need to switch to the balanced mode, e.g. by adding >> 'bond_mode=balance-tcp' to the bond creation command. Or you may set it after >> the fact with: >> >> ovs-vsctl set port bond0 bond_mode=balance-tcp > > After some trial and error I got this command from Redhat bugzilla BZ#1267291 > > ovs-vsctl set port bond0 bond_mode=balance-tcp lacp=active > other-config:lacp-fallback-ab=true other-config:lacp-time=fast > > Now the output is > > ovs-appctl bond/show bond0 > bond0 > bond_mode: balance-tcp > bond may use recirculation: no, Recirc-ID : -1 > bond-hash-basis: 0 > lb_output action: disabled, bond-id: -1 > updelay: 0 ms > downdelay: 0 ms > lacp_status: configured > lacp_fallback_ab: true > active-backup primary: > active member mac: fe:54:00:79:8e:a0(vnet77) > > member vnet74: enabled > may_enable: true > > member vnet75: enabled > may_enable: true > > member vnet76: enabled > may_enable: true > > member vnet77: enabled > active member > may_enable: true > > member vnet78: enabled > may_enable: true > > member vnet79: enabled > may_enable: true > > I can see that all member interfaces are enabled now, and I get an active > member mac. But still there is no ping between two devices in the switch. > Also in linux I can see this error bond0: Warning: No 802.3ad response from > the link partner for any adapters in the bond > > Could lb_output action: disabled, bond-id: -1 be the reason the bond0 is not > getting up? No, it's just an optimization and kernel datapath doesn't support it. So, should not affect the actual traffic. You have an 'lacp_status: configured', meaning the negotiation with the other side didn't work out. It should be in 'negotiated' state if everything is working. I'd suggest you disable the lacp-fallback-ab, so the bond doesn't fall back to active-backup mode on failures. At least while you're troubleshooting. And you probably better use slow lacp-time as well. Also, how does the flow table look like? i.e. what's in the output of 'ovs-ofctl dump-flows ovsbr-lacp0' ? In case there are rules that would drop LACP traffic. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Bonding configuration for KVM guests
On 10/13/23 07:22, Xuo Guoto via discuss wrote:
> --- Original Message ---
> On Thursday, October 12th, 2023 at 9:03 PM, Ilya Maximets
> wrote:
>
> Thanks again for the detailed reply!
>
>> If you want to preserve these, you'll need to re-add them manually.
>> Potentially as part of the same transaction, by adding more commands,
>> i.e.
>>
>> ovs-vsctl \
>> del-port vnet75 \
>> -- del-port vnet78 \
>
> I had followed your advice and used the following command to create the bond
>
> ovs-vsctl \
> del-port vnet75 \
>-- del-port vnet78 \
>-- del-port vnet79 \
>-- del-port vnet77 \
>-- del-port vnet74 \
>-- del-port vnet76 \
>-- add-bond ovsbr-lacp0 bond0 vnet75 vnet78 vnet79 vnet77 vnet74 vnet76
> lacp=active \
>-- set Interface vnet75 \
> external_ids='{attached-mac="52:54:00:0d:61:fc", iface-id="",
> iface-status=active, vm-id=""}' \
>-- set Interface vnet78 \
> external_ids='{attached-mac="52:54:00:24:36:a7", iface-id="",
> iface-status=active, vm-id=""}' \
>-- set Interface vnet79 \
> external_ids='{attached-mac="52:54:00:c7:90:3c", iface-id="",
> iface-status=active, vm-id=""}' \
>-- set Interface vnet77 \
> external_ids='{attached-mac="52:54:00:79:8e:a0", iface-id="",
> iface-status=active, vm-id=""}' \
>-- set Interface vnet74 \
> external_ids='{attached-mac="52:54:00:aa:fb:bb", iface-id="",
> iface-status=active, vm-id=""}' \
>-- set Interface vnet76 \
> external_ids='{attached-mac="52:54:00:d4:cc:88", iface-id="",
> iface-status=active, vm-id=""}'
>
> Note: iface-id and vm-id values have been removed for brevity.
>
> Now the command was successful and I could see the bond0 port with the
> specified interfaces
>
> Bridge ovsbr-lacp0
> Port bond0
> Interface vnet78
> Interface vnet75
> Interface vnet79
> Interface vnet74
> Interface vnet77
> Interface vnet76
> Port ovsbr-lacp0
> Interface ovsbr-lacp0
> type: internal
> Port vnet72
> Interface vnet72
>
> But the bonding is not working correctly
>
> $ sudo ovs-appctl bond/show bond0
> bond0
> bond_mode: active-backup
Here is your problem. The bond is in active-backup mode. This mode doesn't
support LACP. You need to switch to the balanced mode, e.g. by adding
'bond_mode=balance-tcp' to the bond creation command. Or you may set it after
the fact with:
ovs-vsctl set port bond0 bond_mode=balance-tcp
Best regards, Ilya Maximets.
> bond may use recirculation: no, Recirc-ID : -1
> bond-hash-basis: 0
> lb_output action: disabled, bond-id: -1
> updelay: 0 ms
> downdelay: 0 ms
> lacp_status: configured
> lacp_fallback_ab: false
> active-backup primary:
> active member mac: 00:00:00:00:00:00(none)
>
> member vnet74: disabled
> may_enable: false
>
> member vnet75: disabled
> may_enable: false
>
> member vnet76: disabled
> may_enable: false
>
> member vnet77: disabled
> may_enable: false
>
> member vnet78: disabled
> may_enable: false
>
> member vnet79: disabled
> may_enable: false
>
> I have configured bonding on an Ubuntu 22.04 at the other end with following
> configuration:
>
> bonds:
> bond0:
> addresses: [192.168.201.141/24]
> interfaces:
> - enp2s0
> - enp3s0
> - enp4s0
> - enp5s0
> - enp6s0
> - enp7s0
> parameters:
> mode: 802.3ad
> mii-monitor-interval: 100
>
> And I get this error in dmesg: bond0: Warning: No 802.3ad response from the
> link partner for any adapters in the bond
>
> In Ubuntu the bond mode is 802.3ad, while in ovs its shows as active-backup
> could that be the reason for bond not getting online?
>
> I am kind of stuck here!
You can see the
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Bonding configuration for KVM guests
On 10/12/23 15:39, Xuo Guoto via discuss wrote:
> Thanks for your reply!
>
> --- Original Message ---
> On Thursday, October 12th, 2023 at 6:48 PM, Ilya Maximets
> wrote:
>
>
>> You currently have port vnet75 with an interface vnet75, and the
>> port vnet78 with interface vnet78.
>
> The differentiation between port and interface was new to me! Thanks!
'ovs-vsctl show' command makes the structure a little easier to see.
>
>> But you want a port bond0 with interfaces
>> vnet75 and vnet78. In order to achieve
>> that you need to remove ports vnet75 and
>> vnet78 first, and then create a port
>> bond0.
>
> How can I remove ports vnet75 and vnet78 without touching the underlying
> interface? I have the VM1 running and is connected to the bridge, can the
> ports be removed without disturbing the VM?
It might be possible by manually crafting OVSDB transactions, but
it's very prone to errors and I would not recommend that.
I'd recommend removing ports and creating a bond in a single
transaction, by chaining operations with '--':
ovs-vsctl del-port vnet75 \
-- del-port vnet78 \
-- add-bond ovsbr-lacp0 bond0 vnet75 vnet78 lacp=active
The problem with this though is that you will loose configuration
applied to the interfaces while removing them anyway. For example,
these interfaces will no longer have their external_ids.
If you want to preserve these, you'll need to re-add them manually.
Potentially as part of the same transaction, by adding more commands,
i.e.
ovs-vsctl \
del-port vnet75 \
-- del-port vnet78 \
-- add-bond ovsbr-lacp0 bond0 vnet75 vnet78 lacp=active \
-- set Interface vnet75 \
external_ids='{attached-mac="52:54:00:0d:61:fc", iface-id="..",
iface-status=active, vm-id=".."}' \
some_other_column= \
-- set Interface vnet78
external_ids='{attached-mac="52:54:00:24:36:a7", iface-id="..",
iface-status=active, vm-id=".."}' \
...
You may still have some slight network disruption though.
However, since you're enabling LACP you will have disruption anyway
until you bond these interfaces and enable LACP inside VM as well.
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Bonding configuration for KVM guests
On 10/12/23 14:58, Xuo Guoto via discuss wrote:
> Hello all,
>
> I am trying to bond two interfaces connecting my running VM with OVS.
>
> I have a bridge ovsbr-lacp0 with 7 virtio interfaces connected to two VMs, 6
> in VM1 and one in VM2.
>
> # ovs-vsctl list-ifaces ovsbr-lacp0
> vnet72
> vnet74
> vnet75
> vnet76
> vnet77
> vnet78
> vnet79
>
> # ovs-vsctl -f table --columns=name,external_id list interface | grep
> 274ce169-b166-45dc-ba1d-7faa8fea3673
> vnet75 {attached-mac="52:54:00:0d:61:fc", iface-id="..",
> iface-status=active, vm-id=".."}
> vnet78 {attached-mac="52:54:00:24:36:a7", iface-id="..",
> iface-status=active, vm-id=".."}
> vnet79 {attached-mac="52:54:00:c7:90:3c", iface-id="..",
> iface-status=active, vm-id=".."}
> vnet77 {attached-mac="52:54:00:79:8e:a0", iface-id="..",
> iface-status=active, vm-id=".."}
> vnet74 {attached-mac="52:54:00:aa:fb:bb", iface-id="..",
> iface-status=active, vm-id=".."}
> vnet76 {attached-mac="52:54:00:d4:cc:88", iface-id="..",
> iface-status=active, vm-id=".."}
>
> Where 274ce169-b166-45dc-ba1d-7faa8fea3673 is the UUID of VM1
>
> # sudo ovs-vsctl -f table --columns=name,external_id list interface | grep
> 2075bff5-ee89-492e-b50c-36bd3b92b662
> vnet72 {attached-mac="52:54:00:fd:fe:f6", iface-id="..",
> iface-status=active, vm-id=".."}
>
> and 2075bff5-ee89-492e-b50c-36bd3b92b662 is the UUID of VM2. I have removed
> the iface-id and vm-id so that the output is easy to read.
>
> Now I want to configure bonding for all the interface in VM1.
>
> # ovs-vsctl add-bond ovsbr-lacp0 bond0 vnet75 vnet78 lacp=active
> ovs-vsctl: cannot create an interface named vnet75 because a port named
> vnet75 already exists on bridge ovsbr-lacp0
>
> # ovs-vsctl --may-exist add-bond ovsbr-lacp0 bond0 vnet75 vnet78 lacp=active
> ovs-vsctl: cannot create an interface named vnet75 because a port named
> vnet75 already exists on bridge ovsbr-lacp0
>
> I am guessing I am doing some thing wrong, but not able to figure out what.
> Any help to get this working will be much appreciated.
'add-bond' command creates a 'port' with two or more 'interfaces'.
These interfaces are created as part of the port creation. So,
"add-bond ovsbr-lacp0 bond0 vnet75 vnet78" creates a 'port' named
bond0, creates two interfaces vnet75 and vnet78, and adds them into
a port. But interfaces vnet75 and vnet78 already exist, so the
command fails. The 'may-exist' flag only check for existence of
the port, not interfaces in it.
You currently have port vnet75 with an interface vnet75, and the
port vnet78 with interface vnet78. But you want a port bond0
with interfaces vnet75 and vnet78. In order to achieve that you
need to remove ports vnet75 and vnet78 first, and then create
a port bond0.
HTH
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OvS-DPDK compilation fails on NVidia Bluefield-2
On 9/21/23 11:32, Levente Csikor wrote:
> Thanks Ilya for the response.
>
> It turned out that the latest DPDK I can manually compile on the
> Bluefield-2 without errors is v20.08.
>
> After looking at the table you pointed me, my best option was
> OvS 2.14.9 with DPDK-v19.11.13.
>
> Compilations went well without error, now I have the same problem when
> attaching the ports to OvS. In fact, even attaching the physical port
> fails.
>
> The way how I initiate OvS is using my custom tailored script
> https://github.com/cslev/nvidia-bluefield-ovs-scripts/blob/main/start_ovs.sh
> From line 223.
>
>
> After adding the physical port, ovs-vsctl show provides this:
> ```
> Bridge ovs_dpdk_br0
> datapath_type: netdev
> Port dpdk0
> Interface dpdk0
> type: dpdk
> options: {dpdk-devargs=":03:00.0"}
> error: "Error attaching device ':03:00.0' to DPDK"
> Port ovs_dpdk_br0
> Interface ovs_dpdk_br0
> type: internal
> ```
>
> vswitchd log:
> ```
> ...
> 2023-09-21T09:29:48.626Z|00067|bridge|INFO|bridge ovs_dpdk_br0: added
> interface ovs_dpdk_br0 on port 65534
> 2023-09-21T09:29:48.627Z|00068|bridge|INFO|bridge ovs_dpdk_br0: using
> datapath ID d62195e14c4e
> 2023-09-21T09:29:48.627Z|00069|connmgr|INFO|ovs_dpdk_br0: added service
> controller "punix:/usr/local/var/run/openvswitch/ovs_dpdk_br0.mgmt"
> 2023-09-21T09:29:48.718Z|00070|dpdk|ERR|EAL: Driver cannot attach the
> device (:03:00.0)
> 2023-09-21T09:29:48.718Z|00071|dpdk|ERR|EAL: Failed to attach device on
> primary process
> 2023-09-21T09:29:48.719Z|00072|netdev_dpdk|WARN|Error attaching device
> ':03:00.0' to DPDK
> 2023-09-21T09:29:48.719Z|00073|netdev|WARN|dpdk0: could not set
> configuration (Invalid argument)
> 2023-09-21T09:29:48.719Z|00074|dpdk|ERR|Invalid port_id=32
> 2023-09-21T09:29:57.230Z|00075|memory|INFO|27944 kB peak resident set
> size after 10.3 seconds
> 2023-09-21T09:29:57.230Z|00076|memory|INFO|handlers:1 ports:1
> revalidators:1 rules:5
> ```
>
>
> Is there any other log files I can look into for more informational
> debug messages?
You may enable additional DPDK debug logs by setting --log-level
in the dpdk-extra config. However, your DPDK version is older than
a hardware you're using. Meaning, the mlx driver in DPDK 19.11
likely just doesn't recognize the hardware and will not be able to
use it. You need to figure out how to build newer versions of DPDK.
Also, representor syntax changed a few times in the past, so your
script may not work with older versions of DPDK.
If you can't build DPDK on the board itself, cross-compiling may be
an option:
https://doc.dpdk.org/guides/platform/bluefield.html
Also OVS 2.14 is likely a poor choice for hardware offloading.
I would not recommend anything below 2.17. 3.1 would be a better.
>
>
> Cheers,
> levi
>
>
> On Wed, 2023-09-20 at 17:20 +0200, Ilya Maximets wrote:
>> On 9/20/23 06:39, Levente Csikor via discuss wrote:
>>> Hi All,
>>>
>>> I have a long lasting problem I have been trying to resolve for
>>> quite
>>> some time. I am playing around with an NVidia SmartNIC (Bluefield-
>>> 2),
>>> which has OvS installed by default. It works well with the kernel
>>> driver, and even TC hardware offloading is working.
>>>
>>> I want to experiment with DPDK, though.
>>> DPDK is also installed by default on the Bluefield-2.
>>> The details of the versions are as follows:
>>>
>>> OvS 2.17.7
>>> DPDK 22.11.1.4.2
>>>
>>> Following the "NVidia tutorials", I manage to add the physical port
>>> as
>>> a netdev device to an OVS-DPDK bridge, however, adding the virtual
>>> function fails.
>>>
>>> More details about the commands and problem are here:
>>> https://forums.developer.nvidia.com/t/error-with-configuring-ovs-dpdk-on-bluefiled-2/256030/4
>>>
>>>
>>> Anyway, as a last resort, I thought I give a try to install OvS and
>>> DPDK from scratch following
>>> https://docs.openvswitch.org/en/latest/intro/install/dpdk/
>>>
>>> I used the same version for OvS and DPDK; the latter was anyway the
>>> one
>>> recommended by the OvS documentation.
>>>
>>> During the `make` process, I encounter several errors I cannot
>>> really
>>> resolve
>>> ```
>>> In file included from lib/dp-packet.h:29,
>>> from lib/bfd.c:28:
>>> lib/netdev-dpdk.h:95:12: warning: ‘struct
Re: [ovs-discuss] Possible bug with monitor_cond_since and fast re-sync of ovn-controllers
d
> request, method="*monitor_cond_since*", params=["OVN_Southbound",...,
> "*75718296-7ff5-4d6d-9fe8-129fe4ea2075*"], id=40891/
>
> / 2023-09-20T11:31:24.840Z|11890|jsonrpc|DBG|tcp:172.31.23.11:6642:
> received reply,
> result=[*false*,"*aefff81c-76ae-4978-ac38-c129ed1a1009*",{"RBAC_Role],
> id=40891/
>
> / ### downloading of full SB DB contents…./
>
> / /
>
> /
> {"eid":"*75718296-7ff5-4d6d-9fe8-129fe4ea2075*","data":[null,{"Chassis_Private":{"8a94ede6-ee86-443f-a861-19320b500767":{"nb_cfg":1078267,"nb_cfg_timestamp":1695208789023}},"_comment":"ovn-controller","_date":1695208789142,"_is_diff":true}],"term":9866,"index":*8583308*}/
>
> /
> {"eid":"*aefff81c-76ae-4978-ac38-c129ed1a1009*","data":[null,{"Chassis_Private":{"94e607a0-9e1a-4ca4-9e26-b1cbdcdf362a":{"external_ids":["map",[["neutron:ovn-metadata-sb-cfg","1078278"]]]}},"_date":1695209448827,"_is_diff":true}],"term":9866,"index":*8583427*}/
>
>
>
> Before yoga, on kolla images with OVS 2.16, reconnect worked well.
This sounds strange. As you said, with OVS 2.16 python clients didn't
support fast re-sync, so the situation should have been worse, not better.
> In long-term perspective, I can propose to make transaction history limit as
> configurable parameter.
It may be a possibility. Not making it configurable, but just increasing
it to a much higher value (it's capped by the size of the database itself,
so should not grow too large).
On the other hand, starting with OVS 3.1, ovsdb-server should reply to
new monitor requests without fast re-sync way faster. So, upgrade to
that version may fix your problem.
>
> In short-term perspective, all we can do now - avoid transferring of
> leadership (when doing snapshot) doing manual DB compaction time to time,
> whithin 24 hours.
I don't understand why that is supposed to help, because clients should
not disconnect on leadership change in the first place. All the SbDB
clients, except for ovn-northd, should not require leader-only connections.
Could you, please, check why is this happening?
Best regards, Ilya Maximets.
>
> Kindle ask you to review and comment this information.
>
> Thank you!
>
> Oleksandr Mykhalskyi, System Engineer
> *Netcracker Technology*
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OvS-DPDK compilation fails on NVidia Bluefield-2
t; here (not in a function) >72 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_GOOD, > RTE_MBUF_F_RX_IP_CKSUM_GOOD, > | > ^~~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:75:42: error: ‘RTE_MBUF_F_TX_TCP_SEG’ undeclared here > (not in a function); did you mean ‘RTE_MBUF_MAX_NB_SEGS’? >75 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_SEG, RTE_MBUF_F_TX_TCP_SEG, > 0x40), > | ^ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:77:39: error: ‘RTE_MBUF_F_TX_IPV4’ undeclared here (not > in a function) >77 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV4, RTE_MBUF_F_TX_IPV4, > 0x80), > | ^~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:79:39: error: ‘RTE_MBUF_F_TX_IPV6’ undeclared here (not > in a function) >79 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV6, RTE_MBUF_F_TX_IPV6, > 0x100), > | ^~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:81:44: error: ‘RTE_MBUF_F_TX_TCP_CKSUM’ undeclared here > (not in a function) >81 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_CKSUM, > RTE_MBUF_F_TX_TCP_CKSUM, 0x200), > | > ^~~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:83:44: error: ‘RTE_MBUF_F_TX_UDP_CKSUM’ undeclared here > (not in a function) >83 | DEF_OL_FLAG(DP_PACKET_OL_TX_UDP_CKSUM, > RTE_MBUF_F_TX_UDP_CKSUM, 0x400), > | > ^~~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > lib/dp-packet.h:85:45: error: ‘RTE_MBUF_F_TX_SCTP_CKSUM’ undeclared > here (not in a function) >85 | DEF_OL_FLAG(DP_PACKET_OL_TX_SCTP_CKSUM, > RTE_MBUF_F_TX_SCTP_CKSUM, 0x800), > | > ^~~~ > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ >52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > DPDK_DEF > | > ^~~~ > make[2]: *** [Makefile:4645: lib/bfd.lo] Error 1 > make[2]: Leaving directory '/opt/ovs-2.17.7' > make[1]: *** [Makefile:5409: all-recursive] Error 1 > make[1]: Leaving directory '/opt/ovs-2.17.7' > make: *** [Makefile:3079: all] Error 2 > ``` > > Asking google did not help at all :( > Did anyone encounter such errors? Nvidia is using their closed-source versions of OVS and DPDK, so the versions you provided are not what you need. Upstream OVS 2.17 works only with DPDK 21.11. See the "What DPDK version does each Open vSwitch release work with?" section of releases FAQ: https://docs.openvswitch.org/en/latest/faq/releases/ DPDK 22.11 is supported starting with OVS 3.1. But I would guess that Nvidia's DPDK 22.11 is not actually an upstream DPDK 22.11. Best regards, Ilya Maximets. > > btw. the Ethernet ports/hw that the Bluefield-2 has is a > Mellanox/Nvidia ConnectX-6, using the mlx5_core driver. > > > All comments are welcome :) > > Thanks, > Levi ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd/ovn-controller at 100% CPU when using AddressSet with CIDRs in an ACL
On 9/18/23 11:34, Brendan Doyle via discuss wrote:
> Hi Folks,
>
> So we have run into an issue when using an Address Set (AS) containing a
> number of CIDRs in an ACL.
> Initially we observed out of memory (OOM) kills of
> ovs-vswitchd/ovn-controller on ovn-controller
> node chassis, we then upgrade to OVN/OVS latest LTS, this fixed the OOM
> kills but now see
> ovs-vswitchd/ovn-controller hit 100% CPU on ovn-controller node
> chassis, whether it is ovs-vswitchd
> or ovn-controller seems to depend on the size and number of CIDRs in the
> AS. I have narrowed down
> the issue to be 100% reproducible with this very basic OVN configuration
> that just has a gateway with
> distributed router port:
>
>
> ovn-nbctl show
> switch 5f666cde-217c-487d-9470-86e6cb197f39 (ls_vcn2_external_gw)
> port ls_vcn2_external_gw-lr_vcn2_gw
> type: router
> router-port: lr_vcn2_gw-ls_vcn2_external_gw
> port ln-ls_vcn2_external_gw
> type: localnet
> addresses: ["unknown"]
> router 484cbb41-bc6b-4be3-b79e-c05696146786 (lr_vcn2_gw)
> port lr_vcn2_gw-ls_vcn2_external_gw
> mac: "40:44:00:00:01:80"
> networks: ["253.255.80.6/16"]
> gateway chassis: [pcacn003 pcacn001 pcacn002]
>
> It has the following ALC on external gateway switch:
>
> ovn-nbctl acl-list ls_vcn2_external_gw
> from-lport 32700 (inport == "ls_vcn2_external_gw-lr_vcn2_gw" && (ip4.dst
> == 10.80.179.0/28 && ip4.src != $vcn2_as_10_80_179_0_28)) drop
> log(name=vcn2_as_10_80_179_0_28_gw)
>
> The AS has the following CIDRs:
>
> ovn-nbctl list Address_Set vcn2_as_10_80_179_0_28 | grep addre
> addresses : ["192.17.1.0/28", "192.17.1.16/28",
> "192.17.1.32/28", "192.17.1.48/28" ,"192.17.1.64/28", "192.17.1.80/28",
> "192.17.1.96/28"]
>
> If I restrict the AS to just 4 CIDRs, everything is OK, but with 7 or
> more ovs-vswitchd reaches 100% CPU on all ovn-controller
> chassis and stays there. If I change the size of the CIDRs to /24 then
> ovn-controller hits 100% CPU with 5 or more CIDRs in the AS.
>
> Is this a known issue? or what can be happening here?
Hi, Brendan. It is a known issue caused by having many different
networks in the negative match. It is causing an explosion of the
number of OpenFlow rules.
The solution is to try and reduce the number of negative ('!=')
matches in your expression or turn the ACL into multiple ACLs with
positive matches instead, if possible. E.g. by using an explicit
'allow' ACL for these CIDRs and a lower priority catch-all 'deny'.
The issue was mostly addressed in OVN 23.06 by the following commit:
https://github.com/ovn-org/ovn/commit/422ab29e76b5d5bf3865fc743434cb958ce20cc8
So, if many negative matches are unavoidable, you should consider
upgrading to that version or a most recent 23.09 release.
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] open vswitch and libvirt
On 9/8/23 17:11, Laurent Coloby wrote: > Hello Ilya > > I try to install the package Debian for OVS > > > And I facing issue quickly , step 3 > > I cannot find "Build-Depends" . after apt-get install ( which is success ), > all next actions failed As the guide says, they are in debian/control.in file. Why can't you find them? If you're having trouble installing dependencies, run: # apt install linux-headers-$(uname -r) build-essential fakeroot devscripts equivs # ./boot.sh && ./configure && make debian # mk-build-deps --install --root-cmd sudo --remove debian/control # dpkg-checkbuilddeps # make debian-deb That should end up with a pack of debian packages. > > > Regards > > Laurent > > > > > Internal > -Message d'origine- > De : Ilya Maximets > Envoyé : vendredi 8 septembre 2023 14:15 > À : Eelco Chaudron ; Laurent Coloby > > Cc : i.maxim...@ovn.org; b...@openvswitch.org > Objet : Re: [ovs-discuss] open vswitch and libvirt > > [External email: Use caution with links and attachments] > > > > > > On 9/8/23 14:03, Eelco Chaudron via discuss wrote: >> >> >> On 8 Sep 2023, at 13:55, Laurent Coloby via discuss wrote: >> >>> Hello Michal and Ilya >>> >>> Thank you for your answser >>> >>> You are right >>> With trace I can see is it due to permission. >>> >>> All the commands are send with : sudo >>> >>> What do you mean by " because you're starting OVS manually from a root >>> user, IIUC" >>> >>> Sorry I'm a beginner in Linux >>> >>> For starting, I enter >>> >>> sudo ovs-ctl start >>> >>> To launch VM, : >>> sudo virsh start VM > > virsh is just a frontend for libvirtd. Actual VM is started by libvirtd, > which is likely running under libvirt/qemu user and group. It will not have > access to sockets created by ovs-vswitchd unless ovs-vswitchd is running > under the same user or group. > > It would likely be easier if you install OVS as a deb/rpm package depending > on your operating system. It will be managed as a systemd service with a > correct user/group configuration. > > You may install packages from your distribution or build them from sources > using guides: > https://docs.openvswitch.org/en/stable/intro/install/debian/ > https://docs.openvswitch.org/en/stable/intro/install/fedora/ > >>> >>> I have create ovs bridge with sudo , also >> >> >> Here is a link to what I normally do under Fedora/RHEL. However, looking at >> your configuration, you name the bridge in libvirt, ovsbr. But your ovs show >> command does not have that bridge created. > > +1 > That will become an issue once you fix the permissions. > >> >>> >>> Regards >>> >>> >>> >>> >>> Internal >>> -Message d'origine- >>> De : Ilya Maximets Envoyé : jeudi 7 septembre >>> 2023 16:49 À : Michal Prívozník ; Laurent Coloby >>> ; b...@openvswitch.org Cc : i.maxim...@ovn.org >>> Objet : Re: [ovs-discuss] open vswitch and libvirt >>> >>> [External email: Use caution with links and attachments] >>> >>> >>> >>> >>> >>> On 9/7/23 16:38, Michal Prívozník via discuss wrote: >>>> On 9/6/23 15:26, Laurent Coloby via discuss wrote: >>>>> Hello support >>>>> >>>>> I'm blocked on a basic issue >>>>> >>>>> I use libvirt / KVM with unbutu >>>>> >>>>> >>>>> I just want to launch a virtual Machine with a connection to OVS >>>>> >>>>> With command virsh edit VM , I modify to be conform to your >>>>> recommendation >>>>> https://d/ >>>>> oc%2F=05%7C01%7Claurent.coloby%40se.com%7C3fd53a68ae4e443b5040 >>>>> 08dbb0651fad%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C638297720 >>>>> 592311463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM >>>>> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=GzqNpVdyHfh >>>>> HnjH0N4qMkmJQF%2BQybAYf4%2FNG%2FpSs%2FXk%3D=0 >>>>> s.openvswitch.org%2Fen%2Fstable%2Fhowto%2Flibvirt%2F=05%7C01%7 >>>>> Cl >>>>> aurent.coloby%40se.com%7Ca0388c4bc678452caacf08dbafb18316%7C6e51e1a >>>>> dc >>>>> 54b4b39b5980ffe9ae68fef%7C0%7C0%7C6382969499749
Re: [ovs-discuss] open vswitch and libvirt
On 9/8/23 14:21, Laurent Coloby wrote: > One remark > > Currently I use unbuntu > > I should move to RedHat soon. I 'm waiting a new server machine > > But I 'd like to test previously with my unbuntu > > Should I setup Debian OVS ? If you don't want to change OVS sources, then just install openvswitch package with 'apt install openvswitch-switch'. If you plan to modify the code, then follow the guide on how to build a package from sources: https://docs.openvswitch.org/en/stable/intro/install/debian/ > > > Regards > > Laurent > > > > Internal > -Message d'origine- > De : Ilya Maximets > Envoyé : vendredi 8 septembre 2023 14:15 > À : Eelco Chaudron ; Laurent Coloby > > Cc : i.maxim...@ovn.org; b...@openvswitch.org > Objet : Re: [ovs-discuss] open vswitch and libvirt > > [External email: Use caution with links and attachments] > > > > > > On 9/8/23 14:03, Eelco Chaudron via discuss wrote: >> >> >> On 8 Sep 2023, at 13:55, Laurent Coloby via discuss wrote: >> >>> Hello Michal and Ilya >>> >>> Thank you for your answser >>> >>> You are right >>> With trace I can see is it due to permission. >>> >>> All the commands are send with : sudo >>> >>> What do you mean by " because you're starting OVS manually from a root >>> user, IIUC" >>> >>> Sorry I'm a beginner in Linux >>> >>> For starting, I enter >>> >>> sudo ovs-ctl start >>> >>> To launch VM, : >>> sudo virsh start VM > > virsh is just a frontend for libvirtd. Actual VM is started by libvirtd, > which is likely running under libvirt/qemu user and group. It will not have > access to sockets created by ovs-vswitchd unless ovs-vswitchd is running > under the same user or group. > > It would likely be easier if you install OVS as a deb/rpm package depending > on your operating system. It will be managed as a systemd service with a > correct user/group configuration. > > You may install packages from your distribution or build them from sources > using guides: > https://docs.openvswitch.org/en/stable/intro/install/debian/ > https://docs.openvswitch.org/en/stable/intro/install/fedora/ > >>> >>> I have create ovs bridge with sudo , also >> >> >> Here is a link to what I normally do under Fedora/RHEL. However, looking at >> your configuration, you name the bridge in libvirt, ovsbr. But your ovs show >> command does not have that bridge created. > > +1 > That will become an issue once you fix the permissions. > >> >>> >>> Regards >>> >>> >>> >>> >>> Internal >>> -Message d'origine- >>> De : Ilya Maximets Envoyé : jeudi 7 septembre >>> 2023 16:49 À : Michal Prívozník ; Laurent Coloby >>> ; b...@openvswitch.org Cc : i.maxim...@ovn.org >>> Objet : Re: [ovs-discuss] open vswitch and libvirt >>> >>> [External email: Use caution with links and attachments] >>> >>> >>> >>> >>> >>> On 9/7/23 16:38, Michal Prívozník via discuss wrote: >>>> On 9/6/23 15:26, Laurent Coloby via discuss wrote: >>>>> Hello support >>>>> >>>>> I'm blocked on a basic issue >>>>> >>>>> I use libvirt / KVM with unbutu >>>>> >>>>> >>>>> I just want to launch a virtual Machine with a connection to OVS >>>>> >>>>> With command virsh edit VM , I modify to be conform to your >>>>> recommendation >>>>> https://d/ >>>>> oc%2F=05%7C01%7Claurent.coloby%40se.com%7C3fd53a68ae4e443b5040 >>>>> 08dbb0651fad%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C638297720 >>>>> 592311463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM >>>>> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=GzqNpVdyHfh >>>>> HnjH0N4qMkmJQF%2BQybAYf4%2FNG%2FpSs%2FXk%3D=0 >>>>> s.openvswitch.org%2Fen%2Fstable%2Fhowto%2Flibvirt%2F=05%7C01%7 >>>>> Cl >>>>> aurent.coloby%40se.com%7Ca0388c4bc678452caacf08dbafb18316%7C6e51e1a >>>>> dc >>>>> 54b4b39b5980ffe9ae68fef%7C0%7C0%7C638296949974962097%7CUnknown%7CTW >>>>> Fp >>>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI >>>>> 6M >>>>> n0%3D%7C2000%7C%7C%
Re: [ovs-discuss] open vswitch and libvirt
On 9/8/23 14:03, Eelco Chaudron via discuss wrote: > > > On 8 Sep 2023, at 13:55, Laurent Coloby via discuss wrote: > >> Hello Michal and Ilya >> >> Thank you for your answser >> >> You are right >> With trace I can see is it due to permission. >> >> All the commands are send with : sudo >> >> What do you mean by " because you're starting OVS manually from a root user, >> IIUC" >> >> Sorry I'm a beginner in Linux >> >> For starting, I enter >> >> sudo ovs-ctl start >> >> To launch VM, : >> sudo virsh start VM virsh is just a frontend for libvirtd. Actual VM is started by libvirtd, which is likely running under libvirt/qemu user and group. It will not have access to sockets created by ovs-vswitchd unless ovs-vswitchd is running under the same user or group. It would likely be easier if you install OVS as a deb/rpm package depending on your operating system. It will be managed as a systemd service with a correct user/group configuration. You may install packages from your distribution or build them from sources using guides: https://docs.openvswitch.org/en/stable/intro/install/debian/ https://docs.openvswitch.org/en/stable/intro/install/fedora/ >> >> I have create ovs bridge with sudo , also > > > Here is a link to what I normally do under Fedora/RHEL. However, looking at > your configuration, you name the bridge in libvirt, ovsbr. But your ovs show > command does not have that bridge created. +1 That will become an issue once you fix the permissions. > >> >> Regards >> >> >> >> >> Internal >> -Message d'origine- >> De : Ilya Maximets >> Envoyé : jeudi 7 septembre 2023 16:49 >> À : Michal Prívozník ; Laurent Coloby >> ; b...@openvswitch.org >> Cc : i.maxim...@ovn.org >> Objet : Re: [ovs-discuss] open vswitch and libvirt >> >> [External email: Use caution with links and attachments] >> >> >> >> >> >> On 9/7/23 16:38, Michal Prívozník via discuss wrote: >>> On 9/6/23 15:26, Laurent Coloby via discuss wrote: >>>> Hello support >>>> >>>> I'm blocked on a basic issue >>>> >>>> I use libvirt / KVM with unbutu >>>> >>>> >>>> I just want to launch a virtual Machine with a connection to OVS >>>> >>>> With command virsh edit VM , I modify to be conform to your >>>> recommendation >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc >>>> s.openvswitch.org%2Fen%2Fstable%2Fhowto%2Flibvirt%2F=05%7C01%7Cl >>>> aurent.coloby%40se.com%7Ca0388c4bc678452caacf08dbafb18316%7C6e51e1adc >>>> 54b4b39b5980ffe9ae68fef%7C0%7C0%7C638296949974962097%7CUnknown%7CTWFp >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M >>>> n0%3D%7C2000%7C%7C%7C=GA3u82MxAGL7AViinZgd4WMbna2f6QK6U%2FVvvAk >>>> 2wPo%3D=0 >>>> >>>> [cid:image001.png@01D9E0D3.27E6D1E0] >>>> >>>> >>>> And when I start the VM , I have always same issue >>>> >>>> [cid:image002.png@01D9E0D6.2AE7D580] >>> >>> I'm not sure what this list policy is, but if this were libvir-list >>> I'd ask you to NOT take a screenshots of a text only to place them >>> into a text later. >> >> +1 >> Sometimes pictures are also fully stripped from the mail list and so people >> will not even see what you wanted to show. >> >>> >>>> >>>> On Internet , I have seen several people with same issue, but not >>>> fond any explanation >>>> >>>> Please could you help me on this problem ? >>>> Thank you in advance >>>> >>> >>> If you enable libvirt debug logs [1] then you should be able to see >>> what command is libvirt actually executing to add the TAP device into >>> the bridge. The code that drives what arguments are passed to >>> 'ovs-vsctl --may-exist add-port ...' depend on a lot of config knobs >>> [2]. But in this case it should be fairly trivial command. >>> >>> Anyway, if you turn on libvirt debug logs you should be able to see >>> what command was executed (and perhaps even if it reported any error - >>> if it did libvirt should have passed on that error message, so maybe >>> there's an improvement potential on libvirt side too). >> >> +1 as well. Please, look at libvirt logs to see what is going on. >> >> B
Re: [ovs-discuss] open vswitch and libvirt
On 9/7/23 16:38, Michal Prívozník via discuss wrote: > On 9/6/23 15:26, Laurent Coloby via discuss wrote: >> Hello support >> >> I'm blocked on a basic issue >> >> I use libvirt / KVM with unbutu >> >> >> I just want to launch a virtual Machine with a connection to OVS >> >> With command virsh edit VM , I modify to be conform to your recommendation >> https://docs.openvswitch.org/en/stable/howto/libvirt/ >> >> [cid:image001.png@01D9E0D3.27E6D1E0] >> >> >> And when I start the VM , I have always same issue >> >> [cid:image002.png@01D9E0D6.2AE7D580] > > I'm not sure what this list policy is, but if this were libvir-list > I'd ask you to NOT take a screenshots of a text only to place them into > a text later. +1 Sometimes pictures are also fully stripped from the mail list and so people will not even see what you wanted to show. > >> >> On Internet , I have seen several people with same issue, but not fond any >> explanation >> >> Please could you help me on this problem ? >> Thank you in advance >> > > If you enable libvirt debug logs [1] then you should be able to see what > command is libvirt actually executing to add the TAP device into the > bridge. The code that drives what arguments are passed to 'ovs-vsctl > --may-exist add-port ...' depend on a lot of config knobs [2]. But in > this case it should be fairly trivial command. > > Anyway, if you turn on libvirt debug logs you should be able to see what > command was executed (and perhaps even if it reported any error - if it > did libvirt should have passed on that error message, so maybe there's > an improvement potential on libvirt side too). +1 as well. Please, look at libvirt logs to see what is going on. But if I were guessing, I'd say that you may have an issue with permissions, because you're starting OVS manually from a root user, IIUC. Libvirt daemon likely doesn't have permissions to access sockets created by ovs-vswitchd in this case. > > 1: https://libvirt.org/kbase/debuglogs.html > 2: > https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virnetdevopenvswitch.c?ref_type=heads#L130 > > Michal ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 3.2.0] testsuite: 1088 and 2560 failed
On 8/25/23 15:55, Ilya Maximets wrote: > On 8/25/23 10:08, Sangeetha Elumalai via discuss wrote: >> Hi, >> >> Thank you for your quick response and patch. >> The "2560" issue was fixed by applying the patch. >> Still facing the issue for "1088", attached the verbose log here. >> Kindly give me suggestions to solve the issue. > > Please, don't top post. Replies should be made inline. > >> >> Thank you, >> -Sangeetha >> >> On Fri, Aug 25, 2023 at 3:05 AM Frode Nordahl > <mailto:frode.nord...@canonical.com>> wrote: >> >> Hello, Sangeetha, >> >> On Thu, Aug 24, 2023 at 8:42 PM Sangeetha Elumalai via discuss >> mailto:ovs-discuss@openvswitch.org>> wrote: >> > >> > Hi >> > >> > While running the testsuite on beaglebone_black (arm architecture) >> encountered the following issues. >> > Kindly help me solve these issues. >> >> The failure in "2560 testing learning switch - OpenFlow15" is caused >> by unexpected debug logging and is addressed in [0]. >> >> We've not seen failures for "1088 testing PMD - revalidator modify >> overlapping flows" on our armhf and arm64 builds, does it pass on a >> recheck or is something else environmental going on? > > It looks like the other datapath flow gets removed while we're sending > a lot of packets. The machine appears to be slow and hence 10 seconds > pass and the flow is removed before we check. > > Solution might be to stop the time with a time warping mechanism, so > revalidators will not expire datapath flows. If one of you want to fix > that, let me know and feel free to submit a patch. Otherwise, I can take > a stab at fixing this later. Posted a patch to fix an issue here: https://patchwork.ozlabs.org/project/openvswitch/patch/20230828121622.2275756-1-i.maxim...@ovn.org/ Would be great if you can try it out on your system. > > Best regards, Ilya Maximets. > >> >> 0: >> http://patchwork.ozlabs.org/project/openvswitch/patch/20230822084015.2248811-1-frode.nord...@canonical.com/ >> >> <http://patchwork.ozlabs.org/project/openvswitch/patch/20230822084015.2248811-1-frode.nord...@canonical.com/> >> >> -- >> Frode Nordahl >> >> > Thank you, >> > -Sangeetha >> > ___ >> > discuss mailing list >> > disc...@openvswitch.org <mailto:disc...@openvswitch.org> >> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> <https://mail.openvswitch.org/mailman/listinfo/ovs-discuss> >> >> >> ___ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 3.2.0] testsuite: 1088 and 2560 failed
On 8/25/23 10:08, Sangeetha Elumalai via discuss wrote: > Hi, > > Thank you for your quick response and patch. > The "2560" issue was fixed by applying the patch. > Still facing the issue for "1088", attached the verbose log here. > Kindly give me suggestions to solve the issue. Please, don't top post. Replies should be made inline. > > Thank you, > -Sangeetha > > On Fri, Aug 25, 2023 at 3:05 AM Frode Nordahl <mailto:frode.nord...@canonical.com>> wrote: > > Hello, Sangeetha, > > On Thu, Aug 24, 2023 at 8:42 PM Sangeetha Elumalai via discuss > mailto:ovs-discuss@openvswitch.org>> wrote: > > > > Hi > > > > While running the testsuite on beaglebone_black (arm architecture) > encountered the following issues. > > Kindly help me solve these issues. > > The failure in "2560 testing learning switch - OpenFlow15" is caused > by unexpected debug logging and is addressed in [0]. > > We've not seen failures for "1088 testing PMD - revalidator modify > overlapping flows" on our armhf and arm64 builds, does it pass on a > recheck or is something else environmental going on? It looks like the other datapath flow gets removed while we're sending a lot of packets. The machine appears to be slow and hence 10 seconds pass and the flow is removed before we check. Solution might be to stop the time with a time warping mechanism, so revalidators will not expire datapath flows. If one of you want to fix that, let me know and feel free to submit a patch. Otherwise, I can take a stab at fixing this later. Best regards, Ilya Maximets. > > 0: > http://patchwork.ozlabs.org/project/openvswitch/patch/20230822084015.2248811-1-frode.nord...@canonical.com/ > > <http://patchwork.ozlabs.org/project/openvswitch/patch/20230822084015.2248811-1-frode.nord...@canonical.com/> > > -- > Frode Nordahl > > > Thank you, > > -Sangeetha > > ___ > > discuss mailing list > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > <https://mail.openvswitch.org/mailman/listinfo/ovs-discuss> > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] northd: amount of ref_chassis in ha_chassis_group
On 5/31/23 16:40, Max André Lamprecht via discuss wrote: > Hi, > > We noticed in a large openstack cluster (~500 Chassis) that during a > VIP(attached with a floating ip) failover it takes up to 50 seconds until the > traffic gets routed to the correct port. > That is caused due to the slow Logical_Flow update in the southbound db. > Before that update happens all the FIP traffic will be still forwarded to the > previous vip port. > > inc_proc_eng|INFO|node: northd, recompute (forced) took 20502ms > inc_proc_eng|INFO|node: lflow, recompute (forced) took 802ms > > This lflow gets updated by northd. During debugging we discovered that northd > spends a large amount of time at recomputing ref_chassis in the > HA_Chassis_Group table. > > ovnsb_db_run (inlined) > -handle_port_binding_changes (inlined) > - 10.53% build_ha_chassis_group_ref_chassis (inlined) > 8.84% add_to_ha_ref_chassis_info (inlined) > 0.69% hmap_next (inlined) > > Maybe this is special to our environment because we have a few external > stretched l2 networks represented as Logical_Switch. > To these Logical_Switches there are many Logical_Router_Ports attached. e.g. > ~3500 LRPs are attached to one LS. > > compute vm -> internal net -> router -> external net(type=localnet) > N > Routers > C1 -> LS1 -> R1 -> LS2 -> R2..RN -> LS2..LSN -> C2..CN > > Currently we can see that northd adds about ~500 Chassis to each ref_chassis > column. I think that this is too much and not nessasary. Pls correct me if > I´m wrong :) > > If I see this right ref_chassis is only taken to decide where to build the > BFD sessions to. > Is there a reason why this needs to be referenced across chassisredirect > ports and further? > Does it make sense that we stop the whole lookup process in > build_lrouter_groups__() if we have a lrp with a chassis-redirect-port set? Hi, Max. I might have solved some of these issues while investigating a different issue report: https://mail.openvswitch.org/pipermail/ovs-discuss/2023-August/052614.html With these two patches: https://patchwork.ozlabs.org/project/ovn/patch/20230823214140.1779255-1-i.maxim...@ovn.org/ https://patchwork.ozlabs.org/project/ovn/patch/20230823215705.1786348-1-i.maxim...@ovn.org/ Han pointed me to this thread, as it seems like the issue you're facing is practically the same. I agree though that there is a potential for even further improvement as at least there should be a way to not duplicate all the chassis in each group. But I hope that the patches above are enough for now. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Long northd recompute time with OpenStack deployment
On 8/23/23 19:58, Roberto Bartzen Acosta wrote: > Hi Ilya, > > Em qua., 23 de ago. de 2023 às 13:45, Ilya Maximets <mailto:i.maxim...@ovn.org>> escreveu: > > On 8/23/23 07:59, Roberto Bartzen Acosta wrote: > > Hi Ilya, > > > > Regarding what we've been talking about on the openvswitch IRC channel, > I'm adding more information about the setup. > > Hi, Roberto. > Thanks for the information! > > > > > *Summary* > > > > Tests with OVN 22.03 version... (Northd / lflow) recompute time: > > 3k routers - 46183ms/3595ms > > 4k routers - 48008ms/4143ms > > 5k routers - 84347ms/8976ms > > > > Tests with OVN 23.03 / OVS 3.1.2 version (Northd / lflow) recompute > time: > > *5k routers - 75532ms/4750ms > > * > > > > ovn-appctl -t ovn-northd coverage/show > > Event coverage, avg rate over last: 5 seconds, last minute, last hour, > hash=ccb8802c: > > hmap_pathological 0.0/sec 0.000/sec 1.3389/sec > total: 13037 > > hmap_expand 0.0/sec 0.000/sec 694.1992/sec > total: 8429618 > > hmap_reserve 0.0/sec 0.000/sec 438.7769/sec > total: 4365281 > > txn_unchanged 0.0/sec 0.000/sec 0.0619/sec > total: 2366 > > txn_incomplete 0.0/sec 0.000/sec 0.0797/sec > total: 997 > > txn_success 0.0/sec 0.000/sec 0.0194/sec > total: 171 > > txn_try_again 0.0/sec 0.000/sec 0./sec > total: 591 > > poll_create_node 2.0/sec 0.333/sec 0.8039/sec > total: 26251 > > poll_zero_timeout 0.0/sec 0.000/sec 0.0219/sec > total: 305 > > seq_change 1.0/sec 0.167/sec 0.3319/sec > total: 9972 > > pstream_open 0.0/sec 0.000/sec 0./sec > total: 1 > > stream_open 0.0/sec 0.000/sec 0./sec > total: 12 > > unixctl_received 0.0/sec 0.000/sec 0./sec > total: 3 > > unixctl_replied 0.0/sec 0.000/sec 0./sec > total: 3 > > util_xalloc 4.0/sec 0.583/sec 145068.3125/sec > total: 23935972424 > > 93 events never hit > > > > * stopwatch/show [1] > > > > Statistics for 'ovnsb_db_run' Total samples: 121 Maximum: 85562 msec > Minimum: 61409 msec 95th percentile: 81893.110630 msec Short term average: > 83914.189203 msec Long term average: 74790.861215 msec > > > > > > * HA_Chassis_Group table [2] > > * Port_Binding [3] > > * NB database and SB database [4] > > > > > > I included the download link for the databases so we can continue with > the recompute time analysis. > > > > What would be possible to improve in this recompute time considering > the 40K entries in Port_Binding? > > AFAICT, northd is doing horribly inefficient recomputation of ref_chassis > column in HA_Chassis_Group table. It's basically number of ports times > number of ha_chassis_groups (equals to the number of routers in your > setup) times number of chassis. On your databases it turns into about > 270 million operations. I re-worked the code locally to not check the > same things over and over again and re-computation of this particular > part went down from 80 seconds to 52 milliseconds on my machine. > > > That sounds very promising ;) Thank you for your help! > I'll be waiting to backport the patch and test in my setup. > > > > I still need to run some checks to be sure that I'm computing exactly > the same thing that the old code does, after that I'll post the patch. > > We're a little late for the 23.09 release, but I'll try to market this > change as a bug fix, maybe OVN maintainers will agree. :) > > Do you want to be mentioned in a Reported-by tag? > > It would be good to keep tracking. Ack. Just for completeness of the mail thread, here is the list of patches I posted related to this issue: 1. https://patchwork.ozlabs.org/project/ovn/patch/20230823214140.1779255-1-i.maxim...@ovn.org/ 2. https://patchwork.ozlabs.org/project/ovn/patch/20230823215705.1786348-1-i.maxim...@ovn.org/ > > > > Also, this email has "[ovs-discuss]" in the name, but ovs-discuss is > not in CC list, which is a little strange. > > My bad! I f
Re: [ovs-discuss] User-space VxLAN without host IP
On 8/23/23 22:56, krishna khanal via discuss wrote:
> Hello,
>
> I'm following [0] to set up a userspace vxlan tunnel between two hosts and it
> works as expected. But, one of the requirements I have is not to have any IP
> assigned to physical bridge (br-phy) but use the flows to pick up the source
> IP. I've configured the bridges as shown in [1] and added the flow rule as
> shown in [2].
>
> VM is attached to br-int and has an IP 10.10.10.10 assigned to it. My
> expectation is when the packet comes to br-int through br-int internal
> interface, the packet is encapsulated using the flow keys and use the
> local_ip or tun_src as the source IP but when we try to output the packet to
> native tunnel, the source IP lookup fails and the packet is dropped as shown
> in [3].
When the output action is executed, the packet is leaving the
current bridge. So, when it gets encapsulated it has to be
routed to one of the bridges based on the destination IP address
for further processing.
>
> Is there a way to make this work without assigning ip to br-phy? Can the
> routing after vxlan pick up the source IP from the configured rule and not
> depend on the system configured IP?
OVS is using kernel routing configuration for convenience reasons.
You may add routes directly to OVS via 'ovs-appctl ovs/route/add'
command. In your case, you need a route for 192.168.1.20/24 (?)
via br-phy, IIUC. See the 'Tunneling-related Commands' section in [0].
Note, however, that this route will only be stored in memory, so
you will need to re-add it back on OVS restart.
Best regards, Ilya Maximets.
>
> *[0]*
> https://docs.openvswitch.org/en/latest/howto/userspace-tunneling/
> <https://docs.openvswitch.org/en/latest/howto/userspace-tunneling/>
>
> *[1]*
> sudo ovs-vsctl show
> bf7d9abd-7275-4a22-9f80-7733bda04cc2
> Bridge br-phy
> datapath_type: netdev
> Port br-phy
> Interface br-phy
> type: internal
> Port eth1
> Interface eth1
> Bridge br-int
> datapath_type: netdev
> Port br-int
> Interface br-int
> type: internal
> Port vxlan
> Interface vxlan
> type: vxlan
> options: {key=flow, local_ip="192.168.1.10", remote_ip=flow}
>
> *[2]*
> ovs-ofctl add-flow br-int \
> 'in_port=br-int
> actions=set_tunnel:5001,set_field:192.168.1.20->tun_dst,set_field:192.168.1.10->tun_src,output:vxlan'
>
> *[3]*
> ovs-appctl ofproto/trace br-int in_port=br-int
> Flow:
> in_port=LOCAL,vlan_tci=0x,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x
>
> bridge("br-int")
>
> 0. in_port=LOCAL, priority 32768
> set_tunnel:0x1389
> load:0xc0a80114->NXM_NX_TUN_IPV4_DST[]
> load:0xc0a8010a->NXM_NX_TUN_IPV4_SRC[]
> output:3
> -> output to native tunnel
> * >> native tunnel routing failed
> *
> Final flow:
> tun_src=192.168.1.10,tun_dst=192.168.1.20,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=0,tun_ttl=0,tun_erspan_ver=0,gtpu_flags=0,gtpu_msgtype=0,tun_flags=0,in_port=LOCAL,vlan_tci=0x,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x
> Megaflow: recirc_id=0,eth,in_port=LOCAL,dl_type=0x
> *Datapath actions: drop*
> *
> *
> -Krishna
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Cannot clone git repo
On 8/14/23 14:38, Emir Bosnak via discuss wrote: > I would like to obtain the source code to build the repo, but I get the > following error when I run git clone: > > fatal: expected 'packfile' It's unlikely to be specific to OVS. Are you able to clone any other project from GitHub ? Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS Stopped forwarding
On 8/4/23 14:14, Vsevolod Vayner wrote: > Hello Ilya -- > > Thank you for your message and clarification. > > Could you please clarify which method/function called ovs-vswitchd and > ovs-dpctlto open the kernel datapath and set flags? Both are using the same lib/dpif-netlink.c:dpif_netlink_open(). In your particular case, the OVS_DP_F_DISPATCH_UPCALL_PER_CPU breaks upcall delivery, because old OVS 2.12 doesn't support it. > > Thank you. > > > > GCORE <https://gcorelabs.com/> > *Vsevolod Vayner* > > Product Director > > vsevolod.vay...@gcore.com <mailto:vsevolod.vay...@gcore.com> > gcore.com <https://gcore.com/> > <https://www.facebook.com/gcorelabscom/> > <https://www.linkedin.com/company/g-core> > <https://www.youtube.com/channel/UCFhtQE7HB097a1_Nfv-Ma_Q> > <https://www.weibo.com/p/1005056986184457> > > > This message is intended solely for the addressee and may contain > confidential information. If you have received this message in error, please > send it back to us, and immediately and permanently delete it. Do not use, > copy or disclose the information contained in this message or in any > attachment. > > *From: *Ilya Maximets > *Date: *Monday, 31 July 2023 at 20:27 > *To: *Vsevolod Vayner , b...@openvswitch.org > > *Cc: *[Partner] Fedor Tarasenko , [Partner] > Ivan Kramarev , Dmitri Polyakov > , [Partner] Ivan Suchkov , > Aleksandr Fedotov , i.maxim...@ovn.org > > *Subject: *Re: [ovs-discuss] OVS Stopped forwarding > > On 7/28/23 18:14, Vsevolod Vayner via discuss wrote: >> 1. What you did that made the problem appear. >> >> In our env we use openvswitch exporter to collect dump-flows and data from >> production ovs. >> It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to >> prometheus format. >> We run vswitchd, db & exporter in separate containers. >> >> In our production env we have: >> >> Open vSwitch: >> >> ovs-vswitchd (Open vSwitch) 2.12.0 >> DPDK 18.11.2 >> ovs-vsctl (Open vSwitch) 2.12.0 >> DB Schema 8.0.0 >> >> and we also have an installed OpenvSwitch on our exporter to collect >> ovs-ofctl and ovs-dpctl data: >> >> ovs-vswitchd (Open vSwitch) 3.1.3 >> DPDK 22.11.1 >> ovs-vsctl (Open vSwitch) 3.1.3 >> DB Schema 8.3.1 >> >> 2. What you expected to happen. >> >> We expected the exporter to collect statistics using the ovs-ofctl utility >> with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect >> lookups. >> >> 3. What actually happened. >> >> openvswitch stopped to forward traffic. I see only ingress traffic on ports >> of ovs, no traffic outs. >> >> 4. The kernel version on which Open vSwitch is running (from /proc/version) >> and the distribution and version number of your OS (e.g. "Centos 5.0"). >> >> Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org >> <mailto:mockbu...@kbuilder.bsys.centos.org >> <mailto:mockbu...@kbuilder.bsys.centos.org>>) (gcc version 8.5.0 20210514 >> (Red Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022 >> >> >> In some cases, only a host reboot helps. >> >> If this is not well-known bug, I'll try to find resource in production to >> reproduce and provide more debug information. > > Hi, Vsevolod. > > The problem is that you're using drastically different versions > of ovs-vswitchd and ovs-dpctl. Every time you run ovs-dpctl, > it will open the kernel datapath and set flags to it in the process. > In your particular case, these flags change the way upcalls are > delivered to ovs-vswitchd, so no traffic can go to userspace > afterwards and no new flows will be installed to a datapath as > a result, because ovs-vswitchd is too old to understand a new > datapath configuration. The issue should be resolved by re-starting > ovs-vswitchd. > > In general, use of ovs-dpctl while ovs-vswitchd is running is not > safe. Use ovs-appctl dpctl/* commands to get the same information > via ovs-vswitchd instead. Or at least make sure that the version > of ovs-dpctl is exactly the same as version of ovs-vswitchd. > > Best regards, Ilya Maximets. > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS Stopped forwarding
On 7/28/23 18:14, Vsevolod Vayner via discuss wrote: > 1. What you did that made the problem appear. > > In our env we use openvswitch exporter to collect dump-flows and data from > production ovs. > It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to > prometheus format. > We run vswitchd, db & exporter in separate containers. > > In our production env we have: > > Open vSwitch: > > ovs-vswitchd (Open vSwitch) 2.12.0 > DPDK 18.11.2 > ovs-vsctl (Open vSwitch) 2.12.0 > DB Schema 8.0.0 > > and we also have an installed OpenvSwitch on our exporter to collect > ovs-ofctl and ovs-dpctl data: > > ovs-vswitchd (Open vSwitch) 3.1.3 > DPDK 22.11.1 > ovs-vsctl (Open vSwitch) 3.1.3 > DB Schema 8.3.1 > > 2. What you expected to happen. > > We expected the exporter to collect statistics using the ovs-ofctl utility > with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect > lookups. > > 3. What actually happened. > > openvswitch stopped to forward traffic. I see only ingress traffic on ports > of ovs, no traffic outs. > > 4. The kernel version on which Open vSwitch is running (from /proc/version) > and the distribution and version number of your OS (e.g. "Centos 5.0"). > > Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org > <mailto:mockbu...@kbuilder.bsys.centos.org>) (gcc version 8.5.0 20210514 (Red > Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022 > > > In some cases, only a host reboot helps. > > If this is not well-known bug, I'll try to find resource in production to > reproduce and provide more debug information. Hi, Vsevolod. The problem is that you're using drastically different versions of ovs-vswitchd and ovs-dpctl. Every time you run ovs-dpctl, it will open the kernel datapath and set flags to it in the process. In your particular case, these flags change the way upcalls are delivered to ovs-vswitchd, so no traffic can go to userspace afterwards and no new flows will be installed to a datapath as a result, because ovs-vswitchd is too old to understand a new datapath configuration. The issue should be resolved by re-starting ovs-vswitchd. In general, use of ovs-dpctl while ovs-vswitchd is running is not safe. Use ovs-appctl dpctl/* commands to get the same information via ovs-vswitchd instead. Or at least make sure that the version of ovs-dpctl is exactly the same as version of ovs-vswitchd. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Slow Performance with OvS-DPDK when Running Iperf
On 7/27/23 18:48, Matheus Stolet via discuss wrote: > Hello, Hi. > > I am running some performance benchmarks to figure out why I am getting > fairly low performance when running iperf inside a VM with OvS-DPDK. I > started VMs on two different physical machines using QEMU and KVM. In > one machine I ran the iperf server and on the other machine I ran the > iperf client. Both machines are also running OvS with DPDK on the > datapath and using dpdkvhostuser in the appropriate port. To do this I > followed the instructions on these two pages > (https://docs.openvswitch.org/en/latest/intro/install/dpdk/) and > (https://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/). I You might want to read the following doc as well: https://docs.openvswitch.org/en/latest/howto/dpdk/ > would expect some performance decrease when running things inside a VM > and with a virtual switch, but the performance I am getting is > alarmingly low. When running the an iperf client in one VM that contacts > the iperf server running on the other VM i got a throughput of 3.43 > Gb/s. When I ran the same iperf benchmark on the physical machines I got > a throughput of 23.7 Gb/s. 3.43 Gb/s seems like it is way too low and I > feel like I am missing some essential configuration. > > Things I have tried: > - Setting dpdk-lcore-mask to specify the CPU cores to use with DPDK > >ovs-vsctl --no-wait set Open_vSwitch . > other_config:dpdk-lcore-mask=0x2a Nit: Don't set lcore mask, it's generally not needed and usually harmful, unless you know exactly what you're doing. > > - Setting pmd-cpu-mask so that ovs uses multiple pmd threads > >ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xa80 > > - Turning off some nic offload functions in both guest and host OS that > seem to reduce virtio performance with KVM > >ethtool --ofload enp59s0f0np0 gso off tso off sg off gro off > > Turning off the NIC offload operations actually helped. At first I was > getting measly 161 Mb/s throughput with iperf and turning off those > offloads helped get that up to 3.43 Gb/s, which is still far from ideal. > Does anyone have any ideas as to why I am getting such poor performance > when compared to running the same benchmark on the physical machine? > > > OvS commands used to create bridge and ports > > $ ovs-vsctl add-br br0 > $ ovs-vsctl add-port br0 enp59s0f0np0 The first thing you need to change is to open the physical port above as a dpdk port (interface type=dpdk). In the current config it is just open via AF_PACKET socket, and it doesn't have impressive performance. Note that TCP performance is tricky. In your case you may not get the same rate as you have on the host even if you do everything right. The main reason is that OVS with DPDK doesn't support TSO by default. You may enable support for it though, see: https://docs.openvswitch.org/en/latest/topics/userspace-tso/ But it is experimental and not all scenarios will work, e.g. it will not be possible to use tunnels. Packet-per-second performance should be much higher though. > $ ovs-vsctl set bridge br0 datapath_type=netdev > $ ovs-vsctl add-port br0 vhost0 -- set Interface vhost0 options:n_rxq=10 Nit: 'n_rxq' doesn't work for vhost interfaces, number of queues is derived from the actual virtio device in qemu. It's harmless to have it set, but it will not affect anything, OVS will detect 10 queues anyway. Nit: And there is no much sense in having 10 queues and only 3 cores in pmd-cpu-mask. Also, iperf is single threaded by default, so only one queue will likely be utilized anyway. Bets regards, Ilya Maximets. > type=dpdkvhostuser > > QEMU command used to start VM > > $ taskset -c > 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42 \ > sudo qemu-system-x86_64 \ > -nographic -monitor none -serial stdio \ > -machine accel=kvm,type=q35 \ > -cpu host \ > -smp 11 \ > -m 10G \ > -netdev user,id=net0,hostfwd=tcp::2220-:22 \ > -device virtio-net-pci,netdev=net0 \ > -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost0 > \ > -netdev > type=vhost-user,chardev=char0,vhostforce=on,queues=10,id=net1 \ > -device virtio-net-pci,netdev=net1,mac=$mac,mq=on,vectors=22 \ > -object > memory-backend-file,id=mem,size=10G,mem-path=/dev/hugepages,share=on \ > -numa node,memdev=mem -mem-prealloc \ > -drive if=virtio,format=raw,file="base.img" \ > -drive if=virtio,format=raw,file="seed.img" \ > > OvS was compiled to use DPDK with the following configurations: > ./configure --with-dpdk=static CFLAGS="-Ofast -msse4.2 -mpopcnt > -march=native" > > Specs > VM > DPDK: 21.11.4
Re: [ovs-discuss] Marking packet is tunneling packet
On 7/23/23 05:30, PHAM THANH HAI 20191817 via discuss wrote: > Hi, > > I’m working with tunneling traffic (GENEVE and Vxlan). I have some VMs > connect with a OVS bridge. Traffic from VMs is both tunnel and non-tunnel. > Beside that, I have a Ryu controller to config qos for ports in bridge. I > want matching only tunnel traffic in OVS by flow. I know that OVS has > metadata fields like: tun_id, tun_src, tun_dst,… for tunnel traffic, but that > fields is only for tunnel port. I tried set ports which connect with VMs to > tunnel port with remote_ip is flow. But by that way, Ryu controller can’t > find and config this port. > > So i want to ask: Does have any way to OVS matching tunnel traffic in > non-tunnel port or have any way to Ryu controller can find and config to > tunnel port? Hi. I'm not sure why your controller is not able to see tunnel ports, they should be visible like any other ports. But encapsulated packets usually have specific UDP destination port numbers, so you should be able to create OpenFlow rules that match on them, in case you just need to forward these packets without decapsulation. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs pypi package
On 7/19/23 09:52, Felix Huettner via discuss wrote: > Hi everyone, > > i noticed that the latest release of the ovs library on pypi is for > 2.17.1 [1]. Would it be possible to pushlish newer versions of the ovs > python lib there as well, or are there reasons speaking against that? The pypi package is maintained by OpenStack folks mostly for their own use. But, I guess, it should be possible to update to some newer stable release. Terry, what do you think? Best regards, Ilya Maximets. > > [1] https://pypi.org/project/ovs/#history > > Thanks > Felix ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [External] : Re: Latest OVN LTS v22.03.2 working with latest OVS LTS v2.17.7
On 7/18/23 19:46, Joe Liu wrote: > Thanks Ilya for your quick response. One follow-up question: > > In this particular case, the OVN v22.03.2 code base northd/ovn-northd.c > consisted the call to > > ovsdb_idl_set_write_changed_only_all(ovnsb_idl_loop.idl, true); > > which is not available in OVS v1.17.7 source, thus not available to its > runtime either. > Does this mean for OVN v22.03.2 runtime ovn-northd.c, the main() function > will never be > called, thus the function to > ovsdb_idl_set_write_changed_only_all(ovnsb_idl_loop.idl, true); > will never be invoked? OVN statically links with required OVS libraries, so if the function is available in the submodule you build with, it will become part of ovn-northd and other OVN binaries. This way there is no dependency on OVS libraries in runtime. > > Thanks in advance for your timely help, > Joe > ------ > *From:* Ilya Maximets > *Sent:* Tuesday, July 18, 2023 1:36 PM > *To:* Joe Liu ; ovs-discuss@openvswitch.org > > *Cc:* i.maxim...@ovn.org > *Subject:* [External] : Re: [ovs-discuss] Latest OVN LTS v22.03.2 working > with latest OVS LTS v2.17.7 > > On 7/18/23 19:07, Joe Liu via discuss wrote: >> Hi, >> >> I have a question about building lates OVN LTS release v22.03.2 working with >> latest OVS LTS release v2.17.7 >> >> The build failed with the ovn-northd.c referencing to OVS v2.17.7 source >> missing the following function: >> >> northd/ovn-northd.c: >> ovsdb_idl_set_write_changed_only_all(ovnsb_idl_loop.idl, true); >> >> >> From OVN side, I checked the initial OVN LTS v22.03.0 did not have such >> reference, but starting with v22.03.1, the reference was introduced. >> >> From OVS side, I checked the referenced function was not available from OVS >> v2.17.7 but started available from OVS release v3.0.0: >> >> lib/ovsdb-idl.c:ovsdb_idl_set_write_changed_only_all(struct ovsdb_idl *idl, >> bool enable) >> lib/ovsdb-idl.h:void ovsdb_idl_set_write_changed_only_all(struct ovsdb_idl >> *idl, bool enable); >> >> >> My question is: how could we go with both OVN and OVS and follow their >> perspective latest LTS releases? If we go with OVN LTS v22.03.2, do we have >> to go with a non-LTS OVS release (e.g., v3.0+)? Or, if we go with OVS LTS >> v2.17.1, we might stuck with OVN LTS v22.03.0 without LTS update path? > > Hi, Joe. > > The version of OVS to build with and version of OVS to run with > are two different things. You should build OVN with the version > of OVS provided in a submodule. For v22.03.2 it will be OVS > commit 2410b95597fcec5f733caf77febdb46f4ffacd27. > > OVN only uses some libraries from OVS sources. You should be > able to run OVN built this way with your OVS 2.17.7 without any > issues. > > One inconvenience though is that, unfortunately, release archives > on GitHub do not include submodules (GitHub doesn't support that). > So, you'll need to check it out yourself. > > Best regards, Ilya Maximets. > >> >> Thanks in advance for your help, >> Joe > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Latest OVN LTS v22.03.2 working with latest OVS LTS v2.17.7
On 7/18/23 19:07, Joe Liu via discuss wrote: > Hi, > > I have a question about building lates OVN LTS release v22.03.2 working with > latest OVS LTS release v2.17.7 > > The build failed with the ovn-northd.c referencing to OVS v2.17.7 source > missing the following function: > > northd/ovn-northd.c: > ovsdb_idl_set_write_changed_only_all(ovnsb_idl_loop.idl, true); > > > From OVN side, I checked the initial OVN LTS v22.03.0 did not have such > reference, but starting with v22.03.1, the reference was introduced. > > From OVS side, I checked the referenced function was not available from OVS > v2.17.7 but started available from OVS release v3.0.0: > > lib/ovsdb-idl.c:ovsdb_idl_set_write_changed_only_all(struct ovsdb_idl *idl, > bool enable) > lib/ovsdb-idl.h:void ovsdb_idl_set_write_changed_only_all(struct ovsdb_idl > *idl, bool enable); > > > My question is: how could we go with both OVN and OVS and follow their > perspective latest LTS releases? If we go with OVN LTS v22.03.2, do we have > to go with a non-LTS OVS release (e.g., v3.0+)? Or, if we go with OVS LTS > v2.17.1, we might stuck with OVN LTS v22.03.0 without LTS update path? Hi, Joe. The version of OVS to build with and version of OVS to run with are two different things. You should build OVN with the version of OVS provided in a submodule. For v22.03.2 it will be OVS commit 2410b95597fcec5f733caf77febdb46f4ffacd27. OVN only uses some libraries from OVS sources. You should be able to run OVN built this way with your OVS 2.17.7 without any issues. One inconvenience though is that, unfortunately, release archives on GitHub do not include submodules (GitHub doesn't support that). So, you'll need to check it out yourself. Best regards, Ilya Maximets. > > Thanks in advance for your help, > Joe ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Testing for membership in set
On 7/17/23 07:52, Numan Siddique via discuss wrote:
> On Mon, Jul 17, 2023 at 5:51 AM Gavin McKee via discuss
> wrote:
>>
>> Hi,
>>
>> Can someone help me understand how to find a uuid in a set of Ports using
>> the table below as an example?
>>
>> I want to check if the uuid 1191229a-6878-4dfc-b07a-47e6e2a4dcfe is in the
>> set of assigned ports (in some cases I have a large number of ports and want
>> to search for membership in a port list).
>>
>> [root@ovnkube-db-0 ~]# ovn-nbctl --no-leader-only find Logical_Switch
>> name=a_b461d3e2_621c_43f4_8d7c_e39c13bcba08_ls_bca4b75e_9b21_4454_98a3_e490feecae9f
>>
>> _uuid : 7c28c854-a40d-4145-b6f3-7d0f0e69d5ab
>> acls: []
>> copp: []
>> dns_records : []
>> external_ids:
>> {crusoe_subnet_id="bca4b75e-9b21-4454-98a3-e490feecae9f",
>> crusoe_vpc_id="ec7f1ea4-66a2-4b9e-ac0d-79e9ec0998db",
>> customer_id="b461d3e2-621c-43f4-8d7c-e39c13bcba08"}
>> forwarding_groups : []
>> load_balancer : []
>> load_balancer_group : []
>> name:
>> a_b461d3e2_621c_43f4_8d7c_e39c13bcba08_ls_bca4b75e_9b21_4454_98a3_e490feecae9f
>> other_config: {exclude_ips="172.27.16.0..172.27.16.4",
>> subnet="172.27.16.0/20"}
>> ports : [1191229a-6878-4dfc-b07a-47e6e2a4dcfe,
>> 2937d508-3db0-4d40-8c06-10849abcf9d9, 327e037a-b11e-4dc4-8846-08a9c262b911]
>> qos_rules : []
>>
>>
>> Thanks
>>
>
> One way I can think of is running the below command.
>
> # ovn-nbctl get logical_switch
> a_b461d3e2_621c_43f4_8d7c_e39c13bcba08_ls_bca4b75e_9b21_4454_98a3_e490feecae9f
> ports | grep 1191229a-6878-4dfc-b07a-47e6e2a4dcfe
>
> If you don't want to use grep, then I'm not sure.
Something like this should work (not tested):
ovn-nbctl --no-leader-only find Logical_Switch \
name=a_b461d3e2_621c_43f4_8d7c_e39c13bcba08_ls_bca4b75e_9b21_4454_98a3_e490feecae9f
\
ports{\>}1191229a-6878-4dfc-b07a-47e6e2a4dcfe
Following command should check if two ports are there:
ovn-nbctl --no-leader-only find Logical_Switch \
name=a_b461d3e2_621c_43f4_8d7c_e39c13bcba08_ls_bca4b75e_9b21_4454_98a3_e490feecae9f
\
ports{\>}1191229a-6878-4dfc-b07a-47e6e2a4dcfe,2937d508-3db0-4d40-8c06-10849abcf9d9
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Best conference about OVS/Openflow proposals
On 7/12/23 14:05, Lucero Palau, Alejandro via discuss wrote: > Hi, > > I’m working on a paper with some ideas about how OVS and OVS-like software > could benefit from new hardware designs. > > I wonder which would be the right conference for sending the paper to. It > seems there are no OVS conferences anymore, Hi. Strange to hear that. OVS+OVN Conference is happening every year for many years. It's switching between virtual and in-person in recent years for obvious reasons, but it's still happening. Here is a website for the last year (it was hybrid): https://www.openvswitch.org/support/ovscon2022/ And here is a CFP for this year (will be virtual): https://mail.openvswitch.org/pipermail/ovs-announce/2023-July/000324.html Best regards, Ilya Maximets. > and not a Openflow specific one. Obviously, it needs to be a network-related > conference, with the next Linux netdev being one possibility for a first > paper, and I’m considering USENIX NSDI’24 for a full standard paper with all > the detailed ideas described. > > Any advice will be welcome. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] What is the best way to add open flow rules for encapsulated packets coming from custom netdev-provider?
On 7/6/23 11:51, mstolet--- via discuss wrote:
> So I wrote two custom netdev providers, one of them receives GRE encapsulated
> packets and passes them to OvS. These encapsulated packets have an outer
> ethernet header and ip header, a gre header and an inner ip and tcp header. I
> want OvS to create flow rules that match different fields in the inner and
> outer headers and perform different actions based on this and pass the packet
> to my second custom netdev provider. For example, suppose my custom netdev
> providers are identified by in_port=2 and in_port=3. I might want to add the
> following flow:
>
> ovs-ofctl add-flow br0 "table=0, in_port=2, ip, nw_dst=10.0.0.1,
> tun_src=192.168.10.5, actions=set_tunnel:3, mod_nw_src:10.0.0.5, output:3”
>
> So basically in this flow I want every packet coming from port 2 and with
> inner Ip address 10.0.0.1 and outer ip address 192.168.10.5 to be sent to
> port 3, have the gre key set to 3 and the inner ip address to be set to
> 10.0.0.5.
Hi. The main issue here is that OpenFlow and OVS do not provide
a way to match on inner headers or modify them with actions.
'tun_*' fields are only populated from a tunnel header after the
packet is decapsulated. set_tunnel action only sets a tunnel ID
for a subsequent output to a tunnel port for encapsulation.
So, basically, you need a tunnel port that will decapsulate packets,
then you may match on a header of the decapsulated packet and the
tunnel metadata. Then you should be able to send this packet to
a tunnel port to encapsulate it back.
Alternative will be to add new OpenFlow extensions that will be
able to look inside of the packet without decapsulating it. It
will be a bit more work though. You may look at commit
3d2fbd70bda5 ("userspace: Add support for NSH MD1 match fields")
for all the places you'll need to change.
>
> I haven’t been able to get these actions executed on my packets, even a
> fairly simple rule such as:
>
> ovs-ofctl add-flow br0 "table=0, in_port=2, actions=set_tunnel:3 ,output:3”
>
> where I want every packet coming from my custom netdev provider to get its
> gre key changed and the output is sent to the other port running my other
> custom netdev provider. I have noticed that if I don’t have actions messing
> with tunneling information and just have an output action, the packet gets
> properly redirect to port 3, so I imagine I am missing some essential
> tunneling configuration or something of the sort on my custom netdev
> provider. So what are the necessary things I need to add to a custom netdev
> provider that receives encapsulated packets if I want it to be able to match
> encapsulated GRE packets? I have looked at some of the netdev-vport and
> netdev-dpdk for vhost code, to see how this is done, but I can’t identify the
> missing information that leads my packets to get a drop action when I have
> anything related to tunneling information in add-flow.
>
> For reference, this is how I set up my OvS bridge:
>
> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>
> ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre
> options:remote_ip=flow options:local_ip=flow options:key=flow
>
> ovs-vsctl add-port br0 p2 -- set Interface p2 type=customnetdeva of_port=2
>
> ovs-vsctl add-port br0 p3 -- set Interface p3 type=customnetdevb of_port=3
>
>
> I have also tried creating my GRE tunnel with
>
> ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre
> options:remote_ip=192.168.10.6 options:key=3
>
> but this also didn’t change things and the chosen action for the packets was
> to drop them instead, of executing my custom actions.
You can find an example of a tunnel configuration with a userspace
datapath here:
https://docs.openvswitch.org/en/latest/howto/userspace-tunneling/
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] NXAST_CONTROLLER vs NXAST_CONTROLLER2
On 7/10/23 02:16, Ashish Varma via discuss wrote: > Hello, > > From ovs-actions man pages, the controller action is defined as: > > *The controller action* > *Syntax*: > *controller* > *controller:*/max_len/ > *controller(*/key/*[=*/value/*], ...)* > > There is no ‘controller2’ action. So, when programming a flow, can we ask OVS > to use a specific NXAST_CONTROLLER or NXAST_CONTROLLER2 action? Or it depends > on the options provided in the action? It depend on the provided options. If you're using 'userdata' or 'pause', the NXAST_CONTROLLER2 will be used. If only 'reason' and 'controller_id', then NXAST_CONTROLLER will be used. In the most basic case with 'max_len' only, it will use OUTPUT action with OFPP_CONTROLLER as a port. That's how ovs-ofctl utility translates the text form of the action into OpenFlow. If you're writing your own controller, you should be able to create a raw OpenFlow request with any action you want. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Scaling OVN/Southbound
On 7/5/23 18:00, Felix Huettner wrote: > Hi Han, > > On Fri, Jun 30, 2023 at 05:08:36PM -0700, Han Zhou wrote: >> On Wed, May 24, 2023 at 12:26 AM Felix Huettner via discuss < >> ovs-discuss@openvswitch.org> wrote: >>> >>> Hi Ilya, >>> >>> thank you for the detailed reply >>> >>> On Tue, May 23, 2023 at 05:25:49PM +0200, Ilya Maximets wrote: >>>> On 5/23/23 15:59, Felix Hüttner via discuss wrote: >>>>> Hi everyone, >>>> >>>> Hi, Felix. >>>> >>>>> >>>>> we are currently running an OVN Deployment with 450 Nodes. We run a 3 >> node cluster for the northbound database and a 3 nodes cluster for the >> southbound database. >>>>> Between the southbound cluster and the ovn-controllers we have a >> layer of 24 ovsdb relays. >>>>> The setup is using TLS for all connections, however the TLS Server is >> handled by a traefik reverseproxy to offload this from the ovsdb >>>> >>>> The very important part of the system description is what versions >>>> of OVS and OVN are you using in this setup? If it's not latest >>>> 3.1 and 23.03, then it's hard to talk about what/if performance >>>> improvements are actually needed. >>>> >>> >>> We are currently running ovs 3.1 and ovn 22.12 (in the process of >>> upgrading to 23.03). `monitor-all` is currently disabled, but we want to >>> try that as well. >>> >> Hi Felix, did you try upgrading and enabling "monitor-all"? How does it >> look now? > > we did not yet upgrade, but we tried monitor-all and that provided a big > benefit in terms of stability. > >> >>>>> Northd and Neutron is connecting directly to north- and southbound >> databases without the relays. >>>> >>>> One of the big things that is annoying is that Neutron connects to >>>> Southbound database at all. There are some reasons to do that, >>>> but ideally that should be avoided. I know that in the past limiting >>>> the number of metadata agents was one of the mitigation strategies >>>> for scaling issues. Also, why can't it connect to relays? There >>>> shouldn't be too many transactions flowing towards Southbound DB >>>> from the Neutron. >>>> >>> >>> Thanks for that suggestion, that definately makes sense. >>> >> Does this make a big difference? How many Neutron - SB connections are >> there? >> What rings a bell is that Neutron is using the python OVSDB library which >> hasn't implemented the fast-resync feature (if I remember correctly). >> At the same time, there is the feature leader-transfer-for-snapshot, which >> automatically transfer leader whenever a snapshot is to be written, which >> would happen frequently if your environment is very active. >> When a leader transfer happens, if Neutron set the option "leader-only" >> (only connects to leader) to SB DB (could someone confirm?), then when the >> leader transfer happens, all Neutron workers would reconnect to the new >> leader. With fast-resync, like what's implemented in C IDL and Go, the >> client that has cached the data would only request the delta when >> reconnecting. But since the python lib doesn't have this, the Neutron >> server would re-download full data when reconnecting ... >> This is a speculation based on the information I have, and the assumptions >> need to be confirmed. > > We are currently working with upstream neutron to get the leader-only > flag removed wherever we can. I guess in total the amount of connections > depends on the process count which would be ~150 connections in total in > our case. > >> >>>>> >>>>> We needed to increase various timeouts on the ovsdb-server and client >> side to get this to a mostly stable state: >>>>> * inactivity probes of 60 seconds (for all connections between >> ovsdb-server, relay and clients) >>>>> * cluster election time of 50 seconds >>>>> >>>>> As long as none of the relays restarts the environment is quite >> stable. >>>>> However we see quite regularly the "Unreasonably long xxx ms poll >> interval" messages ranging from 1000ms up to 4ms. >>>> >>>> With latest versions of OVS/OVN the CPU usage on Southbound DB >>>> servers without relays in our weekly 500-node ovn-heater runs >>>> stays below 10% during the test phase. No large p
Re: [ovs-discuss] support for udev changes to the OVS Bond
On 6/23/23 08:32, Duraisankar P via discuss wrote: > Hello, > > We have an issue while adding an interface to OVS bond during bootup. > > Issue : > Whenever a default interface name is used eg: eth0 , eth1 in the OVS bond, > there might be a chance for the interface name to be changed from eth0=>eth1 > or vice versa during the bootup. In this case, OVS bonding does not detect > the udev changes and only monitors the RTNETLINK events. Due to this, wrong > interfaces might be added to the OVS bond or it would be an improper > configuration. > > Is there any alternative to solve this issue caused by the udev changes ? The > possible alternative which we could think of is to add udev rules or add a > custom interface name to the OVS bond instead of standard interface names > untill the udev interface names settle. You should be able to fix udev issue with more udev. For quite a few years now it should support predictable device names based on BIOS config or PCI location of your actual NIC: https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [ovs-dev] Request adding long poll interval metrics to coverage metrics
On 6/20/23 16:10, Aaron Conole wrote:
> Adrian Moreno writes:
>
>> On 6/19/23 10:36, Eelco Chaudron wrote:
>>> On 16 Jun 2023, at 19:19, Aaron Conole wrote:
>>>
>>>> Martin Kennelly writes:
>>>>
>>>>> Hey ovs community,
>>>>>
>>>>> I am a developer working on ovn-kubernetes and I want to
>>>>> programmatically consume long poll information
>>>>> i.e:
>>>>> ovs|00211|timeval(handler25)|WARN|Unreasonably long 52388ms poll
>>>>> interval (752ms user, 209ms system)
>>>>>
>>>>> This is currently exposed via journal logs but it's not practical
>>>>> to consume it there programmatically and I was
>>>>> hoping you could add it to coverage metrics.
>>>>
>>>> I think it could be useful. I do want to be careful about exposing
>>>> these kinds of data in a way that could be misinterpreted. Already,
>>>> that log in particular gets misinterpreted quite a bit, and RH gets
>>>> customers claiming OVS is misbehaving when they've oversubscribed the
>>>> system.
>>> +1
>>>
>>
>> Maybe it's a good time to start documenting coverage counters?
>
> I agree - we should have at least some kind of documentation. Actually,
> it would be really nice if we could do something during
> COVERAGE_DEFINE() that would be like:
>
> COVERAGE_DEFINE(ctr, "description")
>
> and then we can generate documentation from the COVERAGE_DEFINE()s as
> well as querying for it with an ovs-appctl command.
>
> That might be trying to be too fancy, though.
The problem with documenting coverage counters is that we can't
and shouldn't claim that these are in any way a stable API.
They are mainly for developers. Code can change and there might
be no meaningful way preserve current counters or their names.
They can change in each release including minor ones without
prior notice.
Best regards, Ilya Maximets.
>
>>>> Mechanically, it would be pretty simple to do something like:
>>>>
>>>> ---
>>>> diff --git a/lib/timeval.c b/lib/timeval.c
>>>> index 193c7bab17..00e5f2a74d 100644
>>>> --- a/lib/timeval.c
>>>> +++ b/lib/timeval.c
>>>> @@ -40,6 +40,7 @@
>>>> #include "openvswitch/vlog.h"
>>>>
>>>> VLOG_DEFINE_THIS_MODULE(timeval);
>>>> +COVERAGE_DEFINE(long_poll_interval);
>>>>
>>>> #if !defined(HAVE_CLOCK_GETTIME)
>>>> typedef unsigned int clockid_t;
>>>> @@ -645,6 +646,8 @@ log_poll_interval(long long int last_wakeup)
>>>> struct rusage rusage;
>>>>
>>>> if (!getrusage_thread()) {
>>>> +COVERAGE_INC(long_poll_interval);
>>>> +
>>>> VLOG_WARN("Unreasonably long %lldms poll interval"
>>>> " (%lldms user, %lldms system)",
>>>> interval,
>>>> ---
>>>>
>>>> This would at least expose the coverage data via the coverage framework
>>>> and it can be queried via ovs-appctl. Actually, the advantage here is
>>>> that the coverage counter can track some details about X/sec over the
>>>> last 5 seconds, minute, hour, in addition to the total, so we can see
>>>> whether the condition is ongoing.
>>> ___
>>> dev mailing list
>>> d...@openvswitch.org
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>>
>
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Unexpected upcalls
On 6/13/23 15:55, Vašek Šraier wrote: > Hi, > > On 2023-06-12 18:40, Ilya Maximets wrote: >> Flushing datapath flows underneath running ovs-vswitchd process >> may cause all kind of weird behaviour. So, the del-flows as well as >> most of the other ovs-dpctl commands should not be used under normal >> circumstances. Here is what the man page says: >> >> """ >>Do not use commands to add or remove or modify datapath flows if >>ovs-vswitchd is running because it interferes with ovs-vswitchd's own >>datapath flow management. Use ovs-ofctl(8), instead, to work with >>OpenFlow flow entries. >> """ > > > Ok. I didn't notice this paragraph. Thank you for pointing that out. > > > Could you please explain in more detail why flushing the datapath flows > would cause these problems? Because some of my assumptions about OVS > must wrong and I don't know which. Just a link to relevant source code > could probably help me a lot. > > My current understanding is that the datapath flows are always installed > as a reaction to an upcall. There are no statically installed datapath > flows. Also, the revalidator threads always dump the real content of the > datapath and then check every dumped flow if it should be deleted. It > does not have to keep track of the flows in between runs. > > So, assuming the above, when I flush the flows, all packets will > generate upcalls and the necessary rules will be immediately installed > again, business as usual. See the following code path in ofproto/ofproto-dpif-upcall.c: handle_upcalls() --> ukey_install() --> ukey_install__() --> try_ukey_replace() Only if ukey_install() succeeds, the flow will be installed to the datapath. In your case the try_ukey_replace() will return false, because we have the exactly same ukey. And this result will float up. Actions will be executed, but the flow will not be installed. > > I'd guess, that if ovs-vswitchd kept a synchronized copy of the kernel > flow table in userspace and we'd desync it by flushing the kernel table, > the next revalidator run would fix the problem and everything would > behave normally again. (I didn't find any code in revalidator, that > would do this though). What's the catch? Revalidators revalidate ukeys for flows that they dump from the datapath. I.e. they iterate over dumped flows, not over all the ukeys during the main phase. Also, these ukeys are getting their stats updated, because there is still traffic going through them via upcalls, so they will not be revalidated, so they are not getting cleaned up during the revalidator_sweep__() phase. Also the dump is not a reliable source of information, because it happens concurrently from multiple threads and also while handlers are installing new flows. So, if the flow is not in the dump, it doesn't generally mean that it's not in the datapath. Best regards, Ilya Maximets. > > > On 6/11/23 14:20, Vašek Šraier via discuss wrote: >> Test environment >> >> >> 3 node Kubernetes cluster using OVN-Kubernetes, Docker and Fedora 38. > > I've managed to reproduce the same behavior on a single system with just > OVS and a single network namespace. No explicit OpenFlow rules are needed. > > I guess this is not unexpected, but I wanted to confirm it regardless. :) > > > Best, > Vašek ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Unexpected upcalls
running ovs-vswitchd process may cause all kind of weird behaviour. So, the del-flows as well as most of the other ovs-dpctl commands should not be used under normal circumstances. Here is what the man page says: """ Do not use commands to add or remove or modify datapath flows if ovs-vswitchd is running because it interferes with ovs-vswitchd's own datapath flow management. Use ovs-ofctl(8), instead, to work with OpenFlow flow entries. """ Best regards, Ilya Maximets. > > Is there something more I could provide to help with reproducing this? > Or should I report it elsewhere? > > > Best, > Vašek Šraier > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [kolla-ansible][ovn][ovs] connection dropped and inactivity errors
On 6/11/23 04:16, Satish Patel via discuss wrote: > Folks, > > I am getting some strange errors on my kolla based OVN deployment. I have > only 5 nodes so it's not a large deployment. Are there any ovn related tuning > options which I missed ? > > I have the following timers configured at present. > > ovn-openflow-probe-interval="60" > > ovn-remote-probe-interval="6" > > > ==> /var/log/kolla/openvswitch/ovsdb-server.log <== > 2023-05-29T02:05:02.891Z|00027|reconnect|WARN|unix#67466: connection dropped > (Connection reset by peer) > 2023-05-31T16:20:33.494Z|00028|reconnect|ERR|tcp:127.0.0.1:59928 > <http://127.0.0.1:59928>: no response to inactivity probe after 5 seconds, > disconnecting > 2023-06-01T20:43:23.516Z|1|vlog|INFO|opened log file > /var/log/kolla/openvswitch/ovsdb-server.log > 2023-06-01T20:43:23.520Z|2|ovsdb_server|INFO|ovsdb-server (Open vSwitch) > 2.17.3 > 2023-06-01T20:43:33.522Z|3|memory|INFO|7216 kB peak resident set size > after 10.0 seconds > 2023-06-01T20:43:33.522Z|4|memory|INFO|atoms:826 cells:770 monitors:5 > sessions:3 > 2023-06-03T07:44:05.774Z|5|reconnect|ERR|tcp:127.0.0.1:40098 > <http://127.0.0.1:40098>: no response to inactivity probe after 5 seconds, > disconnecting > 2023-06-03T09:41:49.039Z|6|reconnect|ERR|tcp:127.0.0.1:60042 > <http://127.0.0.1:60042>: no response to inactivity probe after 5 seconds, > disconnecting All the logs, except for these ones, can appear under normal circumstances and do not indicate any real issues on their own. I'm not sure what is connecting to a local ovsdb-server from the localhost via TCP. It's not OVN. Maybe some OpenStack component? In any case, this doesn't seem related to the core OVN itself. Best regards, Ilya Maximets. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 3.1.90] system-kmod-testsuite: 21 65 98 99 103 156 157 158 169 171 failed
On 6/11/23 16:13, Виктор Андреевич Тураев wrote: > I solved the problems with the tests and tried to write my own test, I wrote > two, but the second one doesn't work for some reason and I don't understand > why, can you please tell me what I'm doing wrong? The test log tells you why. You're trying to start a new instance of a database server while there is already one running. > And also can I ask you where you can find information on how you can write > tests for open vswitch? I don't think there is a guide. You just need a basic understanding of what generic aototest macros are doing. You can look for GNU Autotest documentation. Here, for example: https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Writing-Testsuites.html And you need to have some basic knowledge of shell scripting and look what OVS-defined macros are doing. In the end, autotest scripts are translated into a pure shell. If you don't understand something in how they work after reading the documentation, you may always try reading the generated ./tests/*testsuite shell script, though it's not intended for humans to read, so a bit hard to follow. > Also sometimes test slow_action on geneve6 tunnel fails without any reason > and then if checked again runs ok, and im not sure why. Again, it does print out the reason. It looks like ovs-pcap script encounters unexpected closure of the stdout. I'm guessing, it's because the grep exits early when it finds a first match. I've never seen this particular issue before, but a simple fix might be to store the output of the ovs-pcap in a file and then grep the file instead. Best regards, Ilya Maximets. > > 24.05.2023, 23:08, "Ilya Maximets" : > > On 5/18/23 11:51, Виктор Андреевич Тураев via discuss wrote: > > Thanks a lot for your help! I took the ubuntu config and rebuilt the > kernel with it, check-kernel didn't give any errors but some tests were > skipped, is that normal? According to the unit tests that I ran with the > recheck=yes flag, two fell with an error, maybe I still missed some modules > when building the kernel? I also attached a .config file just in case. > > > You're missing some userspace tools like tftp. > Not sure why unit tests are failing though. > > Best regards, Ilya Maximets. > > > > 09.05.2023, 21:15, "Ilya Maximets" <mailto:i.maxim...@ovn.org>>: > > On 5/9/23 14:34, Виктор Андреевич Тураев via discuss wrote: > > Hello, I am writing a thesis on ovs , but I had problems, > after I completed all the steps necessary to install ovs in unit tests errors > got out, some I could fix myself, some not, I have ubuntu 22.04.2 latest > which I deployed in virtualbox, while I rebuilt the kernel with version 6.2.8 > from kernel.org, maybe I forgot to include something in the config when > building? > > > Yes. You're missing at least a few NF_CONNTRACK config options: > > > > 2023-05-09T12:09:43.851Z|00023|ofproto_dpif|INFO|system@ovs-system > <mailto:system@ovs-system> <mailto:system@ovs-system > <mailto:system@ovs-system>>: Datapath does not support ct_zone > > 2023-05-09T12:09:43.851Z|00024|ofproto_dpif|INFO|system@ovs-system > <mailto:system@ovs-system> <mailto:system@ovs-system > <mailto:system@ovs-system>>: Datapath does not support ct_mark > > 2023-05-09T12:09:43.851Z|00025|ofproto_dpif|INFO|system@ovs-system > <mailto:system@ovs-system> <mailto:system@ovs-system > <mailto:system@ovs-system>>: Datapath does not support ct_label > > > So, CONFIG_NF_CONNTRACK_ZONES, CONFIG_NF_CONNTRACK_MARK, etc. > > This is causing failures of most of the CT tests. You also don't > have NAT for FTP and TFTP built. That might cause problems for > some of the tests as well, but most of these may be skipped if > you > don't have userspace ftp programs installed. > > And if you'll build things as modules, instead of being builtin, > it > will be easier to make changes and test them. Also, some tests > might > not be happy if you can't reload the openvswitch module between > them. > It should not be a problem in theory, but it's hard to tell for > sure. > > In general, if you copy the default config from your Ubuntu > system > and use defaults for any new options, you shouldn't miss any > important > config opti
