On Tue, May 27, 2025 at 01:42:53PM +0200, Felix Huettner via dev wrote: > On Wed, May 21, 2025 at 01:23:28PM +0200, Ales Musil via dev wrote: > > On Mon, May 19, 2025 at 2:48 PM Felix Huettner > > <felix.huettner@stackit.cloud> > > wrote: > > > > > On Tue, May 13, 2025 at 10:09:10AM +0200, Ales Musil wrote: > > > > On Tue, May 13, 2025 at 9:53 AM Felix Huettner > > > <felix.huettner@stackit.cloud> > > > > wrote: > > > > > > > > > On Mon, May 12, 2025 at 03:55:10PM +0200, Ales Musil wrote: > > > > > > On Mon, May 12, 2025 at 3:29 PM Felix Huettner > > > > > <felix.huettner@stackit.cloud> > > > > > > wrote: > > > > > > > > > > > > > On Mon, May 12, 2025 at 10:59:27AM +0200, Ales Musil wrote: > > > > > > > > On Wed, May 7, 2025 at 7:00 PM Ales Musil <amu...@redhat.com> > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 2, 2025 at 7:48 PM Mark Michelson < > > > mmich...@redhat.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Thanks Felix. > > > > > > > > >> > > > > > > > > >> Acked-by: Mark Michelson <mmich...@redhat.com> > > > > > > > > >> > > > > > > > > >> On 4/30/25 03:37, Felix Huettner via dev wrote: > > > > > > > > >> > This adds an additional testcase where the tracked_port of > > > the > > > > > > > > >> > advertised routes is referencing a normal lsp, instead of a > > > lsp > > > > > > > > >> > connected to an lrp. > > > > > > > > >> > > > > > > > > > >> > Signed-off-by: Felix Huettner <felix.huettner@stackit.cloud > > > > > > > > > > > > >> > --- > > > > > > > > >> > tests/system-ovn.at | 73 > > > > > > > > >> +++++++++++++++++++++++++++++++++++++++++++-- > > > > > > > > >> > 1 file changed, 71 insertions(+), 2 deletions(-) > > > > > > > > >> > > > > > > > > > >> > diff --git a/tests/system-ovn.at b/tests/system-ovn.at > > > > > > > > >> > index 6e71286ad..359d8b5a3 100644 > > > > > > > > >> > --- a/tests/system-ovn.at > > > > > > > > >> > +++ b/tests/system-ovn.at > > > > > > > > >> > @@ -16063,6 +16063,14 @@ check ovn-nbctl lsp-add p2 vif2 \ > > > > > > > > >> > -- lsp-set-addresses vif2 "00:00:ff:ff:ff:02 > > > > > 198.51.100.10" > > > > > > > > >> > check ovn-nbctl lr-route-add internet 198.51.100.0/24 > > > > > 192.0.2.3 > > > > > > > > >> > > > > > > > > > >> > +# Setup lsp "vif3" on ls public > > > > > > > > >> > +check ovn-nbctl lsp-add public vif3 \ > > > > > > > > >> > + -- lsp-set-addresses vif3 "00:00:ff:ff:ff:03 > > > 192.0.2.20" \ > > > > > > > > >> > + -- set Logical_Switch_Port vif3 > > > > > options:requested-chassis=hv1 > > > > > > > > >> > +check ovs-vsctl add-port br-int vif3 \ > > > > > > > > >> > + -- set Interface vif3 external-ids:iface-id=vif3 \ > > > > > > > > >> > + -- set Interface vif3 type=internal > > > > > > > > >> > + > > > > > > > > >> > # Configure external connectivity. > > > > > > > > >> > check ovs-vsctl set Open_vSwitch . > > > > > > > > >> external-ids:ovn-bridge-mappings=phynet:br-ext > > > > > > > > >> > check ovn-nbctl lsp-add phys phys1 \ > > > > > > > > >> > @@ -16071,7 +16079,7 @@ check ovn-nbctl lsp-add phys phys1 > > > > > > > > >> > \ > > > > > > > > >> > -- lsp-set-options phys1 network_name=phynet > > > > > > > > >> > > > > > > > > > >> > check ovn-nbctl --wait=hv sync > > > > > > > > >> > -wait_for_ports_up public-internet phys-internet public-pr1 > > > > > p1-pr1 > > > > > > > > >> public-pr2 p2-pr2 > > > > > > > > >> > +wait_for_ports_up public-internet phys-internet public-pr1 > > > > > p1-pr1 > > > > > > > > >> public-pr2 p2-pr2 vif3 > > > > > > > > >> > > > > > > > > > >> > # Now the ovn-controller should have setup a vrf named > > > > > > > "ovnvrf1337". > > > > > > > > >> > # It should contain routes for: > > > > > > > > >> > @@ -16092,6 +16100,7 @@ blackhole 198.51.100.0/24 proto ovn > > > > > metric > > > > > > > > >> 1000]) > > > > > > > > >> > # * 192.0.2.2/32 > > > > > > > > >> > # * 192.0.2.3/32 > > > > > > > > >> > # * 192.0.2.10/32 > > > > > > > > >> > +# * 192.0.2.20/32 > > > > > > > > >> > # The last 3 of them are local to the current chassis so > > > we > > > > > > > expect a > > > > > > > > >> better > > > > > > > > >> > # prio. > > > > > > > > >> > check ovn-nbctl --wait=hv set Logical_Router_Port > > > > > internet-public > > > > > > > \ > > > > > > > > >> > @@ -16102,6 +16111,7 @@ blackhole 192.0.2.1 proto ovn > > > > > > > > >> > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # If the pr1-public lrp is now removed from this > > > hypervisor > > > > > the > > > > > > > route > > > > > > > > >> metric > > > > > > > > >> > @@ -16114,6 +16124,7 @@ blackhole 192.0.2.1 proto ovn > > > > > > > > >> > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 1000 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 1000 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # Moving pr1-public back will also change the route > > > metrics > > > > > again. > > > > > > > > >> > @@ -16124,6 +16135,27 @@ blackhole 192.0.2.1 proto ovn > > > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > +blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > + > > > > > > > > >> > +# Moving vif3 away from hv1 will change the route metric > > > > > > > > >> > to > > > > > > > default. > > > > > > > > >> > +check ovn-nbctl --wait=hv set Logical_Switch_Port vif3 > > > > > > > > >> options:requested-chassis=thisdoesnotexist > > > > > > > > >> > +OVN_ROUTE_EQUAL([ovnvrf1337], [dnl > > > > > > > > >> > +blackhole 192.0.2.1 proto ovn metric 1000 > > > > > > > > >> > +blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 1000 > > > > > > > > >> > +blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > + > > > > > > > > >> > +# Bringing vif3 back will change the route metric to local > > > > > again. > > > > > > > > >> > +check ovn-nbctl --wait=hv set Logical_Switch_Port vif3 > > > > > > > > >> options:requested-chassis=hv1 > > > > > > > > >> > +OVN_ROUTE_EQUAL([ovnvrf1337], [dnl > > > > > > > > >> > +blackhole 192.0.2.1 proto ovn metric 1000 > > > > > > > > >> > +blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # Disable maintaining the vrf as some tests below would > > > cause > > > > > the > > > > > > > > >> vrfs to be > > > > > > > > >> > @@ -16191,6 +16223,7 @@ blackhole 192.0.2.1 proto ovn > > > > > > > > >> > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000 > > > > > > > > >> > 233.252.0.0/24 via 192.168.10.10 dev lo onlink > > > > > > > > >> > 233.253.0.0/24 via 192.168.20.20 dev hv1-mll onlink]) > > > > > > > > >> > @@ -16207,6 +16240,7 @@ blackhole 192.0.2.1 proto ovn > > > > > > > > >> > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000 > > > > > > > > >> > 233.252.0.0/24 via 192.168.10.10 dev lo onlink > > > > > > > > >> > 233.253.0.0/24 via 192.168.20.20 dev hv1-mll onlink]) > > > > > > > > >> > @@ -16219,6 +16253,7 @@ blackhole 192.0.2.1 proto ovn > > > > > > > > >> > metric > > > > > 1000 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000 > > > > > > > > >> > 233.252.0.0/24 via 192.168.10.10 dev lo onlink > > > > > > > > >> > 233.253.0.0/24 via 192.168.20.20 dev hv1-mll onlink]) > > > > > > > > >> > @@ -16389,6 +16424,14 @@ check ovn-nbctl lsp-add p2 vif2 \ > > > > > > > > >> > -- lsp-set-addresses vif2 "00:00:ff:ff:ff:02 > > > > > 198.51.100.10" > > > > > > > > >> > check ovn-nbctl lr-route-add internet 198.51.100.0/24 > > > > > 192.0.2.3 > > > > > > > > >> > > > > > > > > > >> > +# Setup lsp "vif3" on ls public > > > > > > > > >> > +check ovn-nbctl lsp-add public vif3 \ > > > > > > > > >> > + -- lsp-set-addresses vif3 "00:00:ff:ff:ff:03 > > > 192.0.2.20" \ > > > > > > > > >> > + -- set Logical_Switch_Port vif3 > > > > > options:requested-chassis=hv1 > > > > > > > > >> > +check ovs-vsctl add-port br-int vif3 \ > > > > > > > > >> > + -- set Interface vif3 external-ids:iface-id=vif3 \ > > > > > > > > >> > + -- set Interface vif3 type=internal > > > > > > > > >> > + > > > > > > > > >> > # Configure external connectivity. > > > > > > > > >> > check ovs-vsctl set Open_vSwitch . > > > > > > > > >> external-ids:ovn-bridge-mappings=phynet:br-ext > > > > > > > > >> > check ovn-nbctl lsp-add phys phys1 \ > > > > > > > > >> > @@ -16397,7 +16440,7 @@ check ovn-nbctl lsp-add phys phys1 > > > > > > > > >> > \ > > > > > > > > >> > -- lsp-set-options phys1 network_name=phynet > > > > > > > > >> > > > > > > > > > >> > check ovn-nbctl --wait=hv sync > > > > > > > > >> > -wait_for_ports_up public-internet phys-internet public-pr1 > > > > > p1-pr1 > > > > > > > > >> public-pr2 p2-pr2 > > > > > > > > >> > +wait_for_ports_up public-internet phys-internet public-pr1 > > > > > p1-pr1 > > > > > > > > >> public-pr2 p2-pr2 vif3 > > > > > > > > >> > > > > > > > > > >> > # Now the ovn-controller should have setup a vrf named > > > > > > > "ovnvrf1337". > > > > > > > > >> > # It should contain routes for: > > > > > > > > >> > @@ -16418,6 +16461,7 @@ blackhole 198.51.100.0/24 proto ovn > > > > > metric > > > > > > > > >> 1000]) > > > > > > > > >> > # * 192.0.2.2/32 > > > > > > > > >> > # * 192.0.2.3/32 > > > > > > > > >> > # * 192.0.2.10/32 > > > > > > > > >> > +# * 192.0.2.20/32 > > > > > > > > >> > # The last 3 of them are local to the current chassis so > > > we > > > > > > > expect a > > > > > > > > >> better > > > > > > > > >> > # prio. > > > > > > > > >> > check ovn-nbctl --wait=hv set Logical_Router_Port > > > > > internet-public > > > > > > > \ > > > > > > > > >> > @@ -16428,6 +16472,7 @@ blackhole 192.0.2.1 proto ovn > > > metric 100 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # If the pr1-public lrp is now removed from this > > > hypervisor > > > > > the > > > > > > > route > > > > > > > > >> metric > > > > > > > > >> > @@ -16440,6 +16485,7 @@ blackhole 192.0.2.1 proto ovn > > > metric 100 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 1000 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 1000 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # Moving pr1-public back will also change the route > > > metrics > > > > > again. > > > > > > > > >> > @@ -16450,6 +16496,27 @@ blackhole 192.0.2.1 proto ovn > > > metric > > > > > 100 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > +blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > + > > > > > > > > >> > +# Moving vif3 away from hv1 will change the route metric > > > > > > > > >> > to > > > > > > > default. > > > > > > > > >> > +check ovn-nbctl --wait=hv set Logical_Switch_Port vif3 > > > > > > > > >> options:requested-chassis=thisdoesnotexist > > > > > > > > >> > +OVN_ROUTE_EQUAL([ovnvrf1337], [dnl > > > > > > > > >> > +blackhole 192.0.2.1 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 1000 > > > > > > > > >> > +blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > + > > > > > > > > >> > +# Bringing vif3 back will change the route metric to local > > > > > again. > > > > > > > > >> > +check ovn-nbctl --wait=hv set Logical_Switch_Port vif3 > > > > > > > > >> options:requested-chassis=hv1 > > > > > > > > >> > +OVN_ROUTE_EQUAL([ovnvrf1337], [dnl > > > > > > > > >> > +blackhole 192.0.2.1 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000]) > > > > > > > > >> > > > > > > > > > >> > # Disable maintaining the vrf as some tests below would > > > cause > > > > > the > > > > > > > > >> vrfs to be > > > > > > > > >> > @@ -16517,6 +16584,7 @@ blackhole 192.0.2.1 proto ovn > > > metric 100 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000 > > > > > > > > >> > 233.252.0.0/24 via 192.168.10.10 dev lo onlink > > > > > > > > >> > 233.253.0.0/24 via 192.168.20.20 dev hv1-mll onlink]) > > > > > > > > >> > @@ -16529,6 +16597,7 @@ blackhole 192.0.2.1 proto ovn > > > metric 100 > > > > > > > > >> > blackhole 192.0.2.2 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.3 proto ovn metric 100 > > > > > > > > >> > blackhole 192.0.2.10 proto ovn metric 100 > > > > > > > > >> > +blackhole 192.0.2.20 proto ovn metric 100 > > > > > > > > >> > blackhole 198.51.100.0/24 proto ovn metric 1000 > > > > > > > > >> > 233.252.0.0/24 via 192.168.10.10 dev lo onlink > > > > > > > > >> > 233.253.0.0/24 via 192.168.20.20 dev hv1-mll onlink]) > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Thanks Felix and Mark, > > > > > > > > > > > > > > > > > > I applied this to main and branch-25.03. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Ales > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Felix, > > > > > > > > > > > > > > > > the test became pretty flaky after the merge, would you have > > > > > > > > some > > > > > time to > > > > > > > > investigate why is that a case? > > > > > > > > > > > > > > Hi Ales, > > > > > > > > > > > > > > sure. Could you point me to a ci failure? > > > > > > > > > > > > > > > > > > > I couldn't find any upstream failure (I thought there was one). > > > > > > > > > > > > I just tried it locally and could not reproduce any issue. > > > > > > > > > > > > > > > > > > > I was able to reproduce it locally after 1-3 iterations of the > > > updated > > > > > > tests. For me the difference was made by using userspace datapath > > > instead > > > > > > of kernel one. > > > > > > > > > > I just tried with `check-system-userspace TESTSUITEFLAGS="-k > > > > > dynamic-routing"` > > > > > but i could not reproduce it within 30 runs. > > > > > > > > > > Could you maybe send me your test outputs directory? Then i would > > > > > check > > > > > if i see differences there. > > > > > > > > > > > > > Attached one of the failed runs, this one actually has 2 failed tests. > > > > > > > > Hopefully this will help you a bit with identification of what might be > > > > wrong. > > > > > > Hi Ales, > > > > > > > Hi Felix, > > > > > > > i just had a look at the logs and compared them with my working ones. > > > But i did not see any changes in comparison to my logs. > > > > > > Could you maybe do a run with "route_exchange_netlink" and > > > "route_exchange" set to debug? > > > Then we should at least see when it removes the route and adds the other > > > one. > > > > > > > Sorry I didn't have time earlier to try it out again. > > I have attached failures with the additional logging. Hopefully it helps. > > Hi Ales, > > sorry for taking so long to take a look. > However i don't see any attached file. > > But with your configure options i could finally reproduce this on the > testcase: > "dynamic-routing - Gateway Router -- parallelization=yes -- > ovn_monitor_all=no" > > I'll take a look and report back with my findings.
Hi Ales, i think i found the issue and a fix for it. Could you maybe validate that on your side, just to rule out the possibility that it just breaks my reproduction. Commit is at: https://github.com/felixhuettner/ovn/commit/3c45f540945523437349be374c03e2ab07cdbf07 Also i am not really sure if i completely understood the problem we face here: It seems for me that the local IDL might leave optional references set to NULL if the referenced row has not yet been loaded locally. Once we get the initial update from southbound for this row the IDL would then fixup the reference and treat the referencing row as updated. I got this feeling because for the Advertised_Route of 192.0.2.20 that you had problems with did not run into this if condition here: https://github.com/felixhuettner/ovn/blob/fix_ar_handler/controller/route.c#L257 And as the Advertised_Route is never updated i could not think of any other way how we could run into this. Is this understanding correct or might there be something else that causes such behaviour? Thanks a lot, Felix > > Thanks a lot, > Felix > > > > > > > > For me it feels a little like the "route" node gets the event that vif3 > > > is no longer claimed locally and changes to the 1000 metric route. > > > But it then misses the event of vif3 being claimed again so we do not > > > update the metric to 100. > > > > > > But the debug logs should hopefully help. > > > > > > Thanks a lot, > > > Felix > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > Felix > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Felix > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Ales > > > > > > > > > > > > > > > > > > > > Hopefully this helps. > > > > > > > > > > > > Thanks, > > > > > > Ales > > > > > > > > > > > > > > Thanks, > > > > Ales > > > > > > > > Thanks, > > Ales > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
