I tested the package 22.03.8-0ubuntu0.22.04.2 from jammy-proposed and
the service stop path works as expected. Upon executing 'systemclt stop
ovn-controller' the corresponding chassis is removed from the
'chassis_private' and 'chassis' registers in the OVN SB db. After the
subsequent restart, the chassis registers itself in the SB db properly.
The NB db, 'lrp-get-gateway-chassis' shows it with the lowest priority,
which is expected.

Regarding the regression test, I conducted an extensive set of testing on 
Jammy, Noble, and Resolute multi-node environments. The test scenario was the 
same for all three releases:
- ping with 100 ms interval running from bastion to VM1 (north-south traffic)
- ping with 100 ms interval running from VM1 to VM2 (each VM on a separate 
compute node) (east-west traffic)
- at least five consecutive ovn-host and ovn-common package upgrade downgrades 
within the release

Tested package versions:
Jammy: 22.03.8-0ubuntu0.22.04.1 (pre-patch), 22.03.8-0ubuntu0.22.04.2 (with 
patch)
Noble: 24.03.2-0ubuntu0.24.04.2, 24.03.8-0ubuntu0.24.04.1
Resolute: 26.03.0-1 (from my PPA [3]), 26.03.0-2

During the tests on both Jammy and Noble I observed an individual ping packets 
loss during the ovn-controller restart sequence after the ovn-host package 
upgrade/downgrade.
After changing the external_ids:ovn-ofctrl-wait-before-clear to 5000 
milliseconds (the default is 0 [2]), and repeating the same set of tests 
(multiple repetitions), I haven't observed a single packet loss during the 
ovn-host package upgrade/downgrade. I haven't reproduce the data plane outage 
during the apt operations on Resolute running OVN 26.03. 
Considering the fact that the problem on Jammy and Noble was mitigated by 
setting high enough 'external_ids:ovn-ofctrl-wait-before-clear' and the fact 
that OVN 25.09 completely removed the support for this flag (now the OpenFlow 
removal wait time is dynamic and coordinated) [4], meaning Resolute using OVN 
26.03 bas 'buit-in' capability to delay crearing the flows upon controller 
restart, it is important to calculate the flow clear dealy properly (as 
documented in [2]). 
Reading the flag documentation [2] its sole purpose was to mitigate data plane 
outage during the ovn-controller restarts or upgrades. So it is expected that 
the flows are recalculated and reapplied during the controller startup. 
Finally, I was able to reproduce the same data plane packet loss on Jammy 
package version 22.03.8-0ubuntu0.22.04.1, which does not contain the fix by 
forcing package reinstall:

sudo apt install ovn-host=22.03.8-0ubuntu0.22.04.1 ovn-
common=22.03.8-0ubuntu0.22.04.1 ovn-central=22.03.8-0ubuntu0.22.04.1
--reinstall

On controller startup/reconnection, OVN recomputes its desired flows and 
installs the flow/group changes through ofctrl_put(). The initial deletion of 
existing flows and installation of replacement flows are included in the same 
OpenFlow bundle, using OFPBF_ORDERED | OFPBF_ATOMIC, and applied on bundle 
commit. Therefore, observing flow recreation inevitable but does not 
automatically mean the table flow is empty. Atomic installation does not, 
however, guarantee that the initially computed replacement flow set is 
complete. This is where for OVN releases pre 25.09 the 
'external_ids:ovn-ofctrl-wait-before-clear' flag comes into play.
The code links supporting the statement above:
Atomic bundle creation [5]
Initial flow/group deletion [6]
Replacement groups and flows [7]
Bundle commit [8]

Conclusions:
My testing found no evidence that commit [1] introduces a dataplane regression. 
Individual single-packet loss was observed during package operations both 
before and after the change on Jammy. It was also reproduced in a multi-node 
Noble deployment. The losses correlate with ovn-controller startup and are 
consistent with the documented behavior of 'ovn-ofctrl-wait-before-clear' 
(default 0). Configuring a 5000 ms delay avoided observed loss in repeated 
tests on both releases. Repeated multi-node Resolute tests with OVN 26.03, 
which includes the automatic startup coordination introduced in OVN 25.09, 
showed no packet loss. 

[1] 
https://git.launchpad.net/ubuntu/+source/ovn/commit/?h=ubuntu/jammy-proposed&id=72825ae75e969d042f83aa907794fe24337b24ec
[2] 
https://manpages.ubuntu.com/manpages/noble/man8/ovn-controller.8.html#external_ids:ovn~17
[3] https://launchpad.net/~wilkmarcin/+archive/ubuntu/lp2150130
[4] https://www.ovn.org/en/releases/25.09/
[5] 
https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2766
[6] 
https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2779
[7] 
https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2802
[8] 
https://git.launchpad.net/ubuntu/+source/ovn/tree/controller/ofctrl.c?id=99d62d21774b542066c585b86508c3ca35b439bc#n2884

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2150130

Title:
  --restart in ovn-controller leaves gateway ports active

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2150130/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to