Verification performed on the same three-node Ubuntu 22.04.5 LTS
hyperconverged regress-stack lab, upgrading from the verified Zed stage to
Antelope. The candidate OVN packages were
23.03.1-1ubuntu0.23.04.1~cloud4 from jammy-proposed/antelope; OVS was
3.1.3-0ubuntu0.23.04.1~cloud2.
The OVN packages were installed on each host with automatic package service
restarts suppressed. On the first host, `ovn-controller` stopped during
installation despite that policy and was started immediately; northd and both
database PIDs remained unchanged. Controllers were running the new version
on all three hosts before the OVN databases/northd were restarted one node at
a time. OpenStack packages, database schemas, and services were upgraded
afterward. Nova and Cinder online migrations, the Neutron offline-migration
check, and `regress-stack ready` passed.
Each host's Southbound flow query found `flags.from_ctrl` and both ingress
and egress `ct_commit_nat` rules. OVN services were active, `br-int` had
installed flows, and ovn-controller parser-error scans were empty. The three
original Cirros VMs retained their IDs and boot IDs and passed DNS,
east-west, and floating-IP checks after the upgrade and after Tempest. The
continuous flow watcher recorded no missing target rule. Guest pings passed
at the checkpoints between central-node restarts; no zero-downtime claim is
made for the package installation windows.
The focused Tempest Nova server API and Neutron network API run selected 122
tests: 120 passed, 2 skipped, 0 failed (Nova: 57 passed, 2 skipped; Neutron:
63 passed).
LP #2115328 UCA ANTELOPE VERIFICATION EVIDENCE
Source: three-node regress-stack hyperconverged lab; collected 2026-09-16 UTC.
OS: Ubuntu 22.04.5 LTS
Hosts: rsov2115328-r2hc1, rsov2115328-r2hc2, rsov2115328-r2hc3.
Excluded services: Heat, Watcher, Magnum, Octavia.
STEP 1: CANDIDATE PROVENANCE AND PACKAGE VERSIONS
Inst ovn-host [22.09.1-0ubuntu0.22.10.1~cloud2]
(23.03.1-1ubuntu0.23.04.1~cloud4 jammy-proposed [amd64]) []
Inst ovn-central [22.09.1-0ubuntu0.22.10.1~cloud2]
(23.03.1-1ubuntu0.23.04.1~cloud4 jammy-proposed [amd64]) []
Inst ovn-common [22.09.1-0ubuntu0.22.10.1~cloud2]
(23.03.1-1ubuntu0.23.04.1~cloud4 jammy-proposed [amd64])
Baseline hc1 OVN/OVS:
openvswitch-switch 3.0.3-0ubuntu0.22.10.3~cloud3
ovn-central 22.09.1-0ubuntu0.22.10.1~cloud2
ovn-common 22.09.1-0ubuntu0.22.10.1~cloud2
ovn-host 22.09.1-0ubuntu0.22.10.1~cloud2
Installed rsov2115328-r2hc1:
neutron-server 2:22.1.0-0ubuntu1~cloud1
nova-compute 3:27.4.0-0ubuntu1~cloud0
openvswitch-switch 3.1.3-0ubuntu0.23.04.1~cloud2
ovn-central 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-common 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-host 23.03.1-1ubuntu0.23.04.1~cloud4
Installed rsov2115328-r2hc2:
neutron-server 2:22.1.0-0ubuntu1~cloud1
nova-compute 3:27.4.0-0ubuntu1~cloud0
openvswitch-switch 3.1.3-0ubuntu0.23.04.1~cloud2
ovn-central 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-common 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-host 23.03.1-1ubuntu0.23.04.1~cloud4
Installed rsov2115328-r2hc3:
neutron-server 2:22.1.0-0ubuntu1~cloud1
nova-compute 3:27.4.0-0ubuntu1~cloud0
openvswitch-switch 3.1.3-0ubuntu0.23.04.1~cloud2
ovn-central 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-common 23.03.1-1ubuntu0.23.04.1~cloud4
ovn-host 23.03.1-1ubuntu0.23.04.1~cloud4
STEP 2: SERVICE, LOGICAL FLOW, AND OVS GATES
rsov2115328-r2hc1: services=['active', 'active', 'active', 'active', 'active'];
flow-gate=true; flags.from_ctrl=1; ct_commit_nat ingress=1, egress=1; br-int
lines=630; broad parser matches=0
rsov2115328-r2hc2: services=['active', 'active', 'active', 'active', 'active'];
flow-gate=true; flags.from_ctrl=1; ct_commit_nat ingress=1, egress=1; br-int
lines=644; broad parser matches=0
rsov2115328-r2hc3: services=['active', 'active', 'active', 'active', 'active'];
flow-gate=true; flags.from_ctrl=1; ct_commit_nat ingress=1, egress=1; br-int
lines=630; broad parser matches=0
STEP 3: READINESS AND ORIGINAL GUESTS
Deployment ready; workload/failure acceptance is a separate harness test.
Original server IDs equal Yoga: True; original port IDs equal Yoga: True
Guest check completion marker: True
vm-1: boot ID equal Yoga=True; DNS A expected=True; east-west 0% loss=True;
floating-IP 0% loss=True
vm-2: boot ID equal Yoga=True; DNS A expected=True; east-west 0% loss=True;
floating-IP 0% loss=True
vm-3: boot ID equal Yoga=True; DNS A expected=True; east-west 0% loss=True;
floating-IP 0% loss=True
STEP 4: TEMPEST RESULT
Command: tempest run --load-list nova-neutron-tests.txt --concurrency 4
Selection regex:
tempest.api.compute.servers.test_servers|tempest.api.network.test_networks
Selected test-list lines: 122
Ran: 122 tests in 840.7822 sec.
- Passed: 120
- Skipped: 2
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
** Tags removed: verification-antelope-needed
** Tags added: verification-antelope-done
** Description changed:
[Impact]
Updated OVN 22.03 packages in Jammy generate Southbound logical flows using
the ct_commit_nat action and flags.from_ctrl field. The original OVN
22.09.1 packages in Zed do not understand these constructs.
During the upgrade, ovn-controller rejects the affected logical flows,
logging:
error parsing actions "ct_commit_nat;": Syntax error at
`ct_commit_nat' expecting action.
error parsing match "flags.from_ctrl && udp.src == 53": Syntax error at
`flags.from_ctrl' expecting field name.
This prevents the affected flows from being installed and can disrupt
network traffic during the upgrade.
[Test Plan]
- Deploy a Jammy Charmed OpenStack cloud using updated OVN 22.03 packages.
+ Deploy a multinode Jammy Yoga OpenStack cloud using updated OVN 22.03
packages.
Configure instances on different compute hosts, security groups, floating
- IPs, OVN DNS records, and an OVN load balancer.
+ IPs, OVN DNS records.
Confirm that the Southbound database contains flows referencing both
ct_commit_nat and flags.from_ctrl.
Verify that:
* All relevant units install the candidate package and their OVN
services remain healthy.
* No ct_commit_nat or flags.from_ctrl parsing errors appear in the
ovn-controller logs during the upgrade.
* Controllers converge and process subsequent network configuration.
* Existing and newly created instances have working east-west and
floating-IP connectivity.
* OVN DNS resolution works with security groups enabled.
* Related ICMP traffic through an OVN load balancer is delivered with
the correct NAT translation.
- Run the Charmed OpenStack functional and upgrade tests and attach the
+ Run the OpenStack functional and upgrade tests and attach the
results, package versions, and relevant logs before marking verification
complete.
[Where problems could occur]
The patches change logical action parsing, load-balancer handling of
related traffic, and the ACL treatment of OVN-generated DNS responses.
Regressions could affect connection tracking or NAT for related ICMP
traffic, prevent logical flows from being installed, or incorrectly allow
or block DNS traffic.
Validation must cover both existing connections and new network
configuration after the upgrade. DNS testing must confirm that legitimate
OVN-generated replies work while workload-generated traffic remains
subject to the configured security-group rules.
[Other Info]
---
The charmed OpenStack product currently make use of a step by step
upgrade process.
This includes stepping through interim releases of OVN.
While OVN itself supports direct upgrades between LTS versions, it is
impractical in the current organization of apt repositories.
Between 22.03 and 22.04 releases, two new actions has been introduced
and backported to resolve critical issues.
These patches have made their way into the ovn 22.03 package in Jammy,
but not to the intermediate versions currently maintained in UCA. As a
consequence, when users attempt to upgrade they will hit an data path
impacting issue, and find messages below logged by the ovn-controller:
2025-06-19T06:14:46.085Z|00025|lflow|WARN|error parsing actions
"ct_commit_nat;": Syntax error at `ct_commit_nat' expecting action.
2025-06-19T06:14:46.089Z|00026|lflow|WARN|error parsing match
"flags.from_ctrl && udp.src == 53": Syntax error at `flags.from_ctrl' expecting
field name.
2025-06-19T06:22:56.626Z|00032|lflow|WARN|error parsing actions
"ct_commit_nat;": Syntax error at `ct_commit_nat' expecting action.
2025-06-19T06:22:56.628Z|00034|lflow|WARN|error parsing match
"flags.from_ctrl && udp.src == 53": Syntax error at `flags.from_ctrl' expecting
field name.
The required patches to resolve this are:
474bdfcad038 ("Skip only OVN DNS responder packets from OUT_ACL.")
650f06b9f3e2 ("northd: Allow related traffic through LB")
31196346fcad ("actions: Add new action called ct_commit_nat")
** Description changed:
[Impact]
Updated OVN 22.03 packages in Jammy generate Southbound logical flows using
the ct_commit_nat action and flags.from_ctrl field. The original OVN
22.09.1 packages in Zed do not understand these constructs.
During the upgrade, ovn-controller rejects the affected logical flows,
logging:
error parsing actions "ct_commit_nat;": Syntax error at
`ct_commit_nat' expecting action.
error parsing match "flags.from_ctrl && udp.src == 53": Syntax error at
`flags.from_ctrl' expecting field name.
This prevents the affected flows from being installed and can disrupt
network traffic during the upgrade.
[Test Plan]
Deploy a multinode Jammy Yoga OpenStack cloud using updated OVN 22.03
packages.
Configure instances on different compute hosts, security groups, floating
IPs, OVN DNS records.
Confirm that the Southbound database contains flows referencing both
ct_commit_nat and flags.from_ctrl.
Verify that:
* All relevant units install the candidate package and their OVN
services remain healthy.
* No ct_commit_nat or flags.from_ctrl parsing errors appear in the
ovn-controller logs during the upgrade.
* Controllers converge and process subsequent network configuration.
* Existing and newly created instances have working east-west and
floating-IP connectivity.
* OVN DNS resolution works with security groups enabled.
* Related ICMP traffic through an OVN load balancer is delivered with
the correct NAT translation.
Run the OpenStack functional and upgrade tests and attach the
results, package versions, and relevant logs before marking verification
complete.
[Where problems could occur]
The patches change logical action parsing, load-balancer handling of
related traffic, and the ACL treatment of OVN-generated DNS responses.
Regressions could affect connection tracking or NAT for related ICMP
traffic, prevent logical flows from being installed, or incorrectly allow
or block DNS traffic.
Validation must cover both existing connections and new network
configuration after the upgrade. DNS testing must confirm that legitimate
OVN-generated replies work while workload-generated traffic remains
subject to the configured security-group rules.
[Other Info]
+ After discussion with other Cloud Archive admin, we've deemed that a multi
node deployment testing yoga -> zed -> antelope -> bobcat upgrade test is
sufficient to validate these patches, and does not require a full charmed
openstack deployment. Updated the original testplan, and keep this comment for
history.
- ---
+ --- Original Description
The charmed OpenStack product currently make use of a step by step
upgrade process.
This includes stepping through interim releases of OVN.
While OVN itself supports direct upgrades between LTS versions, it is
impractical in the current organization of apt repositories.
Between 22.03 and 22.04 releases, two new actions has been introduced
and backported to resolve critical issues.
These patches have made their way into the ovn 22.03 package in Jammy,
but not to the intermediate versions currently maintained in UCA. As a
consequence, when users attempt to upgrade they will hit an data path
impacting issue, and find messages below logged by the ovn-controller:
2025-06-19T06:14:46.085Z|00025|lflow|WARN|error parsing actions
"ct_commit_nat;": Syntax error at `ct_commit_nat' expecting action.
2025-06-19T06:14:46.089Z|00026|lflow|WARN|error parsing match
"flags.from_ctrl && udp.src == 53": Syntax error at `flags.from_ctrl' expecting
field name.
2025-06-19T06:22:56.626Z|00032|lflow|WARN|error parsing actions
"ct_commit_nat;": Syntax error at `ct_commit_nat' expecting action.
2025-06-19T06:22:56.628Z|00034|lflow|WARN|error parsing match
"flags.from_ctrl && udp.src == 53": Syntax error at `flags.from_ctrl' expecting
field name.
The required patches to resolve this are:
474bdfcad038 ("Skip only OVN DNS responder packets from OUT_ACL.")
650f06b9f3e2 ("northd: Allow related traffic through LB")
31196346fcad ("actions: Add new action called ct_commit_nat")
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2115328
Title:
[uca][zed][antelope][bobcat] Upgrade failure due to missing patches
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2115328/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
