focal-proposed verified using [Test Plan] and with the following output:
# apt-cache policy neutron-common
neutron-common:
Installed: 2:16.3.2-0ubuntu2
Candidate: 2:16.3.2-0ubuntu2
Version table:
*** 2:16.3.2-0ubuntu2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-proposed/main
amd64 Packages
100 /var/lib/dpkg/status
2:16.3.1-0ubuntu1.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main
amd64 Packages
2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64
Packages
$ grep kill_keepalived_monitor_py38 /etc/neutron/rootwrap.d/l3.filters
kill_keepalived_monitor_py38: KillFilter, root, python3.8, -15, -9
** Description changed:
[Impact]
+ Please see original bug description. Without this fix, the neutron-l3-agent
is unable to teardown an HA router and leaves it partially configured on every
node it was running on.
+
[Test Case]
- The victoria release of Openstack received patch [1] which allows the
neutron-l3-agent to SIGKILL or SIGTERM the keepalived-state-change monitor when
running under py38. This patch is needed in Ussuri for users running with py38
so we need to backport it.
+ * deploy Openstack ussuri on Ubuntu Focal
+ * enable L3 HA
+ * create a router and vm on network attached to router
+ * disable or delete the router and check for errors like the one below
+ * ensure that the following line exists in /etc/neutron/rootwrap.d/l3.filters:
+
+ kill_keepalived_monitor_py38: KillFilter, root, python3.8, -15, -9
+
+ -------------------------------------------------------------------------
+
+ The victoria release of Openstack received patch [1] which allows the
+ neutron-l3-agent to SIGKILL or SIGTERM the keepalived-state-change
+ monitor when running under py38. This patch is needed in Ussuri for
+ users running with py38 so we need to backport it.
The consequence of not having this is that you get the following when
you delete or disable a router:
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
[req-8c69af29-8f9c-4721-9cba-81ff4e9be92c - 9320f5ac55a04fb280d9ceb0b1106a6e -
- -] Error while deleting router ab63ccd8-1197-48d0-815e-31adc40e5193:
neutron_lib.exceptions.ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ;
Stderr: /usr/bin/neutron-rootwrap: Unauthorized command: kill -15 2516433 (no
filter matched)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 512, in
_safe_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self._router_removed(ri, router_id)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 548, in
_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.router_info[router_id] = ri
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.force_reraise()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
six.reraise(self.type_, self.value, self.tb)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent raise value
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 545, in
_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent ri.delete()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/dvr_edge_router.py", line 236,
in delete
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
super(DvrEdgeRouter, self).delete()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 492, in
delete
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.destroy_state_change_monitor(self.process_monitor)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 438, in
destroy_state_change_monitor
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
pm.disable(sig=str(int(signal.SIGTERM)))
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/linux/external_process.py", line
113, in disable
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
utils.execute(cmd, run_as_root=self.run_as_root)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/linux/utils.py", line 147, in
execute
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent raise
exceptions.ProcessExecutionError(msg,
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
neutron_lib.exceptions.ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ;
Stderr: /usr/bin/neutron-rootwrap: Unauthorized command: kill -15 2516433 (no
filter matched)
Which results in the router being deleted from neutron but not the node.
In my case i had both a qrouter and snat ns left with IPs still
configured as well as my fip ip rule allocation still present in
/var/lib/neutron/fip-priorities
[1]
https://github.com/openstack/neutron/commit/4fb505891ee32ae41247f1d7a48b7455b342840e
[Regression Potential]
This change is backported from the stable/victoria release to authorize
cleaning up of keepalived-state-chane via rootwrap [1] when running under
python3.8. Where things can go wrong with l3.filters would be in the form of
filter mistakes that allow or disallow running the intended command. In this
case the code is picked straight from what is in stable/victoria and above and
has already been tested by Ed, so it appears to have very low regression
potential.
[1] https://wiki.openstack.org/wiki/Rootwrap
** Description changed:
[Impact]
Please see original bug description. Without this fix, the neutron-l3-agent
is unable to teardown an HA router and leaves it partially configured on every
node it was running on.
- [Test Case]
+ [Test Plan]
* deploy Openstack ussuri on Ubuntu Focal
* enable L3 HA
* create a router and vm on network attached to router
* disable or delete the router and check for errors like the one below
* ensure that the following line exists in /etc/neutron/rootwrap.d/l3.filters:
kill_keepalived_monitor_py38: KillFilter, root, python3.8, -15, -9
-------------------------------------------------------------------------
The victoria release of Openstack received patch [1] which allows the
neutron-l3-agent to SIGKILL or SIGTERM the keepalived-state-change
monitor when running under py38. This patch is needed in Ussuri for
users running with py38 so we need to backport it.
The consequence of not having this is that you get the following when
you delete or disable a router:
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
[req-8c69af29-8f9c-4721-9cba-81ff4e9be92c - 9320f5ac55a04fb280d9ceb0b1106a6e -
- -] Error while deleting router ab63ccd8-1197-48d0-815e-31adc40e5193:
neutron_lib.exceptions.ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ;
Stderr: /usr/bin/neutron-rootwrap: Unauthorized command: kill -15 2516433 (no
filter matched)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 512, in
_safe_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self._router_removed(ri, router_id)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 548, in
_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.router_info[router_id] = ri
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.force_reraise()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
six.reraise(self.type_, self.value, self.tb)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent raise value
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 545, in
_router_removed
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent ri.delete()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/dvr_edge_router.py", line 236,
in delete
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
super(DvrEdgeRouter, self).delete()
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 492, in
delete
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
self.destroy_state_change_monitor(self.process_monitor)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 438, in
destroy_state_change_monitor
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
pm.disable(sig=str(int(signal.SIGTERM)))
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/linux/external_process.py", line
113, in disable
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
utils.execute(cmd, run_as_root=self.run_as_root)
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/linux/utils.py", line 147, in
execute
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent raise
exceptions.ProcessExecutionError(msg,
2021-05-26 02:11:44.653 3457514 ERROR neutron.agent.l3.agent
neutron_lib.exceptions.ProcessExecutionError: Exit code: 99; Stdin: ; Stdout: ;
Stderr: /usr/bin/neutron-rootwrap: Unauthorized command: kill -15 2516433 (no
filter matched)
Which results in the router being deleted from neutron but not the node.
In my case i had both a qrouter and snat ns left with IPs still
configured as well as my fip ip rule allocation still present in
/var/lib/neutron/fip-priorities
[1]
https://github.com/openstack/neutron/commit/4fb505891ee32ae41247f1d7a48b7455b342840e
[Regression Potential]
This change is backported from the stable/victoria release to authorize
cleaning up of keepalived-state-chane via rootwrap [1] when running under
python3.8. Where things can go wrong with l3.filters would be in the form of
filter mistakes that allow or disallow running the intended command. In this
case the code is picked straight from what is in stable/victoria and above and
has already been tested by Ed, so it appears to have very low regression
potential.
[1] https://wiki.openstack.org/wiki/Rootwrap
** Tags removed: verification-needed-focal
** Tags added: verification-done-focal
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1929832
Title:
stable/ussuri py38 support for keepalived-state-change monitor
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1929832/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
