Public bug reported:
On upgrading neutron on a compute node, instances on that node wound up
losing some of their network plumbing via openvswitch.
This cloud: Mitaka, xenial, openvswitch with gre.
2017-07-05 16:17:52, the following auto-upgrade occurred: neutron-
common:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), neutron-plugin-
openvswitch-agent:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), neutron-
openvswitch-agent:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), python-
neutron:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3)
The neutron logs:
2017-07-05 16:17:53.670 6156 CRITICAL neutron
[req-1fcb315b-9f12-4657-aea8-1463f55f0106 - - - - -] ProcessExecutionError:
Exit code: -15; Stdin: ; Stdout: ; Stderr: Signal 15 (TERM) caught by ps
(procps-ng version 3.3.10).
ps:display.c:66: please report this bug
2017-07-05 16:17:53.670 6156 ERROR neutron Traceback (most recent call last):
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2017-07-05 16:17:53.670 6156 ERROR neutron sys.exit(main())
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py",
line 20, in main
2017-07-05 16:17:53.670 6156 ERROR neutron agent_main.main()
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py",
line 49, in main
2017-07-05 16:17:53.670 6156 ERROR neutron mod.main()
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py",
line 36, in main
2017-07-05 16:17:53.670 6156 ERROR neutron
ovs_neutron_agent.main(bridge_classes)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2164, in main
2017-07-05 16:17:53.670 6156 ERROR neutron agent.daemon_loop()
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2087, in daemon_loop
2017-07-05 16:17:53.670 6156 ERROR neutron self.rpc_loop(polling_manager=pm)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2000, in rpc_loop
2017-07-05 16:17:53.670 6156 ERROR neutron if
(self._agent_has_updates(polling_manager) or sync
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1752, in _agent_has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron return
(polling_manager.is_polling_required or
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/common/base_polling.py", line
35, in is_polling_required
2017-07-05 16:17:53.670 6156 ERROR neutron polling_required =
self._is_polling_required()
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 69, in
_is_polling_required
2017-07-05 16:17:53.670 6156 ERROR neutron return self._monitor.has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovsdb_monitor.py", line
74, in has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron if not self.is_active():
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line
100, in is_active
2017-07-05 16:17:53.670 6156 ERROR neutron self.pid,
self.cmd_without_namespace)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line
159, in pid
2017-07-05 16:17:53.670 6156 ERROR neutron run_as_root=self.run_as_root)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 267, in
get_root_helper_child_pid
2017-07-05 16:17:53.670 6156 ERROR neutron pid = find_child_pids(pid)[0]
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 202, in
find_child_pids
2017-07-05 16:17:53.670 6156 ERROR neutron return []
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-05 16:17:53.670 6156 ERROR neutron self.force_reraise()
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-07-05 16:17:53.670 6156 ERROR neutron six.reraise(self.type_,
self.value, self.tb)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 194, in
find_child_pids
2017-07-05 16:17:53.670 6156 ERROR neutron log_fail_as_error=False)
2017-07-05 16:17:53.670 6156 ERROR neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 146, in
execute
2017-07-05 16:17:53.670 6156 ERROR neutron raise ProcessExecutionError(msg,
returncode=returncode)
2017-07-05 16:17:53.670 6156 ERROR neutron ProcessExecutionError: Exit code:
-15; Stdin: ; Stdout: ; Stderr: Signal 15 (TERM) caught by ps (procps-ng
version 3.3.10).
2017-07-05 16:17:53.670 6156 ERROR neutron ps:display.c:66: please report this
bug
2017-07-05 16:17:53.670 6156 ERROR neutron
2017-07-05 16:17:53.670 6156 ERROR neutron
2017-07-05 16:17:58.483 84539 WARNING oslo_config.cfg [-] Option
"prevent_arp_spoofing" from group "AGENT" is deprecated for removal.
Its value may be silently ignored in the future.
<silence>
2017-07-05 17:24:16.795 84539 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
[req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow
cookie=0xae1f44d4f69895b4, duration=521455.277s, table=0, n_packets=0,
n_bytes=0, idle_age=65534, hard_age=65534,
priority=10,icmp6,in_port=5,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:16.888 84539 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
[req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow
cookie=0xae1f44d4f69895b4, duration=521452.099s, table=0, n_packets=0,
n_bytes=0, idle_age=65534, hard_age=65534,
priority=10,icmp6,in_port=121,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:16.992 84539 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
[req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow
cookie=0xae1f44d4f69895b4, duration=521451.309s, table=0, n_packets=0,
n_bytes=0, idle_age=65534, hard_age=65534,
priority=10,icmp6,in_port=4,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:17.102 84539 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
[req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow
cookie=0xae1f44d4f69895b4, duration=521448.587s, table=0, n_packets=0,
n_bytes=0, idle_age=65534, hard_age=65534,
priority=10,icmp6,in_port=6,icmp_type=136 actions=resubmit(,24)
<etc>
nova-compute.log:
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif
[req-f3b2cae2-80e1-4a65-8b36-85b50e9e70f3 404192d1e0df4068ad985bd423db4cae
1f0043836a7d46c0a70d82056f1bc99c - - -] [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Failed while unplugging vif
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Traceback (most recent call last):
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 777, in
unplug_ovs_hybrid
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] run_as_root=True)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 388, in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] return
RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 271, in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] return processutils.execute(*cmd,
**kwargs)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] File
"/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389,
in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] cmd=sanitized_cmd)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] ProcessExecutionError: Unexpected error
while running command.
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Command: sudo nova-rootwrap
/etc/nova/rootwrap.conf brctl delif qbr0254be56-07 qvb0254be56-07
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Exit code: 1
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Stdout: u''
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c] Stderr: u'interface qvb0254be56-07 does
not exist!\n'
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance:
4451405e-5f24-42e3-8984-d6cb2f41e65c]
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif
[req-734b5574-9973-4f5a-834d-158d66856248 404192d1e0df4068ad985bd423db4cae
1f0043836a7d46c0a70d82056f1bc99c - - -] [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Failed while unplugging vif
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Traceback (most recent call last):
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 777, in
unplug_ovs_hybrid
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] run_as_root=True)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 388, in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] return
RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 271, in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] return processutils.execute(*cmd,
**kwargs)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] File
"/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389,
in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] cmd=sanitized_cmd)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] ProcessExecutionError: Unexpected error
while running command.
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Command: sudo nova-rootwrap
/etc/nova/rootwrap.conf brctl delif qbr8e404216-8e qvb8e404216-8e
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Exit code: 1
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Stdout: u''
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa] Stderr: u'interface qvb8e404216-8e does
not exist!\n'
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance:
67d903ad-e751-46c5-9d50-d856bc119efa]
One affected instance was noted to have lost it's 'qvb' interface (seen via
ovs-vsctl show) resulting in loss of connectivity.
At the same time, on a host running the neutron api, we noted the following in
the neutron openswitch log (package update was 2017-07-05 16:23:32):
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-8c6f6c00-d3b5-460c-b93a-ba9fb98ae899 - - - - -] Error while processing VIF
ports
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2038, in rpc_loop
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
ovs_restarted)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1653, in process_network_ports
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
failed_devices['added'] |= self._bind_devices(need_binding_devices)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 884, in _bind_devices
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.conf.host)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 180, in
update_device_list
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
agent_id=agent_id, host=host)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 155, in call
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
time.sleep(wait)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.force_reraise()
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
six.reraise(self.type_, self.value, self.tb)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 136, in call
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
self._original_context.call(ctxt, method, **kwargs)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in
call
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
retry=self.retry)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in
_send
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
timeout=timeout, retry=retry)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
470, in send
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
459, in _send
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
self._waiter.wait(msg_id, timeout)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
342, in wait
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message =
self.waiters.get(msg_id, timeout=timeout)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line
244, in get
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message
ID %s' % msg_id)
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
MessagingTimeout: Timed out waiting for a reply to message ID
d2c512e487d3462982d99c4053472ea3
2017-07-05 16:25:20.736 56273 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Restarting nova-compute on the compute node allowed the interface to recover
and connectivity resumed immediately.
The effect on this cloud was losing connectivity to a large number of
instances (although not all) until we identified the issue and restarted
nova-compute.
** Affects: neutron (Ubuntu)
Importance: Undecided
Status: New
** Tags: canonical-bootstack
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1702595
Title:
Upgrade neutron-plugin-openvswitch-agent package causes nova-compute
to fall over
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1702595/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
