[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2021-06-30 Thread James Page
Marking OVS task as invalid as it appears this is a neutron bug related to configuration of VRRP for HA routers. ** Changed in: openvswitch (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-11-22 Thread James Page
** Changed in: openvswitch (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425 Title: Neutron integrated with OpenVSwitch drops packets and fails to

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-23 Thread Edward Hope-Morley
Hi folks, while trying to reproduce this behaviour myself I think i've stumbled upon some interesting behaviour. I setup a test as follows and checked for errors at specific points. I have a 4 node setup (24 core/64G ram) with 3 gateways and 1 compute. Neutron is configured with l3_ha enabled and

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-16 Thread James Page
I've spent the last two days trying to reproduce these problems on an OpenStack Cloud with three gateway/network units, each with 8 cores and 16G of RAM; I have 99 routers configured on the cloud, plumbed into a shared external provider network, and I've been switching them between ha and non-ha

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-15 Thread James Page
Hi Brian I'm working on a reproducer so this is a little easier to test; once I've reproduced using Ocata, I'll then re-test using the same process with pike and/or queens to see if things have improved. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-15 Thread Brian Haley
And just as a follow-on to my previous comment - is it possible to test this with the latest code from the master branch? I know you're running packages from Ubuntu I just didn't want to waste cycles tracking things down that might be fixed already, and maybe just need a backport to a stable

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread Brian Haley
You described a lot of issues in comment #14: * Missing interfaces inside qrouter namespaces (OVS taps) * Missing iptables rules * Missing floating IP aliases on OVS interfaces inside the qrouter namespaces Some of those might be fixed in master, especially the iptables one, and should have been

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Troup
I've unsubscribed Canonical Field Critical as removing the 4095 tagged ports in OVS on all 3 Neutron Gateway nodes seems to have made the packet loss go away (for now?). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Hebden
Correction - The `remove_vip_by_ip_address` call failing just after the IPTablesManager is called and succeeds. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425 Title: Neutron integrated with

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Hebden
@james-page, @axino - Just a +1 to the HA property being changed requiring the router to be set down prior, and back up after to start the recreation of the router as HA. We have seen various other side effects in Neutron/OVS environments and specifically the environment in question, such as -

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
OK so the original bug report was actually about packet loss in ovs; the network nodes have a large number of lost tap devices tagged with 4095 segmentation id, reporting no such device in OVS. Theory might be is is dosing the ovs-vswitchd process resulting in some userspace datapath lossyness.

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
You can see an update message prior to the remove-from-agent processing with: "ha_vr_id": null, -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425 Title: Neutron integrated with

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
Supposition might be that ha_vr_id has been unset at this point in time? def _get_keepalived_instance(self): return self.keepalived_manager.config.get_instance(self.ha_vr_id) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
OK and now: 2018-02-13 14:18:08.281 2065380 DEBUG neutron.agent.l3.agent [-] Starting router update for 213b6544-ab4b-4e46-a5c6-5d8d587a0c6d, action 1, priority 0 _process_router_update /usr/lib/python2.7/dist- packages/neutron/agent/l3/agent.py:483 which I think is the actual processing of the

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
Some sequencing: unbind from agent message inbound: 2018-02-13 14:12:57.931 2065380 DEBUG neutron.agent.l3.agent [req- 13456a6c-5583-4147-9b47-94026ea7f3b4 b327544aba2a482b9f12f1e6e615c394 9a4311b33381401fbc835c739981ce03 - - -] Got router removed from agent :{u'router_id':

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
@axino yes switching between ha/non-ha is support so long as the router is in admin state false at the time of change (which I believe is the case in this issue). Some further observations; switching a router from ha mode to non-ha mode should remove all existing router agent hosting

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread Junien Fridrick
This is happening on all 3 neutron-l3-agents. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425 Title: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread Junien Fridrick
We're going to provide the logs, but neutron-l3-agent is getting spammed with https://pastebin.canonical.com/p/PBPpJJkRgy/ for dozens of routers. Does neutron support changing the "ha" property of a router live ? -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Page
Would it be possible to get log files from /var/log/neutron and /var/log/openvswitch as well as the syslog and auth.log files from an impacted machine. I suspect that due to the large number of namespaces being managed something is just taking to long to parse/sync/refresh causing this issue. --

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread Chris Sanders
I've subscribed Canonical Field Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425 Title: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Hebden
** Attachment added: "ovs-missing-taps-14-02-2018.log" https://bugs.launchpad.net/neutron/+bug/1749425/+attachment/5054969/+files/ovs-missing-taps-14-02-2018.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Hebden
** Attachment added: "SystemTap output" https://bugs.launchpad.net/neutron/+bug/1749425/+attachment/5054990/+files/drops-14-02-2018.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425

[Bug 1749425] Re: Neutron integrated with OpenVSwitch drops packets and fails to plug/unplug interfaces from OVS on router interfaces at scale

2018-02-14 Thread James Hebden
** Attachment added: "SystemTap script" https://bugs.launchpad.net/neutron/+bug/1749425/+attachment/5054991/+files/drops-14-02-2018.stp -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749425