Public bug reported: Neutron 2023.1 29cc1a634e530972614c09fbb212b5f63fd4c374 Ubuntu 20.04
This issue has been identified in a Neutron system running Linux Bridge networking, but whilst this may no longer be supported I'm posting it in case the same issue might be relevant for other drivers. When running multiple network nodes, the tenant networks in the namespaces on each node share MAC addresses. We have noted that particularly when rebooting a network node, traffic from the Internet to tenant networks can be disrupted when a node which was acting as the backup for a given tenant network comes back online. We have traced this to Linux sending out MLDv2 responses to the upstream switches when the tenant network processes (keepalived etc) start up. As a result, the upstream switches update their MAC tables to prefer that host despite it not being the primary. If there is minimal tenant traffic (such as when running a web server), this network will be inaccessible from the outside until a request is made from the inside to the outside and the switches re-update their MAC tables to reflect the correct state. There is already handling to prevent some IPv6 packets being sent out in these cases here: https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L808, and there is theoretically something explicitly referencing issues with MLDv2 in the same area: https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L813. Unfortunately these don't appear to be sufficient. There is no sysctl mechanism to prevent these gratuitous MLDv2 responses as far as I can tell, so we are working around this by using iptables rules inserted by the L3 agent into tenant networks. There may well be a better solution, but I will link our workaround to this bug report shortly. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2049909 Title: MLDv2 packets sent from L3 Agent managed networks cause backup routers to be preferred Status in neutron: New Bug description: Neutron 2023.1 29cc1a634e530972614c09fbb212b5f63fd4c374 Ubuntu 20.04 This issue has been identified in a Neutron system running Linux Bridge networking, but whilst this may no longer be supported I'm posting it in case the same issue might be relevant for other drivers. When running multiple network nodes, the tenant networks in the namespaces on each node share MAC addresses. We have noted that particularly when rebooting a network node, traffic from the Internet to tenant networks can be disrupted when a node which was acting as the backup for a given tenant network comes back online. We have traced this to Linux sending out MLDv2 responses to the upstream switches when the tenant network processes (keepalived etc) start up. As a result, the upstream switches update their MAC tables to prefer that host despite it not being the primary. If there is minimal tenant traffic (such as when running a web server), this network will be inaccessible from the outside until a request is made from the inside to the outside and the switches re-update their MAC tables to reflect the correct state. There is already handling to prevent some IPv6 packets being sent out in these cases here: https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L808, and there is theoretically something explicitly referencing issues with MLDv2 in the same area: https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L813. Unfortunately these don't appear to be sufficient. There is no sysctl mechanism to prevent these gratuitous MLDv2 responses as far as I can tell, so we are working around this by using iptables rules inserted by the L3 agent into tenant networks. There may well be a better solution, but I will link our workaround to this bug report shortly. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2049909/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp