** Summary changed: - Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein] + [SRU] Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein]
** Description changed: Setting up external gateway on the router, like this: 'openstack router set --external-gateway ext-net ext-net-router', immediately brings all ports of this router Down. At the same time, in neutron-l3-agent.log, the following errors appear: 2019-12-20 08:39:34.140 1163288 INFO neutron.agent.l3.agent [-] Starting router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, action 3, priority 1, update_id 529cd623-445d-4a21-8499-a12517c7043b. Wait time elapsed: 0.000 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent [-] Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.: neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent. 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent router_id=router['id']) 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent. 2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent 2019-12-20 08:39:36.739 1163288 ERROR neutron.agent.l3.agent [-] Removing incompatible router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37': neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent. 2019-12-20 08:39:37.065 1163288 INFO neutron.agent.l3.agent [-] Finished a router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, update_id 529cd623-445d-4a21-8499-a12517c7043b. Time elapsed: 2.925 WORKAROUND When I enabled 'debug' on the neutron-openvswitch to troubleshoot the problem, shortly after the ports on the router went to Active. I tried to disable 'debug' and reproduce the problem, but since then the issue did not show up again. NETWORK TOPOLOGY My network topology is depicted in the attached network-topology.png. I have two external networks. My goal is to create two routers, each connected to different external network. STEPS TO REPRODUCE 1. Create two external networks 2. Create first router - create router - set external gateway - add private subnet to the router So far everything is OK. All ports on first router are Active. Instances created on private network can have Floating IPs assigned and are able to ping each other using FIPs. 3. Create second router - create router (OK, no problems with that): 'openstack router create --distributed --ha --project admin --project-domain admin_domain ext-net-router' - set external gateway (NOT OK): 'openstack router set --external-gateway ext-net --enable-snat ext-net-router'. I tried both with --enable-snat and without this. Same effect. As soon as I run 'openstack router set --external-gateway ext-net ext- net-router', I can immediately see that all ports on this router go Down. As a result, instances created on the second private network (ubuntu-net) can't access external network. I still can assign Floating IPs to the instances but I noticed that fip-* namespaces are not being created. LOG FILE https://pastebin.canonical.com/p/QdRmqQmwYP/ CHARMS CONFIGURATION Full bundle: https://pastebin.canonical.com/p/H8RMqDsKTx/ neutron-openvswitch: bridge-mappings: dcfabric:br-data data-port: br-data:bond1 debug: false disable-security-groups: false dns-servers: <redacted>,<redacted>,<redacted> dpdk-bond-config: :balance-tcp:active:fast dpdk-socket-cores: 1 dpdk-socket-memory: 1024 enable-dpdk: false enable-local-dhcp-and-metadata: true enable-sriov: false firewall-driver: openvswitch firewall-group-log-burst-limit: 25 os-data-network: "" prevent-arp-spoofing: true rabbit-user: neutron rabbit-vhost: openstack security-group-log-burst-limit: 25 sriov-numvfs: auto sysctl: | { net.ipv4.neigh.default.gc_thresh1 : 128, net.ipv4.neigh.default.gc_thresh2 : 28672, net.ipv4.neigh.default.gc_thresh3 : 32768, net.ipv6.neigh.default.gc_thresh1 : 128, net.ipv6.neigh.default.gc_thresh2 : 28672, net.ipv6.neigh.default.gc_thresh3 : 32768, net.nf_conntrack_max : 1000000, net.netfilter.nf_conntrack_buckets : 204800, net.netfilter.nf_conntrack_max : 1000000 } use-dvr-snat: true use-syslog: false verbose: false vlan-ranges: physnet1:1000:2000 worker-multiplier: 0.25 neutron-api: action-managed-upgrade: false allow-automatic-dhcp-failover: true allow-automatic-l3agent-failover: true database: neutron database-user: neutron debug: false default-tenant-network-type: vxlan dhcp-agents-per-network: 9 dhcp-load-type: networks dns-domain: openstack.example.RegionOne.lan. dns-ha: false enable-dvr: true enable-firewall-group-logging: false enable-l3ha: true enable-ml2-dns: true enable-ml2-port-security: true enable-port-forwarding: false enable-qos: false enable-security-group-logging: false enable-sriov: false enable-vlan-trunking: false global-physnet-mtu: 9000 ha-bindiface: eth0 ha-mcastport: 5424 ipv4-ptr-zone-prefix-size: 24 ipv6-ptr-zone-prefix-size: 64 l2-population: true manage-neutron-plugin-legacy-mode: true max-l3-agents-per-router: 3 midonet-origin: midonet-2015.06 min-l3-agents-per-router: 2 nagios_context: juju nagios_servicegroups: "" neutron-external-network: ext_net neutron-plugin: ovs neutron-security-groups: true nsx-password: admin nsx-username: admin nuage-packages: nuage-openstack-neutron nuagenetlib openstack-origin: cloud:bionic-stein os-admin-hostname: neutron.RegionOne.example.com os-internal-hostname: neutron-internal.RegionOne.example.com os-public-hostname: neutron.RegionOne.example.com os-public-network: vxlan gre path-mtu: 9000 physical-network-mtus: dcfabric:1500 plumgrid-password: plumgrid plumgrid-username: plumgrid polling-interval: 2 prefer-ipv6: false quota-floatingip: 50 quota-health-monitors: -1 quota-member: -1 quota-network: 10 quota-pool: 10 quota-port: 50 quota-router: 10 quota-security-group: 10 quota-security-group-rule: 100 quota-subnet: 10 quota-vip: 10 rabbit-user: neutron rabbit-vhost: openstack region: RegionOne report-interval: 30 reverse-dns-lookup: true rpc-response-timeout: 60 ssl_ca: <redacted> ssl_cert: <redacted> ssl_key: <redacted> use-internal-endpoints: true use-policyd-override: false use-syslog: false verbose: false vip: <redacted> <redacted> vip_cidr: 24 vip_iface: eth0 vlan-ranges: dcfabric vni-ranges: 1001:2000 vsd-auth: csproot:csproot vsd-auth-resource: /me vsd-auth-ssl: true vsd-base-uri: /nuage/api/v3_0 vsd-netpart-name: juju-enterprise vsd-organization: csp worker-multiplier: 0.25 + + [Impact] + See above + + [Test Case] + See above and phausman has said he can test this. + + [Regression Potential] + The cherry-picked patch has landed in the upstream stable/stein branch and is already in train upstream and the train package. The fix is minimal and removes code that is restricting (already supported) multiple external networks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1857126 Title: [SRU] Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein] To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1857126/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
