As per comment #39, this can be closed - this bug report is mostly a tracker bug and I'm under most of the races that made test_ha_router fail are resolved.
Some other races are https://bugs.launchpad.net/neutron/+bug/1605285 and https://bugs.launchpad.net/neutron/+bug/1605282, but these can be addressed separately. ** Changed in: neutron Status: In Progress => Fix Released ** Changed in: neutron/kilo Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1499647 Title: test_ha_router fails intermittently Status in neutron: Fix Released Status in neutron kilo series: Fix Released Bug description: I have tested work of L3 HA on environment with 3 controllers and 1 compute (Kilo) keepalived v1.2.13 I create 50 nets with 50 subnets and 50 routers with interface is set for each subnet(Note: I've seem the same errors with just one router and net). I've got the following errors: root@node-6:~# neutron l3-agent-list-hosting-router router-1 Request Failed: internal server error while processing your request. In neutron-server error log: http://paste.openstack.org/show/473760/ When I fixed _get_agents_dict_for_router to skip None for further testing, so then I was able to see: root@node-6:~# neutron l3-agent-list-hosting-router router-1 +--------------------------------------+-------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+-------------------+----------------+-------+----------+ | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True | :-) | active | | c9159f09-34d4-404f-b46c-a8c18df677f3 | node-7.domain.tld | True | :-) | standby | | b458ab49-c294-4bdb-91bf-ae375d87ff20 | node-8.domain.tld | True | :-) | standby | | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True | :-) | active | +--------------------------------------+-------------------+----------------+-------+----------+ root@node-6:~# neutron port-list --device_id=fcf150c0-f690-4265-974d-8db370e345c4 +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+ | 0834f8a2-f109-4060-9312-edebac84aba5 | | fa:16:3e:73:9f:33 | {"subnet_id": "0c7a2cfa-1cfd-4ecc-a196-ab9e97139352", "ip_address": "172.18.161.223"} | | 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:b8:f6:35 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.149"} | | 48c887c1-acc3-4804-a993-b99060fa2c75 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:e7:70:13 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.151"} | | 82ab62d6-7dd1-4294-a0dc-f5ebfbcbb4ca | | fa:16:3e:c6:fc:74 | {"subnet_id": "c4cc21c9-3b3a-407c-b4a7-b22f783377e7", "ip_address": "10.0.40.1"} | | bbca8575-51f1-4b42-b074-96e15aeda420 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:84:4c:fc | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.150"} | | bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:09:a1:ae | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.11"} | | f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:c4:54:b5 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.12"} | +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+ mysql root@192.168.0.2:neutron> SELECT * FROM ha_router_agent_port_bindings WHERE router_id='fcf150c0-f690-4265-974d-8db370e345c4'; +--------------------------------------+--------------------------------------+--------------------------------------+---------+ | port_id | router_id | l3_agent_id | state | |--------------------------------------+--------------------------------------+--------------------------------------+---------| | 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | fcf150c0-f690-4265-974d-8db370e345c4 | c9159f09-34d4-404f-b46c-a8c18df677f3 | standby | | 48c887c1-acc3-4804-a993-b99060fa2c75 | fcf150c0-f690-4265-974d-8db370e345c4 | b458ab49-c294-4bdb-91bf-ae375d87ff20 | standby | | bbca8575-51f1-4b42-b074-96e15aeda420 | fcf150c0-f690-4265-974d-8db370e345c4 | <null> | standby | | bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active | | f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active | +--------------------------------------+--------------------------------------+--------------------------------------+---------+ So extra L3HARouterAgentPortBinding was created for routers. This issue does not reproduced all the time. During sync_routers the following errors in logs appeared: http://paste.openstack.org/show/473839/ http://paste.openstack.org/show/473840/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1499647/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
