Public bug reported: I'm trying to figure out why some live-migrations in our environment causes unreachability of the instance for about ~3 minutes.
When setting neutron-openvswitch in debug mode we can see that the bulk_pull of resources takes about 2-3 minutes before retrieving about 7057 ports(!). While doing this query the instance is not reachable. After the instance is migrated to the compute and is pingable again after 3 minutes we can successfully migrate the instance back and forth without downtime on the compute node. It looks like the cache is already filled the second migration and a bulk_pull is not required again. After some time it looks like the cache needs to be refreshed and a live- migration causes the instance to be unreachable for minutes again. We can clear the cache aswell when doing a restart of neutron-openvswitch- agent. I was able to reproduce this by adding a security group rule with any protocol with a remote-group defined and attach that sec group to an instance. Than migrate the instance to a compute node where neutron- openvswitch is restarted and no other instance is running on it. Info and logs: Openstack version: Pike OS: Redhat instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6 port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected [root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2017-10-11T13:13:48Z | | description | Default security group | | id | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | name | default | | project_id | b9bd8eac1587429aa07ae00661f2a07b | | revision_number | 16 | | | updated_at | 2018-04-17T08:14:55Z | +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2018-11-01T12:25:10Z | | description | | | id | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 | | location | None | | name | temp_sec | | project_id | b9bd8eac1587429aa07ae00661f2a07b | | revision_number | 10 | | | tags | [] | | updated_at | 2018-11-01T13:22:21Z | +-----------------+-----------------------------------------------------+ #Default security group, Broken(downtime) [root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 366a55ae-5086-4baf-a55c-bfef0cc38081 | None | None | | None | | 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp | 192.168.99.220/29 | 8000:8000 | None | | 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp | 0.0.0.0/0 | | None | | b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp | 192.168.99.220/29 | 22:22 | None | | fdfad4d8-95bf-419a-aa17-40ad048da7bf | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ #temp_sec security group, Works(No downtime) [root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None | | 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None | | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None | | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None | | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ When I migrate an instance with default sec group attached to a compute it's unavailable untill all the ports are retrieved with bulk query (timestamp reachable again: 13:00:03.783): #########I've added a log rule for the filter and resource type and query_id######### 2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69 2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site -packages/neutron/agent/resource_cache.py:70 2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48 70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72 <snip> 2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options= [],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id =8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set( [2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413']) 2018-11-01 12:58:51.447 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- 0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e :08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2 -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f']) 2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15 1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e :0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager [req- 08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site- packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2 -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f']) 2018-11-01 12:58:54.804 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- 08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 ######################## Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs ######################## 2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache [req- 6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources returned for queries set([('Port', ('security_group_ids', ('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 ######################### When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating: ######################### 2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter: {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)} _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:69 2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site -packages/neutron/agent/resource_cache.py:70 2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: query_id: set([('SecurityGroupRule', ('security_group_id', (u'13f878ef- eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72 2018-11-01 12:54:44.130 290907 DEBUG neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c- 94c4a7f7476e - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs': {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66 2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1831db02-7538-42ec-b04b-4c8c3d06dadb _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1c49d1ad-a6f1-470b-bd99-36f32932a6e1 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 3c1d82af-acd7-4543-8922-a96c0c67db67 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 452928bf-d5eb-41ab-b520-f34c33c63058 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4e9201db-3583-42c7-9c79-372a3394a159 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4ecc1fa6-d090-4d11-9a3d-323e1f012d40 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 65264a84-ae99-4c90-8626-6dbac869d15d _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 677d7b02-8943-4824-8fa5-990a02503f03 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: d435b0d1-67bb-4834-9518-5bdf8f843610 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req- c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: fc12d72c-94c9-432b-8f14-79412419c7db _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u '13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 #########It is returning just 11 resources which is correct value.############### ######### I can break the working sec group by adding a security group rule with any protocol with a remote-group defined: ######### [root@os-admin (openstack_admin)]# openstack security group rule create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b- 46f8-ac30-f13e48f9d4b1 #Remote group is added: [root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None | | 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None | | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None | | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None | | b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None | None | | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 | | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req- 24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resourcetype: Port _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:70 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req- 24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: query_id: set([('Port', ('security_group_ids', ('5958c3ad-5d2b- 46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7 /site-packages/neutron/agent/resource_cache.py:72 2018-11-01 14:30:24.242 311475 DEBUG neutron.api.rpc.handlers.resources_rpc [req- 24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7 /site-packages/oslo_log/helpers.py:66 2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache [req- 643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache [req- dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port eb9288fe- 9c78-447c-ba77-e2cfd8df624b updated (revision_number 826->828). Old fields: {'status': u'DOWN', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager [req- dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:24.956 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache [req- 643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache [req- 347c971e-3e22-4805-9b95-9a57d75f98aa a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache [req- d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port eb9288fe- 9c78-447c-ba77-e2cfd8df624b updated (revision_number 829->830). Old fields: {'status': u'ACTIVE'} New fields: {'status': u'DOWN'} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager [req- d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:28.381 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache [req- 94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78 -447c-ba77-e2cfd8df624b updated (revision_number 830->831). Old fields: {'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe- 9c78-447c-ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache [req- 94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78 -447c-ba77-e2cfd8df624b updated (revision_number 831->832). Old fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe- 9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager [req- 94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G <snip> ####### 7060 resources matched!: ####### 2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b'] _apply_port_filter /usr/lib/python2.7/site- packages/neutron/agent/securitygroups_rpc.py:135 2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information _update_security_group_info /usr/lib/python2.7/site- packages/neutron/agent/securitygroups_rpc.py:149 I'm not sure why this behave this way, can someone help me with this issue? Regards, Paul Peereboom ODC-Noord ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1801104 Title: Migration causes timeout while doing bulk_pull Status in neutron: New Bug description: I'm trying to figure out why some live-migrations in our environment causes unreachability of the instance for about ~3 minutes. When setting neutron-openvswitch in debug mode we can see that the bulk_pull of resources takes about 2-3 minutes before retrieving about 7057 ports(!). While doing this query the instance is not reachable. After the instance is migrated to the compute and is pingable again after 3 minutes we can successfully migrate the instance back and forth without downtime on the compute node. It looks like the cache is already filled the second migration and a bulk_pull is not required again. After some time it looks like the cache needs to be refreshed and a live-migration causes the instance to be unreachable for minutes again. We can clear the cache aswell when doing a restart of neutron- openvswitch-agent. I was able to reproduce this by adding a security group rule with any protocol with a remote-group defined and attach that sec group to an instance. Than migrate the instance to a compute node where neutron- openvswitch is restarted and no other instance is running on it. Info and logs: Openstack version: Pike OS: Redhat instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6 port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected [root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2017-10-11T13:13:48Z | | description | Default security group | | id | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | name | default | | project_id | b9bd8eac1587429aa07ae00661f2a07b | | revision_number | 16 | | | updated_at | 2018-04-17T08:14:55Z | +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2018-11-01T12:25:10Z | | description | | | id | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 | | location | None | | name | temp_sec | | project_id | b9bd8eac1587429aa07ae00661f2a07b | | revision_number | 10 | | | tags | [] | | updated_at | 2018-11-01T13:22:21Z | +-----------------+-----------------------------------------------------+ #Default security group, Broken(downtime) [root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 366a55ae-5086-4baf-a55c-bfef0cc38081 | None | None | | None | | 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp | 192.168.99.220/29 | 8000:8000 | None | | 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp | 0.0.0.0/0 | | None | | b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 | | e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp | 192.168.99.220/29 | 22:22 | None | | fdfad4d8-95bf-419a-aa17-40ad048da7bf | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ #temp_sec security group, Works(No downtime) [root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None | | 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None | | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None | | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None | | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ When I migrate an instance with default sec group attached to a compute it's unavailable untill all the ports are retrieved with bulk query (timestamp reachable again: 13:00:03.783): #########I've added a log rule for the filter and resource type and query_id######### 2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69 2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site -packages/neutron/agent/resource_cache.py:70 2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48 70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72 <snip> 2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options= [],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id =8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set( [2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413']) 2018-11-01 12:58:51.447 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- 0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e :08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2 -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f']) 2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15 1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e :0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199 2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site- packages/neutron_lib/callbacks/manager.py:167 2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2 -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f']) 2018-11-01 12:58:54.804 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- 08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 ######################## Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs ######################## 2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources returned for queries set([('Port', ('security_group_ids', ('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 ######################### When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating: ######################### 2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter: {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)} _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:69 2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site -packages/neutron/agent/resource_cache.py:70 2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: query_id: set([('SecurityGroupRule', ('security_group_id', (u '13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72 2018-11-01 12:54:44.130 290907 DEBUG neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c- 94c4a7f7476e - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs': {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66 2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1831db02-7538-42ec-b04b-4c8c3d06dadb _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1c49d1ad-a6f1-470b-bd99-36f32932a6e1 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 3c1d82af-acd7-4543-8922-a96c0c67db67 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 452928bf-d5eb-41ab-b520-f34c33c63058 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4e9201db-3583-42c7-9c79-372a3394a159 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4ecc1fa6-d090-4d11-9a3d-323e1f012d40 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 65264a84-ae99-4c90-8626-6dbac869d15d _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 677d7b02-8943-4824-8fa5-990a02503f03 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: d435b0d1-67bb-4834-9518-5bdf8f843610 _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: fc12d72c-94c9-432b-8f14-79412419c7db _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:88 2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u '13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 #########It is returning just 11 resources which is correct value.############### ######### I can break the working sec group by adding a security group rule with any protocol with a remote-group defined: ######### [root@os-admin (openstack_admin)]# openstack security group rule create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b- 46f8-ac30-f13e48f9d4b1 #Remote group is added: [root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | ID | IP Protocol | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None | | 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None | | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None | | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None | | b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None | None | | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 | | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None | +--------------------------------------+-------------+-------------------+------------+--------------------------------------+ 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resourcetype: Port _flood_cache_for_query /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:70 2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: query_id: set([('Port', ('security_group_ids', ('5958c3ad-5d2b- 46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72 2018-11-01 14:30:24.242 311475 DEBUG neutron.api.rpc.handlers.resources_rpc [req- 24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7 /site-packages/oslo_log/helpers.py:66 2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache [req-643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78 -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7 /site-packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 826->828). Old fields: {'status': u'DOWN', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:24.956 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache [req-643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78 -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7 /site-packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache [req-347c971e-3e22-4805-9b95-9a57d75f98aa a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78 -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7 /site-packages/neutron/agent/resource_cache.py:189 2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 829->830). Old fields: {'status': u'ACTIVE'} New fields: {'status': u'DOWN'} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:28.381 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req- d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site- packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 830->831). Old fields: {'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 831->832). Old fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c- ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id =eb9288fe-9c78-447c- ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e- 66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site- packages/neutron/agent/resource_cache.py:197 2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update --9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419 2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88 2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G <snip> ####### 7060 resources matched!: ####### 2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92 2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b'] _apply_port_filter /usr/lib/python2.7/site- packages/neutron/agent/securitygroups_rpc.py:135 2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information _update_security_group_info /usr/lib/python2.7/site- packages/neutron/agent/securitygroups_rpc.py:149 I'm not sure why this behave this way, can someone help me with this issue? Regards, Paul Peereboom ODC-Noord To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1801104/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp