Public bug reported: When remote group member changed, conntrack entry which should be deleted is not deleted.
How to reproduce: * create a VM(VM1) on host-1 (net-a, default security-group) (ex. 10.0.0.3) * create a VM(VM2) on host-2 (net-a, default security-group) (ex. 10.0.0.4) * ssh from VM1(10.0.0.3) to VM2(10.0.0.4) --- host-2:$ sudo conntrack -L |grep 10.0 tcp 6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1 host-2:$ sudo ipset list Name: NIPv492469920-ef76-44af-98c7- Type: hash:net Revision: 4 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 16824 References: 1 Members: 10.0.0.4 10.0.0.3 --- * terminate VM1 (nova delete VM1) expected: the conntrack entry showed above is deleted. actual: not deleted --- host-2:$ sudo conntrack -L |grep 10.0 tcp 6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1 host-2:$ sudo ipset list Name: NIPv492469920-ef76-44af-98c7- Type: hash:net Revision: 4 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 16824 References: 1 Members: 10.0.0.4 --- Applied: liberty, mitaka, master Investigation: summary - devices_with_updated_sg_members is consumed by remove_devices_filter unintentionally. * when ovs-agent receives security_groups_member_updated RPC call, sg_ids and affected devices are registered to self.firewall.devices_with_updated_sg_members. (original intention is that it is handled when refresh_firewall is called. but...) * in the main loop of ovs-agent process_deleted_ports is executed before process_network_ports. process_deleted_ports calls self.sg_agent.remove_devices_filter. if there is any deleted port, remove_devices_filter -> defer_apply_off -> _remove_conntrack_entries_from_sg_update -> _clean_deleted_remote_sg_members_conntrack_entries is called. at this time pre_sg_members and sg_members are same since no port info is updated in remove_devices_filter. so no conntrack entry is removed. but nonetheless devices_with_updated_sg_members is cleared !! * afterwards process_network_ports -> setup_port_filters -> refresh_firewall -> defer_apply_off ...-> _clean_deleted_remote_sg_members_conntrack_entries is called. at this time pre_sg_members and sg_members are different since port info was updated. but no conntrack entry is removed since devices_with_updated_sg_members was cleared. Note: deleting conntrack entries was introduced by https://bugs.launchpad.net/neutron/+bug/1335375. note that conntrack zone was per network at this time. (per port now) The fix of 1335375 is complicated and I wonder it is incomplete (ex. no care of egress & remote-group rule). How about simplify to just record affected ports and do "conntrack -D -w <port's zone id>" for affected ports ? ** Affects: neutron Importance: Undecided Status: New ** Tags: sg-fw ** Tags added: sg-fw -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1580377 Title: conntrack entry is not deleted when security_groups_member_updated Status in neutron: New Bug description: When remote group member changed, conntrack entry which should be deleted is not deleted. How to reproduce: * create a VM(VM1) on host-1 (net-a, default security-group) (ex. 10.0.0.3) * create a VM(VM2) on host-2 (net-a, default security-group) (ex. 10.0.0.4) * ssh from VM1(10.0.0.3) to VM2(10.0.0.4) --- host-2:$ sudo conntrack -L |grep 10.0 tcp 6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1 host-2:$ sudo ipset list Name: NIPv492469920-ef76-44af-98c7- Type: hash:net Revision: 4 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 16824 References: 1 Members: 10.0.0.4 10.0.0.3 --- * terminate VM1 (nova delete VM1) expected: the conntrack entry showed above is deleted. actual: not deleted --- host-2:$ sudo conntrack -L |grep 10.0 tcp 6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1 host-2:$ sudo ipset list Name: NIPv492469920-ef76-44af-98c7- Type: hash:net Revision: 4 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 16824 References: 1 Members: 10.0.0.4 --- Applied: liberty, mitaka, master Investigation: summary - devices_with_updated_sg_members is consumed by remove_devices_filter unintentionally. * when ovs-agent receives security_groups_member_updated RPC call, sg_ids and affected devices are registered to self.firewall.devices_with_updated_sg_members. (original intention is that it is handled when refresh_firewall is called. but...) * in the main loop of ovs-agent process_deleted_ports is executed before process_network_ports. process_deleted_ports calls self.sg_agent.remove_devices_filter. if there is any deleted port, remove_devices_filter -> defer_apply_off -> _remove_conntrack_entries_from_sg_update -> _clean_deleted_remote_sg_members_conntrack_entries is called. at this time pre_sg_members and sg_members are same since no port info is updated in remove_devices_filter. so no conntrack entry is removed. but nonetheless devices_with_updated_sg_members is cleared !! * afterwards process_network_ports -> setup_port_filters -> refresh_firewall -> defer_apply_off ...-> _clean_deleted_remote_sg_members_conntrack_entries is called. at this time pre_sg_members and sg_members are different since port info was updated. but no conntrack entry is removed since devices_with_updated_sg_members was cleared. Note: deleting conntrack entries was introduced by https://bugs.launchpad.net/neutron/+bug/1335375. note that conntrack zone was per network at this time. (per port now) The fix of 1335375 is complicated and I wonder it is incomplete (ex. no care of egress & remote-group rule). How about simplify to just record affected ports and do "conntrack -D -w <port's zone id>" for affected ports ? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1580377/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
