Public bug reported: Hi, we are seeing strange behaviour on our victoria cluster after switching from hyrid firewall driver to native openvswitch firewall driver.
We have to use native openvswitch firewall driver to get firewall logs. After enabling security group logging we had observed that there exist too much DROP actions even any-any ingress-egress rules for all protocols exist in security groups. This seems normal according to [Native Open vSwitch firewall driver](https://docs.openstack.org/neutron/latest/admin/config- ovsfwdriver.html#differences-between-ovs-and-iptables-firewall-drivers) document. But we do not understand why the traffic is marked invalid by conntrack. We are seeing too much traffic marked as INVALID by conntrack, especially for the services which are doing too much traffic. For example etcd heartbeat which send to cluster members for every 100 ms (tcp port 2380) conntrack statistics also show high counts for "insert_failed" and "search_restart". nf_conntrack_buckets=65536 and nf_conntrack_max=262144. We also not see nf_conntrack_count reaches to max. We are seeing random and frequent timeouts on the kubernetes clusters which installed to openstack instances on this cluster. We believe that situation is related this. Especially calico-node pod on k8s cluster gets timeouts for liveness probe checks. Tested calico with both ipip and vxlan mode but no changes. Tested with k8s clusters which are installed to different OS but still no change. (centos 7, debian etcd) Environment Details: OpenStack Victoria Cluster installed via kolla-ansible to Ubuntu 20.04.2 LTS Hosts. (Kernel:5.4.0-80-generic) There exist 5 controller+network node. "neutron-openvswitch-agent", "neutron-l3-agent" and "neutron-server" version is "17.2.2.dev46" OpenvSwitch used in DVR mode with router HA configured. (l3_ha = true) We are using a single centralized neutron router for connecting all tenant networks to provider network. We are using bgp_dragent to announce unique tenant networks. Tenant network type: vxlan External network type: vlan Conntrack Invalid Logs (After enabling nf_conntrack_log_invalid logging) ... ... For etcd port 2380 ... Nov 24 10:45:47 test-compute-07 kernel: [9666429.466072] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=52384 DF PROTO=TCP SPT=33726 DPT=2380 SEQ=1503741580 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:06 test-compute-07 kernel: [9666448.362730] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=42180 DF PROTO=TCP SPT=42286 DPT=2380 SEQ=3794545871 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:11 test-compute-07 kernel: [9666453.465972] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=62831 DF PROTO=TCP SPT=33954 DPT=2380 SEQ=935403626 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:22 test-compute-07 kernel: [9666464.365487] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.168 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=47797 DF PROTO=TCP SPT=46064 DPT=2380 SEQ=4079966865 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467096] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13159 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467658] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13160 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:47:27 test-compute-07 kernel: [9666529.466842] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25780 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:27 test-compute-07 kernel: [9666529.467583] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25781 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:03 test-compute-07 kernel: [9666565.468588] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=44458 DF PROTO=TCP SPT=50346 DPT=2380 SEQ=179714231 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468069] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40395 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468408] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40396 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 ... ... For another ports ... Nov 24 10:45:12 test-compute-07 kernel: [9666394.834132] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.148 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=40824 SEQ=3886318363 ACK=1730529897 WINDOW=1190 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:49:39 test-compute-07 kernel: [9666661.880770] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=41428 SEQ=358351623 ACK=2255766346 WINDOW=1212 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:50:17 test-compute-07 kernel: [9666699.786127] nf_ct_proto_6: invalid rst IN= OUT= SRC=162.247.243.149 DST=10.211.2.251 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=50758 SEQ=1139505987 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 MARK=0x4010000 Conntrack Statistics logs from compute node (root namespace) attached. ** Affects: neutron Importance: Undecided Status: New ** Attachment added: "conntrack statistics" https://bugs.launchpad.net/bugs/1952055/+attachment/5542977/+files/conntrack-statistics.txt ** Description changed: Hi, we are seeing strange behaviour on our victoria cluster after switching from hyrid firewall driver to native openvswitch firewall driver. We have to use native openvswitch firewall driver to get firewall logs. After enabling security group logging we had observed that there exist too much DROP actions even any-any ingress-egress rules for all protocols exist in security groups. This seems normal according to [Native Open vSwitch firewall driver](https://docs.openstack.org/neutron/latest/admin/config- ovsfwdriver.html#differences-between-ovs-and-iptables-firewall-drivers) document. But we do not understand why the traffic is marked invalid by conntrack. We are seeing too much traffic marked as INVALID by conntrack, especially for the services which are doing too much traffic. For example etcd heartbeat which send to cluster members for every 100 ms (tcp port 2380) conntrack statistics also show high counts for "insert_failed" and "search_restart". nf_conntrack_buckets=65536 and nf_conntrack_max=262144. We also not see nf_conntrack_count reaches to max. - We are seeing random and frequent timeouts on the kubernetes cluster + We are seeing random and frequent timeouts on the kubernetes clusters which installed to openstack instances on this cluster. We believe that situation is related this. Especially calico-node pod on k8s cluster gets timeouts for liveness probe checks. Tested calico with both ipip - and vxlan mode but no changes. - + and vxlan mode but no changes. Tested with k8s clusters which are + installed to different OS but still no change. (centos 7, debian etcd) Environment Details: - OpenStack Victoria Cluster installed via kolla-ansible to Ubuntu 20.04.2 LTS Hosts. (Kernel:5.4.0-80-generic) - There exist 5 controller+network node. - "neutron-openvswitch-agent", "neutron-l3-agent" and "neutron-server" version is "17.2.2.dev46" - OpenvSwitch used in DVR mode with router HA configured. (l3_ha = true) - We are using a single centralized neutron router for connecting all tenant networks to provider network. - We are using bgp_dragent to announce unique tenant networks. - Tenant network type: vxlan - External network type: vlan + OpenStack Victoria Cluster installed via kolla-ansible to Ubuntu 20.04.2 LTS Hosts. (Kernel:5.4.0-80-generic) + There exist 5 controller+network node. + "neutron-openvswitch-agent", "neutron-l3-agent" and "neutron-server" version is "17.2.2.dev46" + OpenvSwitch used in DVR mode with router HA configured. (l3_ha = true) + We are using a single centralized neutron router for connecting all tenant networks to provider network. + We are using bgp_dragent to announce unique tenant networks. + Tenant network type: vxlan + External network type: vlan Conntrack Invalid Logs (After enabling nf_conntrack_log_invalid logging) ... ... For etcd port 2380 ... Nov 24 10:45:47 test-compute-07 kernel: [9666429.466072] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=52384 DF PROTO=TCP SPT=33726 DPT=2380 SEQ=1503741580 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:06 test-compute-07 kernel: [9666448.362730] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=42180 DF PROTO=TCP SPT=42286 DPT=2380 SEQ=3794545871 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:11 test-compute-07 kernel: [9666453.465972] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=62831 DF PROTO=TCP SPT=33954 DPT=2380 SEQ=935403626 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:22 test-compute-07 kernel: [9666464.365487] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.168 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=47797 DF PROTO=TCP SPT=46064 DPT=2380 SEQ=4079966865 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467096] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13159 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467658] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13160 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:47:27 test-compute-07 kernel: [9666529.466842] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25780 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:27 test-compute-07 kernel: [9666529.467583] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25781 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:03 test-compute-07 kernel: [9666565.468588] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=44458 DF PROTO=TCP SPT=50346 DPT=2380 SEQ=179714231 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468069] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40395 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468408] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40396 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 ... ... For another ports ... Nov 24 10:45:12 test-compute-07 kernel: [9666394.834132] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.148 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=40824 SEQ=3886318363 ACK=1730529897 WINDOW=1190 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:49:39 test-compute-07 kernel: [9666661.880770] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=41428 SEQ=358351623 ACK=2255766346 WINDOW=1212 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:50:17 test-compute-07 kernel: [9666699.786127] nf_ct_proto_6: invalid rst IN= OUT= SRC=162.247.243.149 DST=10.211.2.251 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=50758 SEQ=1139505987 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 MARK=0x4010000 Conntrack Statistics logs from compute node (root namespace) attached. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952055 Title: native firewall driver - conntrack marks too much traffic as invalid Status in neutron: New Bug description: Hi, we are seeing strange behaviour on our victoria cluster after switching from hyrid firewall driver to native openvswitch firewall driver. We have to use native openvswitch firewall driver to get firewall logs. After enabling security group logging we had observed that there exist too much DROP actions even any-any ingress-egress rules for all protocols exist in security groups. This seems normal according to [Native Open vSwitch firewall driver](https://docs.openstack.org/neutron/latest/admin/config- ovsfwdriver.html#differences-between-ovs-and-iptables-firewall- drivers) document. But we do not understand why the traffic is marked invalid by conntrack. We are seeing too much traffic marked as INVALID by conntrack, especially for the services which are doing too much traffic. For example etcd heartbeat which send to cluster members for every 100 ms (tcp port 2380) conntrack statistics also show high counts for "insert_failed" and "search_restart". nf_conntrack_buckets=65536 and nf_conntrack_max=262144. We also not see nf_conntrack_count reaches to max. We are seeing random and frequent timeouts on the kubernetes clusters which installed to openstack instances on this cluster. We believe that situation is related this. Especially calico-node pod on k8s cluster gets timeouts for liveness probe checks. Tested calico with both ipip and vxlan mode but no changes. Tested with k8s clusters which are installed to different OS but still no change. (centos 7, debian etcd) Environment Details: OpenStack Victoria Cluster installed via kolla-ansible to Ubuntu 20.04.2 LTS Hosts. (Kernel:5.4.0-80-generic) There exist 5 controller+network node. "neutron-openvswitch-agent", "neutron-l3-agent" and "neutron-server" version is "17.2.2.dev46" OpenvSwitch used in DVR mode with router HA configured. (l3_ha = true) We are using a single centralized neutron router for connecting all tenant networks to provider network. We are using bgp_dragent to announce unique tenant networks. Tenant network type: vxlan External network type: vlan Conntrack Invalid Logs (After enabling nf_conntrack_log_invalid logging) ... ... For etcd port 2380 ... Nov 24 10:45:47 test-compute-07 kernel: [9666429.466072] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=52384 DF PROTO=TCP SPT=33726 DPT=2380 SEQ=1503741580 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:06 test-compute-07 kernel: [9666448.362730] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=42180 DF PROTO=TCP SPT=42286 DPT=2380 SEQ=3794545871 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:11 test-compute-07 kernel: [9666453.465972] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=62831 DF PROTO=TCP SPT=33954 DPT=2380 SEQ=935403626 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:22 test-compute-07 kernel: [9666464.365487] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.139 DST=10.211.2.168 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=47797 DF PROTO=TCP SPT=46064 DPT=2380 SEQ=4079966865 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467096] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13159 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:07 test-compute-07 kernel: [9666509.467658] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=13160 DF PROTO=TCP SPT=49816 DPT=2380 SEQ=3428465462 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:47:27 test-compute-07 kernel: [9666529.466842] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25780 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:47:27 test-compute-07 kernel: [9666529.467583] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=25781 DF PROTO=TCP SPT=34674 DPT=2380 SEQ=1778600979 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:03 test-compute-07 kernel: [9666565.468588] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.139 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=44458 DF PROTO=TCP SPT=50346 DPT=2380 SEQ=179714231 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468069] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40395 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 Nov 24 10:48:07 test-compute-07 kernel: [9666569.468408] nf_ct_proto_6: invalid rst IN= OUT= SRC=10.211.2.168 DST=10.211.2.98 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40396 DF PROTO=TCP SPT=35050 DPT=2380 SEQ=1396127788 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 ... ... For another ports ... Nov 24 10:45:12 test-compute-07 kernel: [9666394.834132] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.148 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=40824 SEQ=3886318363 ACK=1730529897 WINDOW=1190 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:46:01 test-compute-07 kernel: [9666444.248252] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=45832 SEQ=2345805154 ACK=1982320186 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8A518E611E801030309) MARK=0x4010000 Nov 24 10:46:02 test-compute-07 kernel: [9666444.490741] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=10.168.112.39 DST=10.211.2.97 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=6533 DPT=59862 SEQ=3082071853 ACK=2961225592 WINDOW=28960 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A1ACBE8E218E612DA01030309) MARK=0x4010000 Nov 24 10:46:19 test-compute-07 kernel: [9666461.590026] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.121 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=56158 SEQ=1845326009 ACK=4146250693 WINDOW=1198 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:47:08 test-compute-07 kernel: [9666510.380344] nf_ct_proto_6: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=52.84.114.5 DST=10.211.2.89 LEN=60 TOS=0x00 PREC=0x00 TTL=228 ID=16475 PROTO=TCP SPT=443 DPT=49610 SEQ=3672172766 ACK=202854934 WINDOW=1428 RES=0x00 ACK SYN URGP=0 OPT (020405A00402080A65E5498918E687F501030309) MARK=0x4010000 Nov 24 10:49:39 test-compute-07 kernel: [9666661.880770] nf_ct_proto_6: invalid packet ignored in state SYN_SENT IN= OUT= SRC=162.247.243.149 DST=10.211.2.246 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=41428 SEQ=358351623 ACK=2255766346 WINDOW=1212 RES=0x00 ACK URGP=0 MARK=0x4010000 Nov 24 10:50:17 test-compute-07 kernel: [9666699.786127] nf_ct_proto_6: invalid rst IN= OUT= SRC=162.247.243.149 DST=10.211.2.251 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=50758 SEQ=1139505987 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 MARK=0x4010000 Conntrack Statistics logs from compute node (root namespace) attached. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1952055/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
