From: <[email protected]> on behalf of Rohith Basavaraja <[email protected]> Date: Wednesday, November 1, 2017 at 10:28 PM To: "[email protected]" <[email protected]> Cc: Jan Scheurich <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK
Hi, It’ been quite some time I raised this issue, thought will update the thread with our findings. Following is the summary of our findings and analysis, and we think OVS user datapath conntrack implementation Needs to be fixed otherwise some of the security group deployments mentioned below might fail. Analysis/Findings =============== Currently OVS kernel datapath implementation have the ct_state (conntrack state) semantics as described In the following document. http://www.iptables.info/en/connection-state.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.iptables.info_en_connection-2Dstate.html&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=cTQCN_hF1TN3jk_TJWYNMha6aUqSOURuVHEc5RCep1Y&s=yCJI8jfdp7_jUXkdLDee-j25N93R7kNawJUgmugy2-M&e=>[1] OVS user datapath doesn’t follow above semantics and also the ct_state description in the OVS specification (http://openvswitch.org/support/dist-docs/ovs-fields.7.pdf)<https://urldefense.proofpoint.com/v2/url?u=http-3A__openvswitch.org_support_dist-2Ddocs_ovs-2Dfields.7.pdf-29&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=cTQCN_hF1TN3jk_TJWYNMha6aUqSOURuVHEc5RCep1Y&s=eGwS0b14Sw_uvAefFrhp3aKmzckD4UOt2Y0nwXsPPS8&e=>[2] is not correct as explained below. The main issue is when the conntrack state “CS_ESTABLISHED” is set for a tracked flow. In the kernel datapath and iptables a tracked flow moves to established state only once it sees a reply packet in the reverse direction. The user-space conntrack, in contrast, moves a tracked connection to established state as soon as a newly tracked connection is looked up the first time, irrespectively of the direction of the packet. [Darrell] are you sure ? The expectation is that the Userspace Datapath 2.6 behavior adheres to the OVS specification below. Please provide a test case that shows this is not the case; I would be interested ? Finally, OVS specification [2] defines the “est” state as “est (0x02) Part of an existing connection. Set to 1 if this is a committed connection”. This means that the tracked connection would move to established state when the ct(commit) action is executed and the semantics don’t match either the kernel or user-space behaviour. Because of the above difference some of the Security Group(SGs) use cases are failing for eg: VMs that have SGs that shall not allow communication among them are not working when VMs are on the same compute node. [Darrell] We had a lengthy offline email discussion about this from 8/23 to 8/30. The last few exchanges are below. /////////////////////////////////////////////// From: Rohith Basavaraja <[email protected]> Date: Wednesday, August 30, 2017 at 9:05 AM To: Darrel Ball <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Darrell, Thanks, a lot for the help and sharing the useful information. Thanks Rohith From: Darrell Ball <[email protected]> Date: Wednesday, 30 August 2017 at 9:18 PM To: Rohith Basavaraja <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Rohith From: Rohith Basavaraja <[email protected]> Date: Wednesday, August 30, 2017 at 3:46 AM To: Darrel Ball <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Darrell, For user datapath do we have any other tools to dump conntrack entries Other than *ovs-appctl dpctl/dump-conntrack* [Darrell] Right now, we have " dump-conntrack [DP] [zone=ZONE] " \ "display conntrack entries for ZONE\n" " flush-conntrack [DP] [zone=ZONE] " \ "delete all conntrack entries in ZONE\n" " ct-stats-show [DP] [zone=ZONE] [verbose] " \ "CT connections grouped by protocol\n" " ct-bkts [DP] [gt=N] display connections per CT bucket\n" This is from ./utilities/ovs-dpctl.c For kernel datapath I see that we can use conntrack –L to dump the entries, Is conntrack tool is only for kernel datapath only? [Darrell] Yes In general, any other conntrack commands or tools available for userdatapath? [Darrell] Right now, the ones mentioned above Of course, the well known commands also tell lots about what is happening in conntrack indirectly sudo ovs-ofctl dump-flows br0 sudo ovs-appctl dpif/dump-flows br0 Sorry for too many queries, Pl let me know if it’s bothering you. [Darrell] No problem at all. Thanks Rohith From: Rohith Basavaraja <[email protected]> Date: Wednesday, 30 August 2017 at 1:57 PM To: Darrell Ball <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Darrell, Thanks for the suggestions. Thanks Rohith From: Darrell Ball <[email protected]> Date: Wednesday, 30 August 2017 at 12:17 PM To: Rohith Basavaraja <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Ho Rohith So the previous answer is the solution then; elaborating: You want a committed connection VM2 -> VM1 (i.e. originated from VM2); this allows VM1 to send replies to VM2. You want to prevent creating a committed connection from VM1 -> VM2 This can be done in various ways by using in_port, zones (per logical ports), dl_src, dl_dst etc So traffic originated from VM1 -> VM2 will always be new Thanks Darrell From: Rohith Basavaraja <[email protected]> Date: Tuesday, August 29, 2017 at 11:19 PM To: Darrel Ball <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Darrell, Just to clarify following is the usecase. 1. VM1 can originate/initiate traffic to VM2 2. VM1 can receive traffic from VM2 3. VM2 should not receive any new connection from VM1 4. VM2 can originate/initiate traffic to VM1 Thanks Rohith From: Darrell Ball <[email protected]> Date: Wednesday, 30 August 2017 at 11:37 AM To: Rohith Basavaraja <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Rohith Just to confirm: 1/ VM1 can never send traffic to VM2 (originate or reply) ? OR 2/ VM1 cannot originate traffic to VM2 but VM1 can send reply traffic to VM2. ? I have now been assuming ‘2’ ? Thanks Darrell /////////////////////////////////////////////// Thanks Rohith From: Rohith Basavaraja <[email protected]> Date: Thursday, 24 August 2017 at 12:43 AM To: Darrell Ball <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Darrell, Yes the expected outcome is to drop new or non related connection, and allow only related or established connections. Just for the clarity adding the details of the topology and pipeline rules. Description about the topology ========================= VM1 and VM4 VMs are on same compute node but with different SGs. For VM4, security rules configured are as below: Egress/Ingress Allow all For VM1, Egress Allow all Ingress Allow only from VMs which are in same security group. For above combination, all conntrack flows required (in tables 213, 214 on VM egress side and 243, 244) are properly programmed in the OVS. For traffic sent from VM4 to VM1 , conntrack is allowing traffic which should have been dropped as the ingress for later is to be allowed only from the VMs of the same SG. For VM1 , conntrack is directly sending traffic to "ct_state==-new+est-rel-inv+trk " flow by-passing "ct_state=+new+trk" flow in the ingress direction. Following is the pipe line rules details ============================== VM4 is on 112/15: (dpdkvhostuser: configured_rx_queues=1, configured_tx_queues=1, mtu=2140, requested_rx_queues=1, requested_tx_queues=1) VM1 is on 108/11: (dpdkvhostuser: configured_rx_queues=1, configured_tx_queues=1, mtu=2140, requested_rx_queues=1, requested_tx_queues=1) VM4 IP: 172.20.1.113 MAC: fa:16:3e:55:9e:33 VM1 IP: 172.20.1.117 MAC : fa:16:3e:f7:72:d3 I am doing Ping from VM4 (172.20.1.113 ) to VM1 (172.20.1.117). cookie=0x8000000, duration=2809.367s, table=0, n_packets=74, n_bytes=10426, priority=4,in_port=112,vlan_tci=0x0000/0x1fff actions=write_metadata:0x19f40000000000/0xffffff0000000001,goto_table:17 cookie=0x6900000, duration=2809.343s, table=17, n_packets=74, n_bytes=10426, priority=10,metadata=0x19f40000000000/0xffffff0000000000 actions=write_metadata:0x4019f40000000000/0xfffffffffffffffe,goto_table:211 cookie=0x6900000, duration=2809.313s, table=211, n_packets=54, n_bytes=8674, priority=61010,ip,metadata=0x19f40000000000/0x1fffff0000000000,dl_src=fa:16:3e:55:9e:33,nw_src=172.20.1.113 actions=goto_table:212 cookie=0x6900000, duration=15546.529s, table=212, n_packets=3669, n_bytes=361562, priority=61010,icmp actions=goto_table:213 cookie=0x6900000, duration=2809.308s, table=213, n_packets=54, n_bytes=8674, priority=61010,ip,metadata=0x19f40000000000/0x1fffff0000000000 actions=ct(table=214,zone=5021) cookie=0x6900000, duration=15546.544s, table=214, n_packets=3660, n_bytes=367508, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,17) cookie=0x6900001, duration=2809.304s, table=214, n_packets=2, n_bytes=180, priority=62015,ct_state=+inv+trk,metadata=0x19f40000000000/0x1fffff0000000000 actions=drop cookie=0x6900000, duration=2809.295s, table=214, n_packets=10, n_bytes=908, priority=1000,ct_state=+new+trk,ip,metadata=0x19f40000000000/0x1fffff0000000000 actions=ct(commit,zone=5021),resubmit(,17) cookie=0x6800001, duration=2807.340s, table=17, n_packets=72, n_bytes=10246, priority=10,metadata=0x4019f40000000000/0xffffff0000000000 actions=write_metadata:0xc019f40000000000/0xfffffffffffffffe,goto_table:60 cookie=0x6800000, duration=15546.596s, table=60, n_packets=262265, n_bytes=19604926, priority=0 actions=resubmit(,17) cookie=0x8040000, duration=2807.338s, table=17, n_packets=70, n_bytes=9562, priority=10,metadata=0xc019f40000000000/0xffffff0000000000 actions=write_metadata:0xe019f4139d000000/0xfffffffffffffffe,goto_table:48 cookie=0x8500000, duration=15546.528s, table=48, n_packets=302458, n_bytes=34190652, priority=0 actions=resubmit(,49),resubmit(,50) cookie=0x805139d, duration=2808.378s, table=50, n_packets=70, n_bytes=9562, priority=20,metadata=0x19f4139d000000/0x1fffffffff000000,dl_src=fa:16:3e:55:9e:33 actions=goto_table:51 cookie=0x803139d, duration=2818.232s, table=51, n_packets=34, n_bytes=4613, priority=20,metadata=0x139d000000/0xffff000000,dl_dst=fa:16:3e:f7:72:d3 actions=load:0x1a5300->NXM_NX_REG6[],resubmit(,220) cookie=0x6900000, duration=2819.193s, table=220, n_packets=3455, n_bytes=207541, priority=6,reg6=0x1a5300 actions=load:0xe01a5300->NXM_NX_REG6[],write_metadata:0xe01a530000000000/0xfffffffffffffffe,goto_table:241 cookie=0x6900000, duration=2819.237s, table=241, n_packets=32, n_bytes=4851, priority=61010,ip,metadata=0x1a530000000000/0x1fffff0000000000,dl_dst=fa:16:3e:f7:72:d3,nw_dst=172.20.1.117 actions=goto_table:242 cookie=0x6900000, duration=15546.579s, table=242, n_packets=3738, n_bytes=368372, priority=61010,icmp actions=goto_table:243 cookie=0x6900000, duration=2819.235s, table=243, n_packets=32, n_bytes=4851, priority=61010,ip,metadata=0x1a530000000000/0x1fffff0000000000 actions=ct(table=244,zone=5021) cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50,ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220) cookie=0x6900000, duration=15546.552s, table=244, n_packets=3819, n_bytes=431050, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220) cookie=0x8000007, duration=2819.193s, table=220, n_packets=107, n_bytes=9431, priority=7,reg6=0xe01a5300 actions=output:108 Thanks Rohith From: Darrell Ball <[email protected]> Date: Thursday, 24 August 2017 at 12:20 AM To: Rohith Basavaraja <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi Rohith I might have missed the alias earlier. From the below o/p, I see the rule cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220) not being hit. I also see the rule cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50, ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop having a drop action. What is the expectation of the test ? Is table 244 intended to drop non-related and non-established packets ? Thanks Darrell From: <[email protected]> on behalf of Rohith Basavaraja <[email protected]> Date: Wednesday, August 23, 2017 at 3:03 AM To: "[email protected]" <[email protected]> Subject: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK Hi, I see that if I have following rules, i.e not allow any new connections and allow only established and related flows, cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50, ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220) cookie=0x6900000, duration=15546.552s, table=244, n_packets=3819, n_bytes=431050, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220) We are still seeing that new connections are getting allowed, we see this behavior/issue only OVS + DPDK and not in OVS kernel mode. Wanted to check if this issue is already reported elsewhere or it’s new issue. Thanks Rohith
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
