Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Darrell Ball]

Hi Jan

As discussed and agreed at OVSCON, I submitted a patch to bring the userspace 
connection tracker established state
In line with that of the kernel.
I used a similar patch to what I earlier suggested earlier in this thread, 
adding a test and also made some documentation updates.

Some of the discussion in this thread was somewhat orthogonal to bringing 
userspace ‘established’ in line with kernel
‘established’, but it appears to have been useful as some new recommendations 
may come out of it with respect to
recommended practices, for conntrack pipeline design.

Thanks Darrell


From: Jan Scheurich 
Date: Saturday, November 4, 2017 at 4:54 AM
To: Darrel Ball , Rohith Basavaraja 

Cc: "d...@openvswitch.org" 
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

The example pipeline I crafted was not meant to be a realistic conntrack 
application but to demonstrate the semantic differences between userspace and 
kernel implementation and to discuss our problems with the current 
documentation.

I fully agree with your proposal for a proper ICMP set of rules. It would work 
equally for kernel and userspace datapath. But there are other rule sets where 
there they behave differently and we believe this is not good.

The original pipeline brought up by Rohith in August is the implementation of 
OpenStack Security Groups in OpenDaylight. In general ODL does not commit 
connections in the untrusted direction. However, in the problematic scenario 
(two Neutron ports in the same Neutron Network but in different Security 
Groups, co-located on the same OVS instance) the connection was committed (as 
trusted) on the sending side. The packet should have been dropped on the 
receiving side but the ct() lookup for the first packet on egress hits the 
committed connection and passes because ODL uses one conntrack zone per Neutron 
Network rather than per Security Group. I think this is wrong and using one 
zone per Security Group would probably solve this specific issue.

But with the kernel datapath this issue never surfaced because the connection 
is not considered established prior to the first reply packet so that the 
second lookup of the first packet on egress still yields +new-est. So the ODL 
developers testing with kernel datapath assumed their design was suitable. You 
can argue that was a misunderstanding of the function but the discrepancy 
between documentation and kernel behavior certainly didn’t help.

Perhaps it is better we continue this discussion in person during the OVS 
conference?

Regards, Jan

From: Darrell Ball [mailto:db...@vmware.com]
Sent: Saturday, 04 November, 2017 01:47
To: Jan Scheurich ; Rohith Basavaraja 

Cc: d...@openvswitch.org
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK



From: Jan Scheurich 
>
Date: Friday, November 3, 2017 at 6:22 AM
To: Darrel Ball >, Rohith Basavaraja 
>
Cc: "d...@openvswitch.org" 
>
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct.

[Darrell] Sure, let us discuss the first example; let me know if you want to 
discuss the second example you gave as well.


Please note that it was not meant as realistic conntrack pipeline

Darrell] Again, I can agree your example is not realistic or recommended. No 
one would write rules like this.
  The rules would certainly be written properly so that the trusted 
direction (the one that does the commit) allows the first packet through;
  this is a fundamental principle of conntrack. There are an 
infinite number of ways to misuse conntrack rules and no one can prevent misuse.
  On a similar topic, another fundamental problem I saw with the 
original discussion (from Aug) is creating a conntrack pipeline that commits
  a connection in the untrusted direction. That is also not 
something we do or recommend others do. This ‘suboptimal design approach’ 
brought
  us to the question on when a packet gets labelled as ESTABLISHED. 
 Normally, the difference would not be noticed, since a connection would not be
  committed in the untrusted direction and hence EST would not be 
possible unless another rule correctly commits in the trusted direction.
  I’ll add more comments below.


but just to demonstrate the misalignment between userspace and kernel conntrack 
and the conflict of both with the documentation.

The following pipeline is now tested:

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Jan Scheurich]

… I think this is wrong and using one zone per Security Group would probably 
solve this specific issue.



[Darrell] Yes, using zoning differently was one suggestion back in Aug/2017 to 
solve the issue.
   However, there is a more fundamental problem.

Here is an excerpt sent from Rohith on Aug 29/2017 for the connection b/w VM1 
and VM2

“
VM1 and VM2 VMs are on same compute node but with different SGs.

For VM1, security rules configured are as below:
Egress/Ingress  Allow all

For VM2,
Egress  Allow all
Ingress  Allow only from VMs which are in same security group.

So if I ping from VM1 to VM2 then VM1 should be able to send the ICMP request 
but
It should not reach VM2 and it should get dropped at the ingress side of VM2.

Since Both VMs are in same network, they will landup in having same zones.

Since VM1 needs to allow all ingress/egress traffic it needs to have the 
following rule.

cookie=0x0, duration=128.074s, table=214, n_packets=1, n_bytes=98,
priority=1000,ct_state=+new+trk,ip,metadata=0x19f400/0x1f00 
actions=ct(commit,zone=5021),resubmit(,17)
“

VM1 has no access restrictions.

VM2 is a restricted access VM, however the connection (the only one being used 
b/w these VMs) is being committed in its untrusted
direction (i.e. coming into VM2).

<

[Jan] I think we all agree that design in ODL to use the same conntrack zone 
for VMs in different SGs is not correct given the semantics of the userspace 
conntrack. The connection should be tracked independently for each SG.

The problem is that the kernel conntrack semantics and behavior is different. 
That is what the ODL developers have based their implementation on (as it 
existed first). The connection committed on the on the sending side is not yet 
considered established at the second lookup on the receiving side because 
conntrack hasn’t seen a reply packet yet. Hence the ct() lookup still returns 
+trk+new and the ingress SG filter of VM2 drops the packet.

Our main point is that OVS conntrack behavior should be consistent between 
kernel and userspace, at least for such basic functionality. Since the kernel 
conntrack has set the standard (which is used also outside OVS) we think 
userspace conntrack should follow the kernel semantics and return +est only for 
connections that are established in both directions. And we should clarify the 
documentation accordingly.

Independent of that we will still push for a more correct use of zones in ODL 
Security Group implementation.

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Darrell Ball]

From: Jan Scheurich 
Date: Saturday, November 4, 2017 at 4:54 AM
To: Darrel Ball , Rohith Basavaraja 

Cc: "d...@openvswitch.org" 
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

The example pipeline I crafted was not meant to be a realistic conntrack 
application but to demonstrate the semantic differences between userspace and 
kernel implementation and to discuss our problems with the current 
documentation.

I fully agree with your proposal for a proper ICMP set of rules. It would work 
equally for kernel and userspace datapath. But there are other rule sets where 
there they behave differently and we believe this is not good.

The original pipeline brought up by Rohith in August is the implementation of 
OpenStack Security Groups in OpenDaylight. In general ODL does not commit 
connections in the untrusted direction. However, in the problematic scenario 
(two Neutron ports in the same Neutron Network but in different Security 
Groups, co-located on the same OVS instance) the connection was committed (as 
trusted) on the sending side. The packet should have been dropped on the 
receiving side but the ct() lookup for the first packet on egress hits the 
committed connection and passes because ODL uses one conntrack zone per Neutron 
Network rather than per Security Group. I think this is wrong and using one 
zone per Security Group would probably solve this specific issue.



[Darrell] Yes, using zoning differently was one suggestion back in Aug/2017 to 
solve the issue.
   However, there is a more fundamental problem.

Here is an excerpt sent from Rohith on Aug 29/2017 for the connection b/w VM1 
and VM2

“
VM1 and VM2 VMs are on same compute node but with different SGs.

For VM1, security rules configured are as below:
Egress/Ingress  Allow all

For VM2,
Egress  Allow all
Ingress  Allow only from VMs which are in same security group.

So if I ping from VM1 to VM2 then VM1 should be able to send the ICMP request 
but
It should not reach VM2 and it should get dropped at the ingress side of VM2.

Since Both VMs are in same network, they will landup in having same zones.

Since VM1 needs to allow all ingress/egress traffic it needs to have the 
following rule.

cookie=0x0, duration=128.074s, table=214, n_packets=1, n_bytes=98,
priority=1000,ct_state=+new+trk,ip,metadata=0x19f400/0x1f00 
actions=ct(commit,zone=5021),resubmit(,17)
“

VM1 has no access restrictions.

VM2 is a restricted access VM, however the connection (the only one being used 
b/w these VMs) is being committed in its untrusted
direction (i.e. coming into VM2).

<



But with the kernel datapath this issue never surfaced because the connection 
is not considered established prior to the first reply packet so that the 
second lookup of the first packet on egress still yields +new-est. So the ODL 
developers testing with kernel datapath assumed their design was suitable. You 
can argue that was a misunderstanding of the function but the discrepancy 
between documentation and kernel behavior certainly didn’t help.

Perhaps it is better we continue this discussion in person during the OVS 
conference?

Regards, Jan

From: Darrell Ball [mailto:db...@vmware.com]
Sent: Saturday, 04 November, 2017 01:47
To: Jan Scheurich ; Rohith Basavaraja 

Cc: d...@openvswitch.org
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK



From: Jan Scheurich 
>
Date: Friday, November 3, 2017 at 6:22 AM
To: Darrel Ball >, Rohith Basavaraja 
>
Cc: "d...@openvswitch.org" 
>
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct.

[Darrell] Sure, let us discuss the first example; let me know if you want to 
discuss the second example you gave as well.


Please note that it was not meant as realistic conntrack pipeline

Darrell] Again, I can agree your example is not realistic or recommended. No 
one would write rules like this.
  The rules would certainly be written properly so that the trusted 
direction (the one that does the commit) allows the first packet through;
  this is a fundamental principle of conntrack. There are an 
infinite number of ways to misuse conntrack rules and no one can prevent misuse.
  On a similar topic, another fundamental problem I saw with the 
original discussion (from Aug) is creating a conntrack pipeline that commits
  a connection in the 

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Rohith Basavaraja]

Hi Jan/Darrell,

Sorry I couldn’t respond earlier as I was on PTO on Friday and didn’t had mail 
access.
Thanks, Jan for detailing the problem and providing the examples. I also think
It will be good if we can get some time to discuss this during OVS conference.

I am still catching up on all the email discussion on this and will respond if 
there are any
Pending queries, I need to answer.


Thanks
Rohith




From: Jan Scheurich 
Date: Saturday, 4 November 2017 at 5:23 PM
To: Darrell Ball , Rohith Basavaraja 

Cc: "d...@openvswitch.org" 
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

The example pipeline I crafted was not meant to be a realistic conntrack 
application but to demonstrate the semantic differences between userspace and 
kernel implementation and to discuss our problems with the current 
documentation.

I fully agree with your proposal for a proper ICMP set of rules. It would work 
equally for kernel and userspace datapath. But there are other rule sets where 
there they behave differently and we believe this is not good.

The original pipeline brought up by Rohith in August is the implementation of 
OpenStack Security Groups in OpenDaylight. In general ODL does not commit 
connections in the untrusted direction. However, in the problematic scenario 
(two Neutron ports in the same Neutron Network but in different Security 
Groups, co-located on the same OVS instance) the connection was committed (as 
trusted) on the sending side. The packet should have been dropped on the 
receiving side but the ct() lookup for the first packet on egress hits the 
committed connection and passes because ODL uses one conntrack zone per Neutron 
Network rather than per Security Group. I think this is wrong and using one 
zone per Security Group would probably solve this specific issue.

But with the kernel datapath this issue never surfaced because the connection 
is not considered established prior to the first reply packet so that the 
second lookup of the first packet on egress still yields +new-est. So the ODL 
developers testing with kernel datapath assumed their design was suitable. You 
can argue that was a misunderstanding of the function but the discrepancy 
between documentation and kernel behavior certainly didn’t help.

Perhaps it is better we continue this discussion in person during the OVS 
conference?

Regards, Jan

From: Darrell Ball [mailto:db...@vmware.com]
Sent: Saturday, 04 November, 2017 01:47
To: Jan Scheurich ; Rohith Basavaraja 

Cc: d...@openvswitch.org
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK



From: Jan Scheurich 
>
Date: Friday, November 3, 2017 at 6:22 AM
To: Darrel Ball >, Rohith Basavaraja 
>
Cc: "d...@openvswitch.org" 
>
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct.

[Darrell] Sure, let us discuss the first example; let me know if you want to 
discuss the second example you gave as well.


Please note that it was not meant as realistic conntrack pipeline

Darrell] Again, I can agree your example is not realistic or recommended. No 
one would write rules like this.
  The rules would certainly be written properly so that the trusted 
direction (the one that does the commit) allows the first packet through;
  this is a fundamental principle of conntrack. There are an 
infinite number of ways to misuse conntrack rules and no one can prevent misuse.
  On a similar topic, another fundamental problem I saw with the 
original discussion (from Aug) is creating a conntrack pipeline that commits
  a connection in the untrusted direction. That is also not 
something we do or recommend others do. This ‘suboptimal design approach’ 
brought
  us to the question on when a packet gets labelled as ESTABLISHED. 
 Normally, the difference would not be noticed, since a connection would not be
  committed in the untrusted direction and hence EST would not be 
possible unless another rule correctly commits in the trusted direction.
  I’ll add more comments below.


but just to demonstrate the misalignment between userspace and kernel conntrack 
and the conflict of both with the documentation.

The following pipeline is now tested:

ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,icmp 
actions=ct(table=1,zone=5000)"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,arp actions=output:2"
ovs-ofctl 

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Jan Scheurich]

Hi Darrel,

The example pipeline I crafted was not meant to be a realistic conntrack 
application but to demonstrate the semantic differences between userspace and 
kernel implementation and to discuss our problems with the current 
documentation.

I fully agree with your proposal for a proper ICMP set of rules. It would work 
equally for kernel and userspace datapath. But there are other rule sets where 
there they behave differently and we believe this is not good.

The original pipeline brought up by Rohith in August is the implementation of 
OpenStack Security Groups in OpenDaylight. In general ODL does not commit 
connections in the untrusted direction. However, in the problematic scenario 
(two Neutron ports in the same Neutron Network but in different Security 
Groups, co-located on the same OVS instance) the connection was committed (as 
trusted) on the sending side. The packet should have been dropped on the 
receiving side but the ct() lookup for the first packet on egress hits the 
committed connection and passes because ODL uses one conntrack zone per Neutron 
Network rather than per Security Group. I think this is wrong and using one 
zone per Security Group would probably solve this specific issue.

But with the kernel datapath this issue never surfaced because the connection 
is not considered established prior to the first reply packet so that the 
second lookup of the first packet on egress still yields +new-est. So the ODL 
developers testing with kernel datapath assumed their design was suitable. You 
can argue that was a misunderstanding of the function but the discrepancy 
between documentation and kernel behavior certainly didn’t help.

Perhaps it is better we continue this discussion in person during the OVS 
conference?

Regards, Jan

From: Darrell Ball [mailto:db...@vmware.com]
Sent: Saturday, 04 November, 2017 01:47
To: Jan Scheurich ; Rohith Basavaraja 

Cc: d...@openvswitch.org
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK



From: Jan Scheurich 
>
Date: Friday, November 3, 2017 at 6:22 AM
To: Darrel Ball >, Rohith Basavaraja 
>
Cc: "d...@openvswitch.org" 
>
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct.

[Darrell] Sure, let us discuss the first example; let me know if you want to 
discuss the second example you gave as well.


Please note that it was not meant as realistic conntrack pipeline

Darrell] Again, I can agree your example is not realistic or recommended. No 
one would write rules like this.
  The rules would certainly be written properly so that the trusted 
direction (the one that does the commit) allows the first packet through;
  this is a fundamental principle of conntrack. There are an 
infinite number of ways to misuse conntrack rules and no one can prevent misuse.
  On a similar topic, another fundamental problem I saw with the 
original discussion (from Aug) is creating a conntrack pipeline that commits
  a connection in the untrusted direction. That is also not 
something we do or recommend others do. This ‘suboptimal design approach’ 
brought
  us to the question on when a packet gets labelled as ESTABLISHED. 
 Normally, the difference would not be noticed, since a connection would not be
  committed in the untrusted direction and hence EST would not be 
possible unless another rule correctly commits in the trusted direction.
  I’ll add more comments below.


but just to demonstrate the misalignment between userspace and kernel conntrack 
and the conflict of both with the documentation.

The following pipeline is now tested:

ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,icmp 
actions=ct(table=1,zone=5000)"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,arp actions=output:2"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=2 actions=output:1"
ovs-ofctl add-flow br0 
"table=1,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"
ovs-ofctl add-flow br0 "table=1,priority=10,in_port=1,ip,ct_state=+new+trk 
actions=ct(commit,zone=5000),goto_table:2"
ovs-ofctl add-flow br0 
"table=2,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"

The ct(commit) action in table 1 commits a new connection entry, but the 
subsequent match in table 2 proves that the ct_state of the packet is still not 
EST despite the commit.

>

[Darrell] I assumed you meant “lack of match in table 2” per your following 
test result.
   You use zoning in your 

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Darrell Ball]

From: Jan Scheurich 
Date: Friday, November 3, 2017 at 6:22 AM
To: Darrel Ball , Rohith Basavaraja 

Cc: "d...@openvswitch.org" 
Subject: RE: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct.

[Darrell] Sure, let us discuss the first example; let me know if you want to 
discuss the second example you gave as well.


Please note that it was not meant as realistic conntrack pipeline

Darrell] Again, I can agree your example is not realistic or recommended. No 
one would write rules like this.
  The rules would certainly be written properly so that the trusted 
direction (the one that does the commit) allows the first packet through;
  this is a fundamental principle of conntrack. There are an 
infinite number of ways to misuse conntrack rules and no one can prevent misuse.
  On a similar topic, another fundamental problem I saw with the 
original discussion (from Aug) is creating a conntrack pipeline that commits
  a connection in the untrusted direction. That is also not 
something we do or recommend others do. This ‘suboptimal design approach’ 
brought
  us to the question on when a packet gets labelled as ESTABLISHED. 
 Normally, the difference would not be noticed, since a connection would not be
  committed in the untrusted direction and hence EST would not be 
possible unless another rule correctly commits in the trusted direction.
  I’ll add more comments below.


but just to demonstrate the misalignment between userspace and kernel conntrack 
and the conflict of both with the documentation.

The following pipeline is now tested:

ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,icmp 
actions=ct(table=1,zone=5000)"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,arp actions=output:2"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=2 actions=output:1"
ovs-ofctl add-flow br0 
"table=1,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"
ovs-ofctl add-flow br0 "table=1,priority=10,in_port=1,ip,ct_state=+new+trk 
actions=ct(commit,zone=5000),goto_table:2"
ovs-ofctl add-flow br0 
"table=2,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"

The ct(commit) action in table 1 commits a new connection entry, but the 
subsequent match in table 2 proves that the ct_state of the packet is still not 
EST despite the commit.

>

[Darrell] I assumed you meant “lack of match in table 2” per your following 
test result.
   You use zoning in your rules without effect and you even split 
the pipeline with goto_table – we would not do this.
   Out of 6 rules, probably at least 4 of them are not what you 
want.
   I think there is a big disconnect here and I feel we are wasting 
time discussing such a contrived pipeline.

Here is a simplified set of rules that might be reasonably used for just icmp:

priority=1,action=drop
priority=10,arp,action=normal
table=0,priority=10,in_port=2,icmp,ct_state=-trk,action=ct(table=0)
table=0,priority=10,in_port=2,icmp,ct_state=+trk+est actions=output:1
table=0,priority=10,in_port=1,icmp actions=ct(commit,table=1)
table=1,priority=10,in_port=1,icmp,ct_state=+trk+est actions=output:2
table=1,priority=10,in_port=1,icmp,ct_state=+trk+new actions=output:2

<


This contradicts the statement in man ovs-fields: “est (0x02)  Part of an 
existing connection. Set to 1 if this is a committed connection.”

>
[Darrell]
No, it does not.
Same answer as earlier

[Darrell   Let me clear up some misconceptions, ct(commit ) is a prerequisite 
for EST being set for a later packet seen. A ‘packet’ (not a connection) that 
hits an
 existing conntrack entry is marked as established and that’s 
what the documentation says; the idea behind the ‘OVS conntrack EST state’ is 
to make
 the state intuitive.

“est (0x02)  Part of an existing connection. Set to 1 if this is a committed 
connection.”

ESTABLISHED is an attribute of a packet hitting an existing conntrack entry 
(“Part of an existing connection”), not the conntrack entry itself.
So, a packet that hits an ‘existing’ entry (which is a committed connection, of 
course) gets its state set to EST.
I agree this is subtle, because the reader has to know that EST is a state of 
the packet not the connection entry itself; the wording could have been better.
<


Consequently the userspace datapath drops the first ICMP packet:

root@ubuntu:~# ip netns exec ns1 ping -c1 192.168.10.20
PING 192.168.10.20 (192.168.10.20) 56(84) bytes of data.
--- 192.168.10.20 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@ubuntu:/opt/ovs# ovs-ofctl -Oopenflow13 

Re: [ovs-dev] [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

[Jan Scheurich]

Hi Darrel,

I have now been able to actually test the example pipelines I provided earlier. 
Turns out that the first one I sent was correct. Please note that it was not 
meant as realistic conntrack pipeline but just to demonstrate the misalignment 
between userspace and kernel conntrack and the conflict of both with the 
documentation.

The following pipeline is now tested:

ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,icmp 
actions=ct(table=1,zone=5000)"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=1,arp actions=output:2"
ovs-ofctl add-flow br0 "table=0,priority=10,in_port=2 actions=output:1"
ovs-ofctl add-flow br0 
"table=1,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"
ovs-ofctl add-flow br0 "table=1,priority=10,in_port=1,ip,ct_state=+new+trk 
actions=ct(commit,zone=5000),goto_table:2"
ovs-ofctl add-flow br0 
"table=2,priority=10,in_port=1,ct_state=-new+est-rel-inv+trk actions=output:2"

The ct(commit) action in table 1 commits a new connection entry, but the 
subsequent match in table 2 proves that the ct_state of the packet is still not 
EST despite the commit. This contradicts the statement in man ovs-fields: “est 
(0x02)  Part of an existing connection. Set to 1 if this is a committed 
connection.”

Consequently the userspace datapath drops the first ICMP packet:

root@ubuntu:~# ip netns exec ns1 ping -c1 192.168.10.20
PING 192.168.10.20 (192.168.10.20) 56(84) bytes of data.
--- 192.168.10.20 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@ubuntu:/opt/ovs# ovs-ofctl -Oopenflow13 dump-flows br0
cookie=0x0, duration=30.885s, table=0, n_packets=1, n_bytes=98, reset_counts 
priority=10,icmp,in_port="br0-ns1" actions=ct(table=1,zone=5000)
cookie=0x0, duration=30.848s, table=0, n_packets=0, n_bytes=0, reset_counts 
priority=10,arp,in_port="br0-ns1" actions=output:"br0-ns2"
cookie=0x0, duration=30.815s, table=0, n_packets=0, n_bytes=0, reset_counts 
priority=10,in_port="br0-ns2" actions=output:"br0-ns1"
cookie=0x0, duration=30.783s, table=1, n_packets=0, n_bytes=0, reset_counts 
priority=10,ct_state=-new+est-rel-inv+trk,in_port="br0-ns1" 
actions=output:"br0-ns2"
cookie=0x0, duration=30.746s, table=1, n_packets=1, n_bytes=98, reset_counts 
priority=10,ct_state=+new+trk,ip,in_port="br0-ns1" 
actions=ct(commit,zone=5000),resubmit(,2)
cookie=0x0, duration=30.712s, table=2, n_packets=0, n_bytes=0, reset_counts 
priority=10,ct_state=-new+est-rel-inv+trk,in_port="br0-ns1" 
actions=output:"br0-ns2"

root@ubuntu:/opt/ovs# ovs-appctl dpctl/dump-flows
recirc_id(0),in_port(2),packet_type(ns=0,id=0),eth_type(0x0806), packets:0, 
bytes:0, used:never, actions:3
recirc_id(0),in_port(3),packet_type(ns=0,id=0),eth_type(0x0806), packets:0, 
bytes:0, used:never, actions:2
ct_state(+new-est-rel-inv+trk),recirc_id(0x2),in_port(2),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no),
 packets:0, bytes:0, used:never, actions:ct(commit,zone=5000)
recirc_id(0),in_port(2),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(proto=1,frag=no),
 packets:0, bytes:0, used:never, actions:ct(zone=5000),recirc(0x2)

root@ubuntu:/opt/ovs# ovs-appctl dpctl/dump-conntrack
icmp,orig=(src=192.168.10.10,dst=192.168.10.20,id=20697,type=8,code=0),reply=(src=192.168.10.20,dst=192.168.10.10,id=20697,type=0,code=0),zone=5000

But when I send two ICMP packets in a row, the second packet hits the 
connection entry committed by the first dropped packet and goes through:

root@ubuntu:~# ip netns exec ns1 ping -c2 192.168.10.20
PING 192.168.10.20 (192.168.10.20) 56(84) bytes of data.
64 bytes from 192.168.10.20: icmp_seq=2 ttl=64 time=1.87 ms
--- 192.168.10.20 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1006ms
rtt min/avg/max/mdev = 1.874/1.874/1.874/0.000 ms

root@ubuntu:/opt/ovs# ovs-ofctl -Oopenflow13 dump-flows br0
cookie=0x0, duration=40.727s, table=0, n_packets=2, n_bytes=196, reset_counts 
priority=10,icmp,in_port="br0-ns1" actions=ct(table=1,zone=5000)
cookie=0x0, duration=40.696s, table=0, n_packets=1, n_bytes=42, reset_counts 
priority=10,arp,in_port="br0-ns1" actions=output:"br0-ns2"
cookie=0x0, duration=40.667s, table=0, n_packets=2, n_bytes=140, reset_counts 
priority=10,in_port="br0-ns2" actions=output:"br0-ns1"
cookie=0x0, duration=40.631s, table=1, n_packets=1, n_bytes=98, reset_counts 
priority=10,ct_state=-new+est-rel-inv+trk,in_port="br0-ns1" 
actions=output:"br0-ns2"
cookie=0x0, duration=40.602s, table=1, n_packets=1, n_bytes=98, reset_counts 
priority=10,ct_state=+new+trk,ip,in_port="br0-ns1" 
actions=ct(commit,zone=5000),resubmit(,2)
cookie=0x0, duration=40.566s, table=2, n_packets=0, n_bytes=0, reset_counts 
priority=10,ct_state=-new+est-rel-inv+trk,in_port="br0-ns1" 
actions=output:"br0-ns2"

root@ubuntu:/opt/ovs# ovs-appctl dpctl/dump-flows
recirc_id(0),in_port(2),packet_type(ns=0,id=0),eth_type(0x0806), packets:0, 
bytes:0, used:never, actions:3
recirc_id(0),in_port(3),packet_type(ns=0,id=0),eth_type(0x0806), packets:0,