Hi Jarno,
2016-12-03, Jarno Rajahalme:
I sent a patch to fix this on OVS master. After it is reviewed I’ll
backport it to branch-2.5 (it only needs trivial fixes to apply, so
you may want to try that as well).
I tested the patch (on branch-2.5) and have mixed results.
In the same setup as the one on which the bug was observed, on an
ovs-appctl fdb/flush the assert/restart does not trigger anymore, but
OVS logs many "failed to setup flow" messages:
2016-12-09T09:11:09.262Z|00079|dpif(handler8)|WARN|system@ovs-system:
failed to put[create] (Invalid argument)
ufid:6c66bfac-26ea-4232-a228-d51d2dfd5752
recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(17),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:4d:f9:0c,dst=00:00:5e:00:43:64),eth_type(0x0800),ipv4(src=192.168.20.7/0.0.0.0,dst=192.168.10.7,proto=6/0,tos=0x10/0xfc,ttl=64,frag=no),tcp(src=22/0,dst=59808/0),tcp_flags(0/0),
actions:11,set(ipv4(dst=192.168.10.7,ttl=63)),set(tunnel(src=192.168.122.81,dst=192.168.122.82,ttl=64,flags(df))),push_mpls(label=173,tc=4,ttl=63,bos=1,eth_type=0x8847),16,pop_mpls(eth_type=0x800),set(ipv4(dst=192.168.10.7,tos=0x10/0xfc,ttl=64)),push_vlan(vid=8,pcp=0),1,pop_vlan,14
2016-12-09T09:11:09.263Z|00080|dpif(handler8)|WARN|system@ovs-system:
execute
11,set(ipv4(dst=192.168.10.7,ttl=63)),set(tunnel(src=192.168.122.81,dst=192.168.122.82,ttl=64,flags(df))),push_mpls(label=173,tc=4,ttl=63,bos=1,eth_type=0x8847),16,pop_mpls(eth_type=0x800),set(ipv4(dst=192.168.10.7,tos=0x10/0xfc,ttl=64)),push_vlan(vid=8,pcp=0),1,pop_vlan,14
failed (Invalid argument) on packet
tcp,vlan_tci=0x0000,dl_src=fa:16:3e:4d:f9:0c,dl_dst=00:00:5e:00:43:64,nw_src=192.168.20.7,nw_dst=192.168.10.7,nw_tos=16,nw_ecn=0,nw_ttl=64,tp_src=22,tp_dst=59808,tcp_flags=psh|ack
tcp_csum:e3c
mtu 0
(7 times on one test, 6 times on another, 9 on yet another)
However, this serie of logs stops, and the traffic is actually restored.
(Note that the modules is DKMS 2.5.1)
Best,
-Thomas
On Dec 1, 2016, at 5:57 PM, Jarno Rajahalme <ja...@ovn.org
<mailto:ja...@ovn.org>> wrote:
On Nov 30, 2016, at 8:50 PM, Ben Pfaff <b...@ovn.org
<mailto:b...@ovn.org>> wrote:
On Wed, Nov 30, 2016 at 06:58:57PM -0800, Jarno Rajahalme wrote:
On Nov 30, 2016, at 8:41 AM, Ben Pfaff <b...@ovn.org
<mailto:b...@ovn.org>> wrote:
On Wed, Nov 30, 2016 at 12:05:46PM +0100, Thomas Morin wrote:
Hi Ben,
2016-11-30, Ben Pfaff:
Do you have any idea what in your OpenFlow pipeline might do that,
i.e. is there anything especially tricky in the OpenFlow flows?
Are you willing to show us your OpenFlow flow table?
The setup involves three OVS bridges connected with patch-ports:
br-int --
br-tun -- br-mpls, with the traffic that triggers the assert
being processed
by br-int with a NORMAL action (ie. MAC learning).
The flows in this setup aren't particularly tricky, I think,
although I'm
not sure what qualifies as tricky or non-tricky :)
Anyway, since yesterday I managed to identify the event that
trigger the
assert, by adding more logging before the assert and displaying
the actions
taken:
2016-11-29T14:44:40.126Z|00001|odp_util(revalidator45)|WARN|commit_set_ipv4_action
assert would fail....
2016-11-29T14:44:40.126Z|00002|odp_util(revalidator45)|WARN|
base_flow:
ip,in_port=5,dl_vlan=3,dl_vlan_pcp=0,dl_src=fa:16:3e:33:f7:fe,dl_dst=00:00:5e:00:43:64,nw_src=0.0.0.0,nw_dst=0.0.0.0,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
2016-11-29T14:44:40.126Z|00003|odp_util(revalidator45)|WARN|
flow:
tcp,in_port=5,dl_vlan=3,dl_vlan_pcp=0,dl_src=fa:16:3e:33:f7:fe,dl_dst=00:00:5e:00:43:64,nw_src=10.0.1.22,nw_dst=10.0.0.3,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=53295,tp_dst=8080,tcp_flags=psh|ack
2016-11-29T14:44:40.126Z|00004|odp_util(revalidator45)|WARN|
masks:
recirc_id=0xffffffff,reg0=0xffffffff,in_port=4294967295,dl_vlan=4095,dl_vlan_pcp=7,dl_src=ff:ff:ff:ff:ff:ff,dl_dst=ff:ff:ff:ff:ff:ff,dl_type=0xffff
2016-11-29T14:44:40.126Z|00005|odp_util(revalidator45)|WARN|
actions:
set(ipv4(src=10.0.1.22,dst=10.0.0.3,ttl=63)),set(eth(src=b8:2a:72:de:1b:e3,dst=00:17:cb:79:2c:01)),push_mpls(label=410384,tc=0,ttl=63,bos=1,eth_type=0x8847),9,set(eth(src=fa:16:3e:33:f7:fe,dst=00:00:5e:00:43:64)),pop_mpls(eth_type=0x800),push_vlan(vid=3,pcp=0),1
push_mpls clears L3/L4, while pop_mpls re-populates them, and then
processing the output to port 1 hits the assert?
That's what I'm thinking too.
Jarno, is this something you have time to look into? It'd be great, if
you do. I'm way behind.
I’m looking at this.
Based on the trace given it seems that:
1. Packet is received on br-int port 32, which outputs it via NORMAL
action over a patch port to another bridge. The only patch-port on
br-int is 2 (patch-tun). The NORMAL action adds dl_vlan=1.
2. br-tun receives the packet on in_port 1 (patch-int), and outputs
it on it’s port 2 (patch-to-mpls)
3. br-mpls receives the packet on it’s in_port 2 (patch-to-tun), does
pop_vlan, and outputs to it’s port 21 (ipvpn-pp-out), which is also
an patch port.
4. br-mpls (?) receives the packet on it’s in_port 20 (ipvpn-pp-in),
does
dec_ttl,push_mpls:0x8847,load:0x644c0->OXM_OF_MPLS_LABEL[],set_field:b8:2a:72:de:1b:e3->eth_src,set_field:00:17:cb:79:2c:01->eth_dst,output:1
All this generates a megaflow:
set(ipv4(src=10.0.1.23,dst=10.0.0.3,ttl=63)),set(eth(src=b8:2a:72:de:1b:e3,dst=00:17:cb:79:2c:01)),push_mpls(label=410816,tc=0,ttl=63,bos=1,eth_type=0x8847),9
This is only the beginning part of the trouble-some megaflow, in
which br-int sends the packet also to another port (vlan 3), and as
part of that pops the MPLS and restores the original ethernet
addresses. Maybe this would happen with the trace too, if you flushed
MACs before the trace?
The patch ports 21 and 20 appear to be in the same bridge and patched
to each other. Is this the case?
The crashing megaflow has in_port=5,dl_vlan=3. Is this also on br-int?
Also, OVS 2.6 is a little bit less aggressive about avoiding
recirculation after mpls operations, and I’d be interested to know if
your case fails the same way with OVS 2.6?
Thanks,
Jarno
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
