Hi, I have upgraded kernel : uname -a Linux ubuntu 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
I am seeing this weird behavior, I see traffic as expected for sometime and then OVS start dropping all the packets. Below is the log I see: 2017-04-11T15:09:23.496Z|00083|dpif(handler15)|WARN|system@ovs-system: execute set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x4f) failed (Invalid argument) on packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:c0:5a:48,dl_dst=fa:16:3e:57:fb:ce,nw_src=51.0.0.8,nw_dst=51.0.0.14,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=52953,tp_dst=80,tcp_flags=syn tcp_csum:8603 mtu 0 2017-04-11T15:09:24.494Z|00084|dpif(handler15)|WARN|system@ovs-system: failed to put[create] (Invalid argument) ufid:ac4fb7ad-b03b-4e92-b4a9-ea0f7dfa20d3 recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:57:fb:ce),eth_type(0x0800),ipv4(src=51.0.0.8,dst=51.0.0.14,proto=6,tos=0/0xfc,ttl=64,frag=no),tcp(src=52953,dst=80),tcp_flags(0/0), actions:set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x50) 2017-04-11T15:09:28.425Z|00085|dpif(handler15)|WARN|Dropped 1 log messages in last 4 seconds (most recently, 4 seconds ago) due to excessive rate 2017-04-11T15:09:28.425Z|00086|dpif(handler15)|WARN|system@ovs-system: failed to put[create] (Invalid argument) ufid:cfea6416-11bf-40db-929e-419a16b7d0eb recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:4c:54:07),eth_type(0x0800),ipv4(src=51.0.0.8,dst=51.0.0.5,proto=6,tos=0/0xfc,ttl=64,frag=no),tcp(src=41485,dst=80),tcp_flags(0/0), actions:set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x51) 2017-04-11T15:09:28.425Z|00087|dpif(handler15)|WARN|system@ovs-system: execute set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x51) failed (Invalid argument) on packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:c0:5a:48,dl_dst=fa:16:3e:4c:54:07,nw_src=51.0.0.8,nw_dst=51.0.0.5,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=41485,tp_dst=80,tcp_flags=syn tcp_csum:2b4a mtu 0 2017-04-11T15:09:29.421Z|00088|dpif(handler15)|WARN|system@ovs-system: failed to put[create] (Invalid argument) ufid:cfea6416-11bf-40db-929e-419a16b7d0eb recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:4c:54:07),eth_type(0x0800),ipv4(src=51.0.0.8,dst=51.0.0.5,proto=6,tos=0/0xfc,ttl=64,frag=no),tcp(src=41485,dst=80),tcp_flags(0/0), actions:set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x52) 2017-04-11T15:09:29.422Z|00089|dpif(handler15)|WARN|system@ovs-system: execute set(eth(src=fa:16:3e:c0:5a:48,dst=fa:16:3e:db:81:44)),set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1,eth_type=0x8847),8,set(tunnel(tun_id=0x2,src=192.168.2.91,dst=192.168.2.141,ttl=64,flags(df|key))),8,pop_mpls(eth_type=0x800),recirc(0x52) failed (Invalid argument) on packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:c0:5a:48,dl_dst=fa:16:3e:4c:54:07,nw_src=51.0.0.8,nw_dst=51.0.0.5,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=41485,tp_dst=80,tcp_flags=syn tcp_csum:2a50 mtu 0 Openvswitch: -------------------- sudo dpkg -l | grep openvswitch ii openvswitch-common 2.6.1-1 amd64 Open vSwitch common components ii openvswitch-switch 2.6.1-1 amd64 Open vSwitch switch implementations On Thu, Apr 6, 2017 at 6:48 AM, f 62 <[email protected]> wrote: > Thanks Justin, > > Will upgrade kernel. > > On Thu, Apr 6, 2017 at 1:30 AM, Justin Pettit <[email protected]> wrote: > >> I know very little about the MPLS integration, but I have one thing that >> might be worth checking. Are you using the OVS kernel module that came >> with OVS 2.6.1 or the one that comes with Linux 4.2? Linux 4.2 came out a >> year before OVS 2.6, and 2.6 introduced MPLS TTL. It looks like the kernel >> datapath is rejecting the attempt to add that flow, so I'd make sure that >> you're using the kernel datapath that came with the OVS release. >> >> --Justin >> >> >> > On Apr 5, 2017, at 2:29 AM, f 62 <[email protected]> wrote: >> > >> > This is the log of ovs-vswitchd: >> > >> > 2017-04-05T05:45:45.944Z|00205|dpif(handler14)|WARN|system@ovs-system: >> failed to put[create] (Invalid argument) >> ufid:d62cd3db-17dd-488f-99bc-9da8b7e59c24 >> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_ >> mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label( >> 0/0),eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:78:13:cd),eth_ >> type(0x0800),ipv4(src=51.0.0.4,dst=51.0.0.13,proto=6,tos=0/ >> 0xfc,ttl=64,frag=no),tcp(src=48250,dst=80),tcp_flags(0/0), >> actions:set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e) >> ),set(tunnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2. >> 177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255, >> bos=1,eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst= >> fa:16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls( >> label=510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 >> > 2017-04-05T05:45:45.944Z|00206|dpif(handler14)|WARN|system@ovs-system: >> execute set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e)),set(tu >> nnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2.177,ttl=64, >> flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1, >> eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst=fa: >> 16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls(label= >> 510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 failed (Invalid argument) on >> packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:7f:19:13,dl_dst=fa:16:3e >> :78:13:cd,nw_src=51.0.0.4,nw_dst=51.0.0.13,nw_tos=0,nw_ecn= >> 0,nw_ttl=64,tp_src=48250,tp_dst=80,tcp_flags=syn tcp_csum:91ac >> > mtu 0 >> > 2017-04-05T05:45:53.952Z|00207|dpif(handler14)|WARN|system@ovs-system: >> failed to put[create] (Invalid argument) >> ufid:d62cd3db-17dd-488f-99bc-9da8b7e59c24 >> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_ >> mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label( >> 0/0),eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:78:13:cd),eth_ >> type(0x0800),ipv4(src=51.0.0.4,dst=51.0.0.13,proto=6,tos=0/ >> 0xfc,ttl=64,frag=no),tcp(src=48250,dst=80),tcp_flags(0/0), >> actions:set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e) >> ),set(tunnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2. >> 177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255, >> bos=1,eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst= >> fa:16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls( >> label=510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 >> > 2017-04-05T05:45:53.952Z|00208|dpif(handler14)|WARN|system@ovs-system: >> execute set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e)),set(tu >> nnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2.177,ttl=64, >> flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1, >> eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst=fa: >> 16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls(label= >> 510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 failed (Invalid argument) on >> packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:7f:19:13,dl_dst=fa:16:3e >> :78:13:cd,nw_src=51.0.0.4,nw_dst=51.0.0.13,nw_tos=0,nw_ecn= >> 0,nw_ttl=64,tp_src=48250,tp_dst=80,tcp_flags=syn tcp_csum:89da >> > mtu 0 >> > 2017-04-05T05:46:09.984Z|00209|dpif(handler14)|WARN|system@ovs-system: >> failed to put[create] (Invalid argument) >> ufid:d62cd3db-17dd-488f-99bc-9da8b7e59c24 >> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(10),skb_ >> mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label( >> 0/0),eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:78:13:cd),eth_ >> type(0x0800),ipv4(src=51.0.0.4,dst=51.0.0.13,proto=6,tos=0/ >> 0xfc,ttl=64,frag=no),tcp(src=48250,dst=80),tcp_flags(0/0), >> actions:set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e) >> ),set(tunnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2. >> 177,ttl=64,flags(df|key))),push_mpls(label=511,tc=0,ttl=255, >> bos=1,eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst= >> fa:16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls( >> label=510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 >> > 2017-04-05T05:46:09.984Z|00210|dpif(handler14)|WARN|system@ovs-system: >> execute set(eth(src=fa:16:3e:7f:19:13,dst=fa:16:3e:48:cd:9e)),set(tu >> nnel(tun_id=0x49,src=192.168.2.91,dst=192.168.2.177,ttl=64, >> flags(df|key))),push_mpls(label=511,tc=0,ttl=255,bos=1, >> eth_type=0x8847),9,set(eth(src=fa:16:3e:7f:19:13,dst=fa: >> 16:3e:78:13:cd)),pop_mpls(eth_type=0x800),push_mpls(label= >> 510,tc=0,ttl=254,bos=1,eth_type=0x8847),9 failed (Invalid argument) on >> packet tcp,vlan_tci=0x0000,dl_src=fa:16:3e:7f:19:13,dl_dst=fa:16:3e >> :78:13:cd,nw_src=51.0.0.4,nw_dst=51.0.0.13,nw_tos=0,nw_ecn= >> 0,nw_ttl=64,tp_src=48250,tp_dst=80,tcp_flags=syn tcp_csum:7a32 >> > mtu 0 >> > >> > >> > >> > On Wed, Apr 5, 2017 at 10:28 AM, f 62 <[email protected]> wrote: >> > I have configured ovs flows like this on ovs bridge: >> > >> > 1. cookie=0x0, duration=2261.104s, table=0, n_packets=23, n_bytes=1702, >> idle_age=1877, tcp,in_port=8,nw_src=51.0.0.4 actions=group:1,resubmit(8,7) >> > >> > 2. group_id=1,type=select,bucket=actions=set_field:fa:16:3e:48: >> cd:9e->eth_dst,resubmit(,5) >> > >> > 3. cookie=0xb2c6f4127f2f27cc, duration=24849.587s, table=5, >> n_packets=303, n_bytes=22830, idle_age=1940, >> priority=0,ip,dl_dst=fa:16:3e:48:cd:9e >> actions=push_mpls:0x8847,load:0x1ff->OXM_OF_MPLS_LABEL[],set >> _mpls_ttl(255),mod_vlan_vid:3,output:2 >> > >> > 4. cookie=0x0, duration=17142.766s, table=7, n_packets=36, >> n_bytes=2664, idle_age=1996, priority=2,ip,dl_dst=fa:16:3e:78:13:cd >> actions=push_mpls:0x8847,load:0x1fe->OXM_OF_MPLS_LABEL[],set >> _mpls_ttl(254),mod_vlan_vid:3,output:2 >> > >> > ovs port:2 is patch port to tunnel bridge (another ovs bridge) >> > >> > On tunnel bridge, this is the flow: >> > >> > 5. cookie=0x0, duration=2465.760s, table=22, n_packets=36, >> n_bytes=2664, idle_age=267, priority=1,dl_vlan=3 >> actions=strip_vlan,load:0x49->NXM_NX_TUN_ID[],output:2 >> > >> > If in (3) and (4) I do not set MPLS fields, traffic works as expected. >> > >> > Other relevant info: >> > cat /proc/version >> > Linux version 4.2.0-42-generic (buildd@lgw01-55) (gcc version 4.8.4 >> (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #49~14.04.1-Ubuntu SMP Wed Jun 29 >> 20:22:11 UTC 2016 >> > >> > ovs-vswitchd --version >> > ovs-vswitchd (Open vSwitch) 2.6.1 >> > >> > sudo ovs-dpctl show >> > system@ovs-system: >> > lookups: hit:605248 missed:239296 lost:1270 >> > flows: 0 >> > masks: hit:1966804 total:1 hit/pkt:2.33 >> > port 0: ovs-system (internal) >> > port 1: br-int (internal) >> > port 2: br-ex (internal) >> > port 3: br-tun (internal) >> > port 4: tape79dba01-eb (internal) >> > port 5: qr-fc1d6bca-50 (internal) >> > port 6: qg-27b44ec1-74 (internal) >> > port 7: qr-1dd26eee-a7 (internal) >> > port 8: tap29132628-3d (internal) >> > port 9: vxlan_sys_4789 (vxlan) >> > port 10: tap605a9cd5-98 >> > port 11: tapb80b5a72-d7 >> > port 12: qr-815256d5-48 (internal) >> > port 13: tap96f28437-46 (internal) >> > port 14: tap2ad1704d-cd >> > port 15: qr-fdfee796-74 (internal) >> > port 16: tapea2cd065-96 >> > port 17: tapd75ec3fa-97 >> > port 18: tapa8e07edf-0e >> > >> > sudo ovs-vsctl show >> > 52d8f41f-ba03-4006-b666-6d36c79f32d2 >> > Manager "ptcp:6640:127.0.0.1" >> > is_connected: true >> > Bridge br-ex >> > Controller "tcp:127.0.0.1:6633" >> > is_connected: true >> > fail_mode: secure >> > Port br-ex >> > Interface br-ex >> > type: internal >> > Port phy-br-ex >> > Interface phy-br-ex >> > type: patch >> > options: {peer=int-br-ex} >> > Bridge br-int >> > Controller "tcp:127.0.0.1:6633" >> > is_connected: true >> > fail_mode: secure >> > Port "tap96f28437-46" >> > tag: 4 >> > Interface "tap96f28437-46" >> > type: internal >> > Port patch-tun >> > Interface patch-tun >> > type: patch >> > options: {peer=patch-int} >> > Port "tap29132628-3d" >> > tag: 3 >> > Interface "tap29132628-3d" >> > type: internal >> > Port "qr-fc1d6bca-50" >> > tag: 1 >> > Interface "qr-fc1d6bca-50" >> > type: internal >> > Port "qr-815256d5-48" >> > tag: 3 >> > Interface "qr-815256d5-48" >> > type: internal >> > Port "qg-27b44ec1-74" >> > tag: 2 >> > Interface "qg-27b44ec1-74" >> > type: internal >> > Port int-br-ex >> > Interface int-br-ex >> > type: patch >> > options: {peer=phy-br-ex} >> > Port "tapea2cd065-96" >> > tag: 3 >> > Interface "tapea2cd065-96" >> > Port "tap605a9cd5-98" >> > tag: 3 >> > Interface "tap605a9cd5-98" >> > Port "tap2ad1704d-cd" >> > tag: 4 >> > Interface "tap2ad1704d-cd" >> > Port br-int >> > Interface br-int >> > type: internal >> > Port "tapa8e07edf-0e" >> > tag: 3 >> > Interface "tapa8e07edf-0e" >> > Port "tape79dba01-eb" >> > tag: 1 >> > Interface "tape79dba01-eb" >> > type: internal >> > Port "qr-1dd26eee-a7" >> > tag: 1 >> > Interface "qr-1dd26eee-a7" >> > type: internal >> > Port "qr-fdfee796-74" >> > tag: 4 >> > Interface "qr-fdfee796-74" >> > type: internal >> > Port "tapb80b5a72-d7" >> > tag: 3 >> > Interface "tapb80b5a72-d7" >> > Port "tapd75ec3fa-97" >> > tag: 3 >> > Interface "tapd75ec3fa-97" >> > Bridge br-tun >> > Controller "tcp:127.0.0.1:6633" >> > is_connected: true >> > fail_mode: secure >> > Port patch-int >> > Interface patch-int >> > type: patch >> > options: {peer=patch-tun} >> > Port br-tun >> > Interface br-tun >> > type: internal >> > Port "vxlan-c0a802b1" >> > Interface "vxlan-c0a802b1" >> > type: vxlan >> > options: {df_default="true", in_key=flow, >> local_ip="192.168.2.91", out_key=flow, remote_ip="192.168.2.177"} >> > ovs_version: "2.6.1" >> > >> > >> > _______________________________________________ >> > discuss mailing list >> > [email protected] >> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> >> >
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
