On Fri, Aug 7, 2020 at 12:10 AM Tony Liu <[email protected]> wrote: > > Inline... > > Thanks! > > Tony > > -----Original Message----- > > From: Numan Siddique <[email protected]> > > Sent: Thursday, August 6, 2020 10:03 AM > > To: Tony Liu <[email protected]> > > Cc: [email protected]; [email protected] > > Subject: Re: [ovs-dev] packet drop > > > > > > > > On Thu, Aug 6, 2020 at 4:05 AM Tony Liu <[email protected] > > <mailto:[email protected]> > wrote: > > > > > > > > The drop is caused by flow change. > > > > When packet is dropped. > > ======== > > recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge > > neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(- > > df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00 > > /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0 > > xf8), packets:14, bytes:1372, used:0.846s, actions:drop > > recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df: > > 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10. > > 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no > > <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3, > > ttl=64,frag=no> ),icmp(type=0), packets:6, bytes:588, used:8.983s, > > actions:drop > > ======== > > > > When packet goes through. > > ======== > > recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge > > neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(- > > df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00 > > /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0 > > xf8), packets:3, bytes:294, used:0.104s, actions:12 > > recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df: > > 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10. > > 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no > > <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3, > > ttl=64,frag=no> ),icmp(type=0), packets:3, bytes:294, used:0.103s, > > actions:ct_clear,set(tunnel(tun_id=0x1a8ee,dst=10.6.30.92,ttl=64,tp_dst= > > 6081,geneve({class=0x102,type=0x80,len=4,0x1000b}),flags(df|csum|key))), > > set(eth(src=fa:16:3e:75:b7:e5,dst=52:54:00:0c:ef:b9)),set(ipv4(ttl=63)), > > 3 > > ======== > > > > Is that flow programmed by ovn-controller via ovs-vswitchd? > > > > What version of OVN and OVS are you using ? > > ovn-20.03.0-2.el8.x86_64 > openvswitch-2.12.0-1.el8.x86_64 > > > Can you share your OVN NB DB ? > > Yes, I can. Let me know how. > > > If I understand correctly the packet is received from the patch port to > > br-int on the gateway node and then tunnelled to the compute node right ? > > And the packet is dropped on the compute node ? > > Yes and yes. > > > If you could share your NB DB if it's fine with you and tell the > > destination logical port, I can try it out locally. > > Here is what I had. > On compute node, ovn-controller is very busy. It keeps saying > "commit failed". > ==================== > 2020-08-05T02:44:23.927Z|04125|reconnect|INFO|tcp:10.6.20.84:6642: connected > 2020-08-05T02:44:23.936Z|04126|main|INFO|OVNSB commit failed, force recompute > next time. > 2020-08-05T02:44:23.938Z|04127|ovsdb_idl|INFO|tcp:10.6.20.84:6642: clustered > database server is disconnected from cluster; trying another server > 2020-08-05T02:44:23.939Z|04128|reconnect|INFO|tcp:10.6.20.84:6642: connection > attempt timed out > 2020-08-05T02:44:23.939Z|04129|reconnect|INFO|tcp:10.6.20.84:6642: waiting 2 > seconds before reconnect > ==================== > > The connection to local OVSDB keeps being dropped, because no probe > response. > ======== > 2020-08-05T02:47:15.437Z|04351|poll_loop|INFO|wakeup due to [POLLIN] on fd 20 > (10.6.20.22:42362<->10.6.20.86:6642) at lib/stream-fd.c:157 (100% CPU usage) > 2020-08-05T02:47:15.438Z|04352|reconnect|WARN|tcp:127.0.0.1:6640: connection > dropped (Broken pipe) > 2020-08-05T02:47:15.438Z|04353|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T02:47:15.449Z|04354|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: > connected > ======== > > After set probe interval to 30s, the problem seems gone. I mentioned > this in another thread "[OVN] no response to inactivity probe". > > I can restore probe interval back to default 5s, see if the problem > can be reproduced. For me, it's important to understand what happens > behind it.
Ok. This makes sense. Since the connection between ovn-controller and ovs-vswitchd is braking, the flows are deleted and reprogrammed and this results in a loop. And the packet gets dropped if flows are missing. I think increasing the probe interval - ovn-openflow-probe-interval seems fine to me. It indicates that ovn-controller is taking more than 5 seconds to do a full recompute. If you take ovn-20.06.01 then ovn-controller should not break often as there are many optimizations in the latest version to handle the DB changes more efficiently. Thanks Numan > > > > > Thanks > > Numan > > > > > > > > > > > > Thanks! > > > > Tony > > > > > -----Original Message----- > > > From: discuss <[email protected] <mailto:ovs- > > [email protected]> > On Behalf Of Tony > > > Liu > > > Sent: Wednesday, August 5, 2020 2:48 PM > > > To: [email protected] <mailto:ovs- > > [email protected]> ; [email protected] <mailto:ovs- > > [email protected]> > > > Subject: [ovs-discuss] packet drop > > > > > > Hi, > > > > > > I am running ping from external to VM via OVN gateway. > > > On the compute node, ICMP request packet is consistently coming > > into > > > interface "ovn-gatewa-1". But there is about 10 out of 25 packet > > loss on > > > tap interface. It's like the switch pauses 10s after every 15s. > > > > > > Has anyone experiences such issue? > > > Any advice how to look into it? > > > > > > ======== > > > 21fed09f-909e-4efc-b117-f5d5fcb636c9 > > > Bridge br-int > > > fail_mode: secure > > > datapath_type: system > > > Port "ovn-gatewa-0" > > > Interface "ovn-gatewa-0" > > > type: geneve > > > options: {csum="true", key=flow, > > remote_ip="10.6.30.91"} > > > bfd_status: {diagnostic="No Diagnostic", > > flap_count="1", > > > forwarding="true", remote_diagnostic="No Diagnostic", > > remote_state=up, > > > state=up} > > > Port "tap2588bb4e-35" > > > Interface "tap2588bb4e-35" > > > Port "ovn-gatewa-1" > > > Interface "ovn-gatewa-1" > > > type: geneve > > > options: {csum="true", key=flow, > > remote_ip="10.6.30.92"} > > > bfd_status: {diagnostic="No Diagnostic", > > flap_count="1", > > > forwarding="true", remote_diagnostic="No Diagnostic", > > remote_state=up, > > > state=up} > > > Port "tap37f6b2d7-cc" > > > Interface "tap37f6b2d7-cc" > > > Port "tap2c4b3b0f-8b" > > > Interface "tap2c4b3b0f-8b" > > > Port "tap23245491-a4" > > > Interface "tap23245491-a4" > > > Port "tap51660269-2c" > > > Interface "tap51660269-2c" > > > Port "tap276cd1ef-e1" > > > Interface "tap276cd1ef-e1" > > > Port "tap138526d3-b3" > > > Interface "tap138526d3-b3" > > > Port "tapd1ae48a1-2d" > > > Interface "tapd1ae48a1-2d" > > > Port br-int > > > Interface br-int > > > type: internal > > > Port "tapdd08f476-94" > > > Interface "tapdd08f476-94" > > > ======== > > > > > > > > > Thanks! > > > > > > Tony > > > > > > _______________________________________________ > > > discuss mailing list > > > [email protected] <mailto:[email protected]> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > _______________________________________________ > > dev mailing list > > [email protected] <mailto:[email protected]> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > _______________________________________________ > discuss mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
