Acutally, I do think I have a megaflow (or other caching) issue. We use OVS for L2 packet forwarding; that is, given a packet, we don't need OVS to look at other protocols beyond the Ethernet layer. Additionally, we use VXLAN to establish L2 overlay networks across multiple OVS servers.
Just to make thing clear, these are some typical flow rules that you might see on a bridge: - in_port=1,actions=2,3 - in_port=42,actions=FLOOD - actions=NORMAL No IP matching, conntrack, etc. We're experiencing severe performance issues with OVS - in this use case, it cannot handle more than couple thousand packets/s. After some exploring, I've noticed that the installed megaflows try to match on fields that are not present in the rules, apparently for no reason. Here's a complete example to reproduce, using OVS 2.12.0: # ip link add dev a-blue type veth peer name a-red # ip link add dev b-blue type veth peer name b-red # ovs-vsctl add-br br0 # ovs-vsctl add-port br0 a-blue # ovs-vsctl add-port br0 b-blue # ovs-ofctl del-flows br0 # ovs-ofctl add-flow br0 in_port=a-blue,actions=b-blue # ovs-ofctl add-flow br0 in_port=b-blue,actions=a-blue After injecting ~100 random packets (IP, IPv6, TCP, UDP, ARP with random addresses) to one of the red interfaces (with https://pastebin.com/Y6dPFCKJ), these are the installed flows: # ovs-dpctl dump-flows recirc_id(0),in_port(2),eth(),eth_type(0x0806), packets:54, bytes:2268, used:1.337s, actions:3 recirc_id(0),in_port(2),eth(),eth_type(0x86dd),ipv6(frag=no), packets:28, bytes:1684, used:1.430s, flags:S, actions:3 recirc_id(0),in_port(2),eth(),eth_type(0x0800),ipv4(frag=no), packets:15, bytes:610, used:1.270s, flags:S, actions:3 As you can see, for some reason, OVS had split the single relevant OpenFlow rule to three separate megaflows, one for each eth_type (and even other fields - IP fragmentation?). In my production scenario, the packets are even more diversified, and we see OVS installing flows which match on even more fields, including specific Ethernet and IP addresses. This leads to a large number of flows that have extremely low hit rate - each flow handles not more than ~100 packets (!) during its entire lifetime. We suspect that this causes the performance peanalty; either 1) The EMC/megaflow table is full, so vswitchd upcalls are all over the place, or 2) The huge number of inefficient megaflows leads to terrible lookup times in the in-kernel megaflow table itslef (due to large number of masks, etc.) In short: how can I just make OVS oblivious to these fields? Why does it try to match on irrlevant fields? Thanks, Matan On Thu, 19 Dec 2019 at 02:19, Ben Pfaff <[email protected]> wrote: > On Wed, Dec 18, 2019 at 04:26:09PM +0200, Matan Rosenberg wrote: > > Hey, > > I'm using ovs for L2 tunneling (across lots of endpoints and several ovs > > nodes), and experiencing some unexpected performance issues (can't get > more > > that a couple thousand packets/s on rather beefy servers). > > > > I've come across the talk about megaflow issues in OVSFall2018, and I > > suspect that it might be applicable to my situation. > > > > Is there any way to view the megaflows and masks currently used by the > > kernel datapath? > > I can't see how this would be a megaflow problem, but either way: > > ovs-dpctl dump-flows >
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
