At Mon, 1 Feb 2016 01:02:48 +0000, Victor Orlikowski wrote: > > Folks, > > We ran into a fun time debugging an issue w/ one of our switch vendors, and I > wanted to share it, since some of the sample applications make use of the > same functionality.
Thanks for a nice heads up. > The controller application that we are developing (based off of rest_router) > was seeing PacketIn storms related to long-running flows on our hardware > switches. > > The issue was traced down to be a result of this wording regarding OFPFC_ADD > in the OpenFlow specification, which has been there since version 1.0: > > " > If a flow entry with identical header fields and priority already resides in > any table, then that entry, > including its counters, must be removed, and the new flow entry added. > " > > In effect, if a FlowMod is sent that exactly matches the match and priority > of a rule already programmed into the switch, the switch *must* remove the > existing rule, and replace it with the one specified by the received FlowMod, > *unless* the OFPFF_CHECK_OVERLAP flag is set. FWIW, OpenStack Neutron's ovs-agent does a similar thing on restart. (*) Such time windows of service interruption must be implementation dependent and I'm not sure if openvswitch suffers from this too, though. (* - That is with a recent "fix", and it used to be much worse. ;) > The problem with that wording - if you do an OFPFC_ADD to "re-add" a rule > that's already passing traffic, the rule disappears from the flow table for a > brief window of time. > If there are (lower priority than the one that was removed) rules that catch > and forward the packets from that flow to the controller, the high speed flow > will pound the switch with packets that will have to packaged as PacketIn > events - and the switch will subsequently pound the controller. > > We have worked around this behavior in our modification of rest_router by > setting OFPFF_CHECK_OVERLAP on the OFPFC_ADD in set_routing_flow(), and have > removed the only call to _set_route_packetin(). > Things seem to be working - but we will be looking to see if removing > _set_route_packetin() causes any unforeseen problems in practice. > > Nevertheless - this issue is one that will probably need to be addressed, so > that users are not surprised by the sudden bursts of PacketIn events that can > occur. > > The specific thing that caused sudden bursts of PacketIn events in our case > was an ARP cache entry expiring on an end host that was transferring traffic > at 10 Gbps. The end host sent an ARP "who-has" - which the application used > for "mac learning," and resulted in the OFPFC_ADD. The OFPFC_ADD removed the > pre-existing rule, and both the switch and controller got swamped with > PacketIn events. > > Best, > Victor > -- > Victor J. Orlikowski <> vjo@[cs.]duke.edu > > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 > _______________________________________________ > Ryu-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ryu-devel > ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 _______________________________________________ Ryu-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ryu-devel
