Hi Eelco, Thanks for your review and sorry for missing it. I trust that Sriharsha will address it in follow-up patches.
2018/10/19 15:27 Eelco Chaudron <echau...@redhat.com>: > Hi Simon, > > You might have missed my general comments email before you committed the > patchset to master. > > Just now I also sent my full review, and it looks like there is one > nasty memory trashing one in 3/3 which need fixing. It’s the > x2nrealloc() always allocating 1 entry, but we write to other offsets. > > //Eelco > > > On 19 Oct 2018, at 11:28, Simon Horman wrote: > > > On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via > > dev wrote: > >> With the current OVS offload design, when an offload-device fails to > >> add a > >> flow rule and returns an error, OVS adds the rule to the kernel > >> datapath. > >> The flow gets processed by the kernel datapath for the entire life of > >> that > >> flow. This is fine when an error is returned by the device due to > >> lack of > >> support for certain keys or actions. > >> > >> But when an error is returned due to temporary conditions such as > >> lack of > >> resources to add a flow rule, the flow continues to be processed by > >> kernel > >> even when resources become available later. That is, those flows > >> never get > >> offloaded again. This problem becomes more pronounced when a flow > >> that has > >> been initially offloaded may have a smaller packet rate than a later > >> flow > >> that could not be offloaded due to lack of resources. This leads to > >> inefficient use of HW resources and wastage of host CPU cycles. > >> > >> This patch-set addresses this issue by providing a way to detect > >> temporary > >> offload resource constraints (Out-Of-Resource or OOR condition) and > >> to > >> selectively and dynamically offload flows with a higher > >> packets-per-second > >> (pps) rate. This dynamic rebalancing is done periodically on netdevs > >> that > >> are in OOR state until resources become available to offload all > >> pending > >> flows. > >> > >> The patch-set involves the following changes at a high level: > >> > >> 1. Detection of Out-Of-Resources (OOR) condition on an > >> offload-capable > >> netdev. > >> 2. Gathering flow offload selection criteria for all flows on an OOR > >> netdev; > >> i.e, packets-per-second (pps) rate of flows for offloaded and > >> non-offloaded (pending) flows. > >> 3. Dynamically replacing offloaded flows with a lower pps-rate, with > >> non-offloaded flows with a higher pps-rate, on an OOR netdev. A > >> new > >> OpenvSwitch configuration option - "offload-rebalance" to enable > >> this policy. > >> > >> Cost/benefits data points: > >> > >> 1. Rough cost of the new rebalancing, in terms of CPU time: > >> > >> Ran a test that replaced 256 low pps-rate flows(pings) with 256 > >> high > >> pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @ > >> 2.40GHz; > >> 2 cores with hw threads enabled, rest disabled). The data showed > >> that cpu > >> utilization increased by about ~20%. This increase occurs during > >> the > >> specific second in which rebalancing is done. And subsequently > >> (from the > >> next second), cpu utilization decreases significantly due to > >> offloading > >> of higher pps-rate flows. So effectively there's a bump in cpu > >> utilization > >> at the time of rebalancing, that is more than compensated by > >> reduced cpu > >> utilization once the right flows get offloaded. > >> > >> 2. Rough benefits to the user in terms of offload performance: > >> > >> The benefits to the user is reduced cpu utilization in the host, > >> since > >> higher pps-rate flows get offloaded, replacing lower pps-rate > >> flows. > >> Replacing a single offloaded flood ping flow with an iperf flow > >> (multiple > >> connections), shows that the cpu %used that was originally 100% on > >> a > >> single cpu (rebalancing disabled) goes down to 35% (rebalancing > >> enabled). > >> That is, cpu utilization decreased 65% after rebalancing. > >> > >> 3. Circumstances under which the benefits would show up: > >> > >> The rebalancing benefits would show up once offload resources are > >> exhausted and new flows with higher pps-rate are initiated, that > >> would > >> otherwise be handled by kernel datapath costing host cpu cycles. > >> > >> This can be observed using 'ovs appctl dpctl/ dump-flows' command. > >> Prior > >> to rebalancing, any high pps-rate flows that couldn't be offloaded > >> due to > >> resource crunch would show up in the output of 'dump-flows > >> type=ovs' and > >> after rebalancing such flows would appear in the output of > >> 'dump-flows type=offloaded'. > > > > Thanks, applied to master. > > _______________________________________________ > > dev mailing list > > d...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev