Hi Eelco, On Mon, Oct 22, 2018 at 12:30 PM Simon Horman <simon.hor...@netronome.com> wrote: > > Hi Eelco, > > Thanks for your review and sorry for missing it. I trust that Sriharsha will > address it in follow-up patches. > > > 2018/10/19 15:27 Eelco Chaudron <echau...@redhat.com>: >> >> Hi Simon, >> >> You might have missed my general comments email before you committed the >> patchset to master. >> >> Just now I also sent my full review, and it looks like there is one >> nasty memory trashing one in 3/3 which need fixing. It’s the >> x2nrealloc() always allocating 1 entry, but we write to other offsets.
I've responded to your comments on the 3 patches. I've also answered to your above concern on x2nrealloc(); there's no issue. Please see my email on 3/3 for details. Thanks, -Harsha >> >> //Eelco >> >> >> On 19 Oct 2018, at 11:28, Simon Horman wrote: >> >> > On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via >> > dev wrote: >> >> With the current OVS offload design, when an offload-device fails to >> >> add a >> >> flow rule and returns an error, OVS adds the rule to the kernel >> >> datapath. >> >> The flow gets processed by the kernel datapath for the entire life of >> >> that >> >> flow. This is fine when an error is returned by the device due to >> >> lack of >> >> support for certain keys or actions. >> >> >> >> But when an error is returned due to temporary conditions such as >> >> lack of >> >> resources to add a flow rule, the flow continues to be processed by >> >> kernel >> >> even when resources become available later. That is, those flows >> >> never get >> >> offloaded again. This problem becomes more pronounced when a flow >> >> that has >> >> been initially offloaded may have a smaller packet rate than a later >> >> flow >> >> that could not be offloaded due to lack of resources. This leads to >> >> inefficient use of HW resources and wastage of host CPU cycles. >> >> >> >> This patch-set addresses this issue by providing a way to detect >> >> temporary >> >> offload resource constraints (Out-Of-Resource or OOR condition) and >> >> to >> >> selectively and dynamically offload flows with a higher >> >> packets-per-second >> >> (pps) rate. This dynamic rebalancing is done periodically on netdevs >> >> that >> >> are in OOR state until resources become available to offload all >> >> pending >> >> flows. >> >> >> >> The patch-set involves the following changes at a high level: >> >> >> >> 1. Detection of Out-Of-Resources (OOR) condition on an >> >> offload-capable >> >> netdev. >> >> 2. Gathering flow offload selection criteria for all flows on an OOR >> >> netdev; >> >> i.e, packets-per-second (pps) rate of flows for offloaded and >> >> non-offloaded (pending) flows. >> >> 3. Dynamically replacing offloaded flows with a lower pps-rate, with >> >> non-offloaded flows with a higher pps-rate, on an OOR netdev. A >> >> new >> >> OpenvSwitch configuration option - "offload-rebalance" to enable >> >> this policy. >> >> >> >> Cost/benefits data points: >> >> >> >> 1. Rough cost of the new rebalancing, in terms of CPU time: >> >> >> >> Ran a test that replaced 256 low pps-rate flows(pings) with 256 >> >> high >> >> pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @ >> >> 2.40GHz; >> >> 2 cores with hw threads enabled, rest disabled). The data showed >> >> that cpu >> >> utilization increased by about ~20%. This increase occurs during >> >> the >> >> specific second in which rebalancing is done. And subsequently >> >> (from the >> >> next second), cpu utilization decreases significantly due to >> >> offloading >> >> of higher pps-rate flows. So effectively there's a bump in cpu >> >> utilization >> >> at the time of rebalancing, that is more than compensated by >> >> reduced cpu >> >> utilization once the right flows get offloaded. >> >> >> >> 2. Rough benefits to the user in terms of offload performance: >> >> >> >> The benefits to the user is reduced cpu utilization in the host, >> >> since >> >> higher pps-rate flows get offloaded, replacing lower pps-rate >> >> flows. >> >> Replacing a single offloaded flood ping flow with an iperf flow >> >> (multiple >> >> connections), shows that the cpu %used that was originally 100% on >> >> a >> >> single cpu (rebalancing disabled) goes down to 35% (rebalancing >> >> enabled). >> >> That is, cpu utilization decreased 65% after rebalancing. >> >> >> >> 3. Circumstances under which the benefits would show up: >> >> >> >> The rebalancing benefits would show up once offload resources are >> >> exhausted and new flows with higher pps-rate are initiated, that >> >> would >> >> otherwise be handled by kernel datapath costing host cpu cycles. >> >> >> >> This can be observed using 'ovs appctl dpctl/ dump-flows' command. >> >> Prior >> >> to rebalancing, any high pps-rate flows that couldn't be offloaded >> >> due to >> >> resource crunch would show up in the output of 'dump-flows >> >> type=ovs' and >> >> after rebalancing such flows would appear in the output of >> >> 'dump-flows type=offloaded'. >> > >> > Thanks, applied to master. >> > _______________________________________________ >> > dev mailing list >> > d...@openvswitch.org >> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev