On Thu, 2022-07-07 at 13:53 +0200, Eelco Chaudron wrote: > On 2 Jul 2022, at 5:18, Jianbo Liu wrote: > > > Fow now, add introduction and the limitation of meter offload. > > > > Signed-off-by: Jianbo Liu <[email protected]> > > --- > > Documentation/automake.mk | 1 + > > Documentation/howto/index.rst | 1 + > > Documentation/howto/tc-offload.rst | 75 > > ++++++++++++++++++++++++++++++ > > 3 files changed, 77 insertions(+) > > create mode 100644 Documentation/howto/tc-offload.rst > > > > diff --git a/Documentation/automake.mk b/Documentation/automake.mk > > index 6c2c57739..185349c14 100644 > > --- a/Documentation/automake.mk > > +++ b/Documentation/automake.mk > > @@ -79,6 +79,7 @@ DOC_SOURCE = \ > > Documentation/howto/vlan.png \ > > Documentation/howto/vlan.rst \ > > Documentation/howto/vtep.rst \ > > + Documentation/howto/tc-offload.rst \ > > Documentation/ref/index.rst \ > > Documentation/faq/index.rst \ > > Documentation/faq/configuration.rst \ > > diff --git a/Documentation/howto/index.rst > > b/Documentation/howto/index.rst > > index 60fb8a717..1812f6a11 100644 > > --- a/Documentation/howto/index.rst > > +++ b/Documentation/howto/index.rst > > @@ -49,4 +49,5 @@ OVS > > vtep > > sflow > > dpdk > > + tc-offload > > > > diff --git a/Documentation/howto/tc-offload.rst > > b/Documentation/howto/tc-offload.rst > > new file mode 100644 > > index 000000000..09a136264 > > --- /dev/null > > +++ b/Documentation/howto/tc-offload.rst > > @@ -0,0 +1,75 @@ > > +.. > > + Licensed under the Apache License, Version 2.0 (the > > "License"); you may > > + not use this file except in compliance with the License. You > > may obtain > > + a copy of the License at > > + > > + http://www.apache.org/licenses/LICENSE-2.0 > > + > > + Unless required by applicable law or agreed to in writing, > > software > > + distributed under the License is distributed on an "AS IS" > > BASIS, WITHOUT > > + WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > implied. See the > > + License for the specific language governing permissions and > > limitations > > + under the License. > > + > > + Convention for heading levels in Open vSwitch documentation: > > + > > + ======= Heading 0 (reserved for the title in a document) > > + ------- Heading 1 > > + ~~~~~~~ Heading 2 > > + +++++++ Heading 3 > > + ''''''' Heading 4 > > + > > + Avoid deeper levels because they do not render well. > > + > > +===================================== > > +Hardware offload with Linux TC flower > > +===================================== > > + > > +This document describes how to offload with TC flower. It is > > disabled by > > +default and can be enabled by:: > > + > > > How about re-arranging the layout a bit. For example: > > ========================================== > Flow Hardware offload with Linux TC flower > ========================================== > > This document describes how to offload flows with TC flower. > > Flow Hardware Offload > --------------------- > > The flow hardware offload is disabled by default and can be enabled > by:: > > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true > > TC flower has one additional configuration option caled ``tc- > policy``. > For more details see ``man ovs-vswitchd.conf.db``. > > > + $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true > > + > > +Offload meter to TC > > +------------------- > > + > > +After hardware offload is enabled, meter offload is enabled as > > well. > > + > > What about: > > TC Meter Offload > ---------------- > > Offloading meters to TC does not require any additional configuration > and is > enabled automatically when possible. Offloading with meters does > require the > tc-police action to be available in the Linux kernel. For more > details on the > tc-police action, see ``man tc-police``. > > > > +Configuration > > +~~~~~~~~~~~~~ > > + > > +There is no parameter change in ovs-ofctl command, to configue a > > meter and use > > +it for a flow in the offload way. Usually the commands are like:: > > + > > + $ ovs-ofctl -O OpenFlow13 add-meter br0 "meter=1 pktps > > bands=type=drop rate=1" > > + $ ovs-ofctl -O OpenFlow13 add-flow br0 > > "priority=10,in_port=ovs-p0,udp actions=meter:1,normal" > > + > > + > + For more details, see ``man ovs-ofctl``. > > > > +.. note:: > > + Each meter is mapped to one TC police action. To avovid the > > conflicton, the > > + police action index of 0x10000000-0x1fffffff are resevered for > > the mapping. > > + You can check the police actions by the command "tc action ls > > action police" > > + in Linux system. > > ``tc action ls action police`` > > > + > > +Dump meter statistics > > +--------------------- > > + > > +You can use ovs-ofctl to display the statisticsof meter:: > > + > > + $ ovs-ofctl -O OpenFlow13 meter-stats br0 > > + OFPST_METER reply (OF1.3) (xid=0x2): > > + meter:1 flow_count:1 packet_in_count:11 byte_in_count:377 > > duration:3.199s bands: > > + 0: packet_count:9 byte_count:0 > > + > > +The stats of the dropped packets is stored to band 0. > > + > > +.. note:: > > + Meter in OVS DP is not synced with the mapped poloce action, and > > the first > > + packet for DP learning does not pass TC police action, as the > > rule is not > > + offloaded yet. So, you may notice that packet_count of band 0 > > (for dropped > > + packets) is counted after the rule is offoalded, while > > packet_in_count is > > + summed with the numbers of packets of both DP and TC police > > action. > > + > > +.. note:: > > + Linux TC does not support the dropped bytes count, so the > > byte_count of band > > + 0 is always 0 if meter is offloaded. > > + > > I would remove the Dump meter statistics section and replace it with > a known limitations section, i.e.: > > Known TC flow offload limitations > --------------------------------- > > General > ~~~~~~~ > > These sections describe limitations to the general TC flow offload > implementation. > > Flow bytes count > ++++++++++++++++ > > Flows that are offloaded with TC do not include the L2 bytes in the > packet > byte count. Take the datapath flow dump below as an example. The > first one > is from the none-offloaded case the second one is from a TC offloaded > flow:: > > in_port(2),eth(macs),eth_type(0x0800),ipv4(proto=17,frag=no), > packets:10, bytes:470, used:0.001s, actions:outputmeter(0),3 > > in_port(2),eth(macs),eth_type(0x0800),ipv4(proto=17,frag=no), > packets:10, bytes:330, used:0.001s, actions:outputmeter(0),3 > > As you can see above the none-offload case reports 140 bytes more, > which is 14 > bytes per packet. This represents the L2 header, in this case, 2 * > *Ethernet > address* + *Ethertype*. > > TC Meter Offload > ~~~~~~~~~~~~~~~~ > > These sections describe limitations related to the TC meter offload > implementation. > > Missing byte count drop statistics > ++++++++++++++++++++++++++++++++++ > > The kernel's TC infrastructure is only counting the number of dropped > packet, > not their byte size. This results in the meter statistics always > showing 0 > for byte_count. Here is an example:: > > $ ovs-ofctl -O OpenFlow13 meter-stats br0 > OFPST_METER reply (OF1.3) (xid=0x2): > meter:1 flow_count:1 packet_in_count:11 byte_in_count:377 > duration:3.199s bands: > 0: packet_count:9 byte_count:0 > > First flow packet not processed by meter > ++++++++++++++++++++++++++++++++++++++++ > > Packets that are received by ovs-vswitchd through an upcall before > the actual > meter flow is installed, are not passing TC police action and > therefore are > not considered for policing. > > > > Just to be compete here is a single diff with all my changes on top > of you patch:
I completely agree with your changes. It help me a lot. Thanks you very much! > > > diff --git a/Documentation/howto/tc-offload.rst > b/Documentation/howto/tc-offload.rst > index 09a136264..3105c8d21 100644 > --- a/Documentation/howto/tc-offload.rst > +++ b/Documentation/howto/tc-offload.rst > @@ -21,19 +21,30 @@ > > Avoid deeper levels because they do not render well. > > -===================================== > -Hardware offload with Linux TC flower > -===================================== > +========================================== > +Flow Hardware offload with Linux TC flower > +========================================== > > -This document describes how to offload with TC flower. It is > disabled by > -default and can be enabled by:: > +This document describes how to offload flows with TC flower. > > - $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true > +Flow Hardware Offload > +--------------------- > + > +The flow hardware offload is disabled by default and can be enabled > by:: > + > + $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true > + > +TC flower has one additional configuration option caled ``tc- > policy``. > +For more details see ``man ovs-vswitchd.conf.db``. > + > +TC Meter Offload > +---------------- > > -Offload meter to TC > -------------------- > +Offloading meters to TC does not require any additional > configuration and is > +enabled automatically when possible. Offloading with meters does > require the > +tc-police action to be available in the Linux kernel. For more > details on the > +tc-police action, see ``man tc-police``. > > -After hardware offload is enabled, meter offload is enabled as well. > > Configuration > ~~~~~~~~~~~~~ > @@ -41,35 +52,63 @@ Configuration > There is no parameter change in ovs-ofctl command, to configue a > meter and use > it for a flow in the offload way. Usually the commands are like:: > > - $ ovs-ofctl -O OpenFlow13 add-meter br0 "meter=1 pktps > bands=type=drop rate=1" > - $ ovs-ofctl -O OpenFlow13 add-flow br0 > "priority=10,in_port=ovs-p0,udp actions=meter:1,normal" > + $ ovs-ofctl -O OpenFlow13 add-meter br0 "meter=1 pktps > bands=type=drop rate=1" > + $ ovs-ofctl -O OpenFlow13 add-flow br0 "priority=10,in_port=ovs- > p0,udp actions=meter:1,normal" > + > +For more details, see ``man ovs-ofctl``. > > .. note:: > Each meter is mapped to one TC police action. To avovid the > conflicton, the > police action index of 0x10000000-0x1fffffff are resevered for the > mapping. > - You can check the police actions by the command "tc action ls > action police" > + You can check the police actions by the command ``tc action ls > action police`` > in Linux system. > > -Dump meter statistics > ---------------------- > > -You can use ovs-ofctl to display the statisticsof meter:: > +Known TC flow offload limitations > +--------------------------------- > > - $ ovs-ofctl -O OpenFlow13 meter-stats br0 > - OFPST_METER reply (OF1.3) (xid=0x2): > - meter:1 flow_count:1 packet_in_count:11 byte_in_count:377 > duration:3.199s bands: > - 0: packet_count:9 byte_count:0 > +General > +~~~~~~~ > > -The stats of the dropped packets is stored to band 0. > +These sections describe limitations to the general TC flow offload > +implementation. > > -.. note:: > - Meter in OVS DP is not synced with the mapped poloce action, and > the first > - packet for DP learning does not pass TC police action, as the rule > is not > - offloaded yet. So, you may notice that packet_count of band 0 (for > dropped > - packets) is counted after the rule is offoalded, while > packet_in_count is > - summed with the numbers of packets of both DP and TC police > action. > +Flow bytes count > +++++++++++++++++ > > -.. note:: > - Linux TC does not support the dropped bytes count, so the > byte_count of band > - 0 is always 0 if meter is offloaded. > +Flows that are offloaded with TC do not include the L2 bytes in the > packet > +byte count. Take the datapath flow dump below as an example. The > first one > +is from the none-offloaded case the second one is from a TC > offloaded flow:: > + > + in_port(2),eth(macs),eth_type(0x0800),ipv4(proto=17,frag=no), > packets:10, bytes:470, used:0.001s, actions:outputmeter(0),3 > + > + in_port(2),eth(macs),eth_type(0x0800),ipv4(proto=17,frag=no), > packets:10, bytes:330, used:0.001s, actions:outputmeter(0),3 > + > +As you can see above the none-offload case reports 140 bytes more, > which is 14 > +bytes per packet. This represents the L2 header, in this case, 2 * > *Ethernet > +address* + *Ethertype*. > + > +TC Meter Offload > +~~~~~~~~~~~~~~~~ > + > +These sections describe limitations related to the TC meter offload > +implementation. > + > +Missing byte count drop statistics > +++++++++++++++++++++++++++++++++++ > + > +The kernel's TC infrastructure is only counting the number of > dropped packet, > +not their byte size. This results in the meter statistics always > showing 0 > +for byte_count. Here is an example:: > + > + $ ovs-ofctl -O OpenFlow13 meter-stats br0 > + OFPST_METER reply (OF1.3) (xid=0x2): > + meter:1 flow_count:1 packet_in_count:11 byte_in_count:377 > duration:3.199s bands: > + 0: packet_count:9 byte_count:0 > + > +First flow packet not processed by meter > +++++++++++++++++++++++++++++++++++++++++ > > +Packets that are received by ovs-vswitchd through an upcall before > the actual > +meter flow is installed, are not passing TC police action and > therefore are > +not considered for policing. > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
