Having reference benchmarks is important in order to obtain reproducible performance figures.
This patch describes required steps to configure a PVP setup using testpmd in both host and guest. Not relying on external vSwitch ease integration in a CI loop by not being impacted by DPDK API changes. Signed-off-by: Maxime Coquelin <maxime.coque...@redhat.com> --- Thanks to all the reviewers, this v2 should take all remarks into account. -- Maxime doc/guides/howto/img/pvp_2nics.svg | 556 +++++++++++++++++++++++++++ doc/guides/howto/index.rst | 1 + doc/guides/howto/pvp_reference_benchmark.rst | 395 +++++++++++++++++++ 3 files changed, 952 insertions(+) create mode 100644 doc/guides/howto/img/pvp_2nics.svg create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst diff --git a/doc/guides/howto/img/pvp_2nics.svg b/doc/guides/howto/img/pvp_2nics.svg new file mode 100644 index 0000000..517a800 --- /dev/null +++ b/doc/guides/howto/img/pvp_2nics.svg @@ -0,0 +1,556 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/"; + xmlns:cc="http://creativecommons.org/ns#"; + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"; + xmlns:svg="http://www.w3.org/2000/svg"; + xmlns="http://www.w3.org/2000/svg"; + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"; + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"; + width="127.46428mm" + height="139.41411mm" + viewBox="0 0 451.64508 493.987" + id="svg2" + version="1.1" + inkscape:version="0.92pre2 r" + sodipodi:docname="pvp_2nics.svg" + inkscape:export-filename="/home/max/Pictures/dpdk/pvp/pvp.png" + inkscape:export-xdpi="90" + inkscape:export-ydpi="90"> + <defs + id="defs4"> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker4760" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4762" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="marker4642" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4644" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="marker10370" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path10372" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker10306" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + id="path10308" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9757" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend" + inkscape:collect="always"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + id="path9759" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4224" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path4227" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4227-27" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-9" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4224-3" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9757-0" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + id="path9759-6" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4224-0" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-62" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4227-6" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="marker10370-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path10372-9" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker9757-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow1Lend"> + <path + transform="matrix(-0.8,0,0,-0.8,-10,0)" + style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + id="path9759-0" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-9-2" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4224-3-3" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-1-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path4227-27-5" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="0.49497475" + inkscape:cx="206.7485" + inkscape:cy="227.93958" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:object-nodes="true" + inkscape:window-width="1916" + inkscape:window-height="1040" + inkscape:window-x="0" + inkscape:window-y="38" + inkscape:window-maximized="0" + inkscape:snap-grids="true" + inkscape:snap-to-guides="true" + inkscape:snap-others="false" + inkscape:snap-nodes="false" + inkscape:snap-global="false" /> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage"; /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(-5.3301459,-7.348317)"> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.78969002;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4140" + width="434.38919" + height="75.295639" + x="21.691195" + y="404.59354" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="421.47873" + y="501.3353" + id="text4142"><tspan + sodipodi:role="line" + id="tspan4144" + x="421.47873" + y="501.3353" + style="font-size:25px;line-height:125%">TE</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4146" + width="92.934036" + height="32.324883" + x="182.57764" + y="372.03574" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="183.5878" + y="397.28958" + id="text4148"><tspan + sodipodi:role="line" + id="tspan4150" + x="183.5878" + y="397.28958" + style="font-size:25px">10G NIC</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="166.92024" + y="451.33276" + id="text4152"><tspan + sodipodi:role="line" + id="tspan4154" + x="166.92024" + y="451.33276">Moongen</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.39882457;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4156" + width="449.73071" + height="244.32167" + x="6.0295582" + y="29.046324" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="405.31628" + y="25.048317" + id="text4158"><tspan + sodipodi:role="line" + id="tspan4160" + x="405.31628" + y="25.048317" + style="font-size:25px">DUT</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.14168489;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4162" + width="418.69415" + height="107.50462" + x="19.038134" + y="41.044758" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="395.38812" + y="66.496857" + id="text4164"><tspan + sodipodi:role="line" + id="tspan4166" + x="395.38812" + y="66.496857" + style="font-size:25px">VM</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4146-3" + width="92.934036" + height="32.324883" + x="183.0827" + y="274.05093" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="184.09286" + y="299.30475" + id="text4148-6"><tspan + sodipodi:role="line" + id="tspan4150-7" + x="184.09286" + y="299.30475" + style="font-size:25px">10G NIC</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.4804399;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4189" + width="398.00476" + height="65.451302" + x="26.901583" + y="82.647781" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="30.683046" + y="108.31288" + id="text4191"><tspan + sodipodi:role="line" + id="tspan4193" + x="30.683046" + y="108.31288">TestPMD</tspan><tspan + sodipodi:role="line" + x="30.683046" + y="139.56288" + id="tspan10476">(macswap)</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.49124122;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4189-5" + width="397.22263" + height="66.152573" + x="29.743357" + y="207.6543" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="42.720772" + y="231.14902" + id="text4191-3"><tspan + sodipodi:role="line" + id="tspan4193-5" + x="42.720772" + y="231.14902">TestPMD </tspan><tspan + sodipodi:role="line" + x="42.720772" + y="262.39902" + id="tspan9747">(io)</tspan></text> + <path + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)" + d="M 202.56669,371.44487 V 308.37034" + id="path4218" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart)" + d="M 252.03098,369.63533 V 307.25568" + id="path4218-9" + inkscape:connector-curvature="0" /> + <path + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1)" + d="M 198.63811,207.44389 V 150.47507" + id="path4218-0" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9)" + d="M 255.56859,206.9303 V 147.01008" + id="path4218-9-6" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757)" + d="M 199.50513,271.00921 V 207.3696" + id="path9749" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370)" + d="M 255.56859,270.56991 V 206.9303" + id="path9749-2" + inkscape:connector-curvature="0" /> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4146-36" + width="92.934036" + height="32.324883" + x="304.05591" + y="372.52954" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="305.06607" + y="397.78339" + id="text4148-7"><tspan + sodipodi:role="line" + id="tspan4150-5" + x="305.06607" + y="397.78339" + style="font-size:25px">10G NIC</tspan></text> + <rect + style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect4146-3-3" + width="92.934036" + height="32.324883" + x="306.07623" + y="273.53461" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + x="307.0864" + y="298.78842" + id="text4148-6-5"><tspan + sodipodi:role="line" + id="tspan4150-7-6" + x="307.0864" + y="298.78842" + style="font-size:25px">10G NIC</tspan></text> + <path + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-62)" + d="M 323.7504,370.24835 V 307.17382" + id="path4218-1" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-6)" + d="M 373.21469,368.43881 V 306.05916" + id="path4218-9-8" + inkscape:connector-curvature="0" /> + <path + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1-7)" + d="M 324.93036,207.24894 V 150.28012" + id="path4218-0-9" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9-2)" + d="M 381.86084,206.73535 V 146.81513" + id="path4218-9-6-2" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757-2)" + d="M 325.79738,270.81426 V 207.17465" + id="path9749-28" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370-7)" + d="M 381.86084,270.37496 V 206.73535" + id="path9749-2-9" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4642)" + d="M 198.57143,148.79077 V 95.93363 h 182.85714 v 50" + id="path3748" + inkscape:connector-curvature="0" /> + <path + style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.01005316;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.02010632, 1.01005316;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4760)" + d="m 325.70774,148.78714 v -32.84999 h -70.7012 v 30.70761" + id="path4634" + inkscape:connector-curvature="0" /> + </g> +</svg> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst index 5575b27..712a9f3 100644 --- a/doc/guides/howto/index.rst +++ b/doc/guides/howto/index.rst @@ -38,3 +38,4 @@ HowTo Guides lm_bond_virtio_sriov lm_virtio_vhost_user flow_bifurcation + pvp_reference_benchmark diff --git a/doc/guides/howto/pvp_reference_benchmark.rst b/doc/guides/howto/pvp_reference_benchmark.rst new file mode 100644 index 0000000..c5bbd32 --- /dev/null +++ b/doc/guides/howto/pvp_reference_benchmark.rst @@ -0,0 +1,395 @@ +.. BSD LICENSE + Copyright(c) 2016 Red Hat, Inc. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +PVP reference benchmark setup using testpmd +=========================================== + +This guide lists the steps required to setup a PVP benchmark using testpmd as +a simple forwarder between NICs and Vhost interfaces. The goal of this setup +is to have a reference PVP benchmark without using external vSwitches (OVS, +VPP, ...) to make it easier to obtain reproducible results and to facilitate +continuous integration testing. + +The guide covers two ways of launching the VM, either by directly calling the +QEMU command line, or by relying on libvirt. It has been tested with DPDK +v16.11 using RHEL7 for both host and guest. + + +Setup overview +-------------- + +.. _figure_pvp_2nics: + +.. figure:: img/pvp_2nics.* + + PVP setup using 2 NICs + +In this diagram, each red arrow represents one logical core. This use-case +requires 6 dedicated logical cores. A forwarding configuration with a single +NIC is also possible, requiring 3 logical cores. + + +Host setup +---------- + +In this setup, we isolate 6 cores (from CPU2 to CPU7) on the same NUMA +node. Two cores are assigned to the VM vCPUs running testpmd and four are +assigned to testpmd on the host. + + +Host tuning +~~~~~~~~~~~ + +#. On BIOS, disable turbo-boost and hyper-threads. + +#. Append these options to Kernel command line: + + .. code-block:: console + + intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7 iommu=pt intel_iommu=on + +#. Disable hyper-threads at runtime if necessary or if BIOS is not accessible: + + .. code-block:: console + + cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \ + | sort | uniq \ + | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}' + +#. Disable NMIs: + + .. code-block:: console + + echo 0 > /proc/sys/kernel/nmi_watchdog + +#. Exclude isolated CPUs from the writeback cpumask: + + .. code-block:: console + + echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask + +#. Isolate CPUs from IRQs: + + .. code-block:: console + + clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs + for i in /proc/irq/*/smp_affinity + do + echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i + done + + +Qemu build +~~~~~~~~~~ + +Build Qemu: + + .. code-block:: console + + git clone git://git.qemu.org/qemu.git + cd qemu + mkdir bin + cd bin + ../configure --target-list=x86_64-softmmu + + +DPDK build +~~~~~~~~~~ + +Build DPDK: + + .. code-block:: console + + git clone git://dpdk.org/dpdk + cd dpdk + export RTE_SDK=$PWD + make install T=x86_64-native-linuxapp-gcc DESTDIR=install + + +Testpmd launch +~~~~~~~~~~~~~~ + +#. Assign NICs to DPDK: + + .. code-block:: console + + modprobe vfio-pci + $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1 + + .. Note:: + + The Sandy Bridge family seems to have some IOMMU limitations giving poor + performance results. To achieve good performance on these machines + consider using UIO instead. + +#. Launch the testpmd application: + + .. code-block:: console + + $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \ + --vdev 'net_vhost0,iface=/tmp/vhost-user1' \ + --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \ + --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1 + --nb-cores=4 --forward-mode=io + +With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads. + +#. In testpmd interactive mode, set the portlist to obtain the correct port + chaining: + + .. code-block:: console + + set portlist 0,2,1,3 + start + + +VM launch +~~~~~~~~~ + +The VM may be launched either by calling QEMU directly, or by using libvirt. + +#. Qemu way: + +Launch QEMU with two Virtio-net devices paired to the vhost-user sockets +created by testpmd. Below example uses default Virtio-net options, but options +may be specified, for example to disable mergeable buffers or indirect +descriptors. + + .. code-block:: console + + <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \ + -enable-kvm -cpu host -m 3072 -smp 3 \ + -chardev socket,id=char0,path=/tmp/vhost-user1 \ + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ + -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \ + -chardev socket,id=char1,path=/tmp/vhost-user2 \ + -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \ + -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \ + -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \ + -numa node,memdev=mem -mem-prealloc \ + -net user,hostfwd=tcp::1002$1-:22 -net nic \ + -qmp unix:/tmp/qmp.socket,server,nowait \ + -monitor stdio <vm_image>.qcow2 + +You can use this `qmp-vcpu-pin <https://patchwork.kernel.org/patch/9361617/>`_ +script to pin vCPUs. + +It can be used as follows, for example to pin 3 vCPUs to CPUs 1, 6 and 7, +where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs: + + .. code-block:: console + + export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp + ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7 + +#. Libvirt way: + +Some initial steps are required for libvirt to be able to connect to testpmd's +sockets. + +First, SELinux policy needs to be set to permissive, since testpmd is +generally run as root (note, as reboot is required): + + .. code-block:: console + + cat /etc/selinux/config + + # This file controls the state of SELinux on the system. + # SELINUX= can take one of these three values: + # enforcing - SELinux security policy is enforced. + # permissive - SELinux prints warnings instead of enforcing. + # disabled - No SELinux policy is loaded. + SELINUX=permissive + + # SELINUXTYPE= can take one of three two values: + # targeted - Targeted processes are protected, + # minimum - Modification of targeted policy. + # Only selected processes are protected. + # mls - Multi Level Security protection. + SELINUXTYPE=targeted + + +Also, Qemu needs to be run as root, which has to be specified in +``/etc/libvirt/qemu.conf``: + + .. code-block:: console + + user = "root" + +Once the domain created, the following snippet is an extract of he most +important information (hugepages, vCPU pinning, Virtio PCI devices): + + .. code-block:: xml + + <domain type='kvm'> + <memory unit='KiB'>3145728</memory> + <currentMemory unit='KiB'>3145728</currentMemory> + <memoryBacking> + <hugepages> + <page size='1048576' unit='KiB' nodeset='0'/> + </hugepages> + <locked/> + </memoryBacking> + <vcpu placement='static'>3</vcpu> + <cputune> + <vcpupin vcpu='0' cpuset='1'/> + <vcpupin vcpu='1' cpuset='6'/> + <vcpupin vcpu='2' cpuset='7'/> + <emulatorpin cpuset='0'/> + </cputune> + <numatune> + <memory mode='strict' nodeset='0'/> + </numatune> + <os> + <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> + <boot dev='hd'/> + </os> + <cpu mode='host-passthrough'> + <topology sockets='1' cores='3' threads='1'/> + <numa> + <cell id='0' cpus='0-2' memory='3145728' unit='KiB' memAccess='shared'/> + </numa> + </cpu> + <devices> + <interface type='vhostuser'> + <mac address='56:48:4f:53:54:01'/> + <source type='unix' path='/tmp/vhost-user1' mode='client'/> + <model type='virtio'/> + <driver name='vhost' rx_queue_size='256' /> + <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/> + </interface> + <interface type='vhostuser'> + <mac address='56:48:4f:53:54:02'/> + <source type='unix' path='/tmp/vhost-user2' mode='client'/> + <model type='virtio'/> + <driver name='vhost' rx_queue_size='256' /> + <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/> + </interface> + </devices> + </domain> + + +Guest setup +----------- + + +Guest tuning +~~~~~~~~~~~~ + +#. Append these options to the Kernel command line: + + .. code-block:: console + + default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2 + +#. Disable NMIs: + + .. code-block:: console + + echo 0 > /proc/sys/kernel/nmi_watchdog + +#. Exclude isolated CPU1 and CPU2 from the writeback cpumask: + + .. code-block:: console + + echo 1 > /sys/bus/workqueue/devices/writeback/cpumask + +#. Isolate CPUs from IRQs: + + .. code-block:: console + + clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs + for i in /proc/irq/*/smp_affinity + do + echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i + done + + +DPDK build +~~~~~~~~~~ + +Build DPDK: + + .. code-block:: console + + git clone git://dpdk.org/dpdk + cd dpdk + export RTE_SDK=$PWD + make install T=x86_64-native-linuxapp-gcc DESTDIR=install + + +Testpmd launch +~~~~~~~~~~~~~~ + +Probe vfio module without iommu: + + .. code-block:: console + + modprobe -r vfio_iommu_type1 + modprobe -r vfio + modprobe vfio enable_unsafe_noiommu_mode=1 + cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + modprobe vfio-pci + +Bind the virtio-net devices to DPDK: + + .. code-block:: console + + $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0 0000:00:11.0 + +Start testpmd: + + .. code-block:: console + + $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \ + --proc-type auto --file-prefix pg -- \ + --portmask=3 --forward-mode=macswap --port-topology=chained \ + --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \ + --rxd=256 --txd=256 --nb-cores=2 --auto-start + +Results template +---------------- + +Below template should be used when sharing results: + + .. code-block:: none + + Traffic Generator: <Test equipment (e.g. IXIA, Moongen, ...)> + Acceptable Loss: <n>% + Validation run time: <n>min + Host DPDK version/commit: <version, SHA-1> + Guest DPDK version/commit: <version, SHA-1> + Patches applied: <link to patchwork> + QEMU version/commit: <version> + Virtio features: <features (e.g. mrg_rxbuf='off', leave empty if default)> + CPU: <CPU model>, <CPU frequency> + NIC: <NIC model> + Result: <n> Mpps -- 2.9.3
