Hi Yujun, Inline…
From: "Yujun Zhang (ZTE)" <zhangyujun+...@gmail.com> Date: Wednesday, April 12, 2017 at 5:56 PM To: "Frank Brockners (fbrockne)" <fbroc...@cisco.com>, TSC OPNFV <opnfv-...@lists.opnfv.org>, TECH-DISCUSS OPNFV <opnfv-tech-discuss@lists.opnfv.org>, test-wg <test...@lists.opnfv.org> Cc: Carsten Rossenhoevel <cr...@eantc.de>, "Alec Hothan (ahothan)" <ahot...@cisco.com> Subject: Re: [opnfv-tech-discuss] New project proposal: NFVbench Some question after reviewing the proposal. · Technically, is data plane / L2/L3 forwarding the only performance indicator for NFV? Generally speaking, L2 forwarding is the main performance indicator for an NFVi data plane: measure how many frame/sec can the data plane handle under a well defined set of constraints. The set of constraints is actually a very important part of the picture because it can cloud the picture if not defined properly. What NFVbench brings is an automation of the well defined set of constraints so that these benchmarks can be repeated by more than 1 party and so that 2 different implementations can be compared more easily. The description mentions L2/L3 forwarding performance, the L3 part is simply because all frames are fully formed IPv4 L3 frames (even if the L3 header is not usually used by the NFVi data plane) and as such - this will also allow simple L3 forwarding benchmarking of any service chain. There is no plan to cover full blown VNF performance benchmarking in this project. To answer your question to the letter, L2/L3 forwarding performance is not the only performance indicator for NFV (but is an important one). · Will NFVbench project expand to other performance metrics besides the main focus? The simple answer is no. Many open source projects suffer from over-diversification or over-scoping (want to do too many things). Measuring the NFVi data plane properly is already a pretty difficult task that requires experience, focus and discipline and this already represents a lot more work to cover than we have resources. · In what way will NFVbench plan to complement and leverage related projects? I will reply to the positioning with other OPNFV projects in a separate email. · What will be the main components of the toolkit? A test runner? Test cases? Drivers? The toolkit will be a compact specialized test runner that can run standalone but can also be easily integrated into a more generic test harness (such as Jenkins, yardstic or qtip) using well defined APIs/interfaces. Regarding test cases, we have decided to follow a different approach with the use of pre-built packet paths. Many benchmarking tools provide the ability for the user to fully customize the benchmark parameters (as in writing python code). With NFVbench we want to limit that ability to developers of the tool itself for the following reasons: · The crafting of network benchmarking conditions is best done by experts in networking, these are people who are the best positioned to know what makes sense to test and in what conditions · having too many people or worst - users of the tool - modify the conditions of the benchmark will have a huge negative impact on the usability of the results generated (in that – it will be a lot more difficult to compare them since the conditions have been modified) · and finally, the vast majority of targeted users of NFVbench will be non dataplane experts that have no intent to modify the code or to add more variations to the benchmarks than necessary So, NFVbench will define what the benchmark is and how it is conducted, it will be open source so everyone can discuss publicly on benchmarking conditions and see how the benchmark is conducted and automated. One important note is that there is no such a thing as a perfect benchmark tool, but it is possible to get one that is good enough for the community. Today, it is impossible to compare 2 full stack data plane benchmarks because the conditions they are conducted are generally very different and often too subtle for many people to make sense of. These benchmarks are also often very difficult to reproduce (not mentioning by third party).This is what NFVbench is planning to address. Thanks Alec The following text are quoted for quick reference The NFVbench project develops a toolkit that allows developers, integrators, testers and customers to measure and assess the L2/L3 forwarding performance of an NFV-infrastructure solution stack (i.e. OPNFV scenario) using a black-box approach. ... The main focus of NFVbench is the NFVI full stack data plane benchmarking using realistic production deployment conditions. NFVbench does not focus on the following areas and will align with, complement and leverage projects that already cover properly these areas: ... On Thu, Apr 13, 2017 at 12:40 AM Frank Brockners (fbrockne) <fbroc...@cisco.com<mailto:fbroc...@cisco.com>> wrote: Hi OPNFV, over the past few weeks we’ve distilled a proposals to create a toolkit to allow for black-box performance testing of NFVI with a network focus: NFVbench: https://wiki.opnfv.org/display/nfvbench/NFVbench+Project+Proposal The NFVbench project is to develop a toolkit that allows developers, integrators, testers and customers to measure and assess the L2/L3 forwarding performance of an NFV-infrastructure solution stack (i.e. OPNFV scenario) using a black-box approach. We’re hoping for a discussion in the technical community meeting on April/20, and are also asking for an official TSC review post the technical community review on May/2, so that NFVbench can participate in Euphrates. Consequently, NFVbench asks for tentative inclusion into Euphrates. Your thoughts and ideas are greatly appreciated. Thanks much, Frank, Carsten, Alec _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss -- Yujun Zhang
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
