I think it would be helpful if we can evolve our test ecosystem diagram to show a distinction between “developer” type test tools and “user” type test tools … something to discuss in the Test Working Group.
/Trevor From: test-wg-boun...@lists.opnfv.org [mailto:test-wg-boun...@lists.opnfv.org] On Behalf Of morgan.richo...@orange.com Sent: Wednesday, October 11, 2017 2:49 AM To: Alec Hothan (ahothan) <ahot...@cisco.com>; Frank Brockners (fbrockne) <fbroc...@cisco.com>; Sofia Wallin <sofia.wal...@ericsson.com> Cc: OLLIVIER Cédric IMT/OLN <cedric.olliv...@orange.com>; 'Brandon Wick' <bw...@linuxfoundation.org>; Jose Lausuch <jalaus...@suse.com>; test...@lists.opnfv.org; opnfv-tech-discuss@lists.opnfv.org Subject: Re: [test-wg] Releasing OPNFV tools in Euphrates Agree to extend to mailing list :) just to summarize, we started the discussion on doc for calipso for Euphrates, it triggered discussions more general topics including - a topic on how to include NFVBench in Euphrates (Operation/testing) - a discussion on OPNFV release / tooling release decorrelation we can add a slot for this during the next weekly meeting https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting (maybe a bit late regarding the release date) I was not able to attend the APAC slot this morning, according to IRC, I was not the only one... /Morgan On 10/10/2017 21:02, Alec Hothan (ahothan) wrote: I wish this discussion was brought up on the tech-discuss mailing list ;-) I had exactly the same questions than Frank at the very bottom and asked similar questions to David this morning on the release packaging. Heather is also right on target to feel like tools should be independent of the platform release! This is clearly one of the benefits of having an independent versioning for projects vs release which I have been trying to push with the release tagging discussion. For me there is absolutely no question that testing tools developed in OPNFV will be used outside of OPNFV release cycle and potentially on NFVi platforms that are not even in OPNFV labs and this clearly mandates an independent versioning for each tool project. Trying to version tools in lock step to OPNFV releases was reasonable for the early releases but we saw first hand how it became more and more difficult to keep this going as we get more tools/projects and more dependencies between projects – and now more users of these projects. As a matter of fact, functest is the first to break ranks with the way they are planning to manage their version tags for euphrates. I feel like functest will not even really use the official euphrates image (the one tagged with “5.0.0” or now “opnfv-5.0.0”) because it is just too static/slow-paced. Instead functest will use a “stable” or “euphrates” tagged image (from discussion with Cedric and Morgan). Functest is used outside of OPNFV labs so clearly they need it versioned properly outside of the OPNFV release cycle. I’m in the same situation with NFVbench, which is already used outside of OPNFV labs. Having said that, the sky is not falling on us because these are actually good problems to have and shows industry adoption of OPNFV projects. But we need to adjust the way projects are versioned, from a best effort ad-hoc basis to a more focused/streamlined one. More inline on NFVbench… From: "morgan.richo...@orange.com"<mailto:morgan.richo...@orange.com> <morgan.richo...@orange.com><mailto:morgan.richo...@orange.com> Date: Tuesday, October 10, 2017 at 9:25 AM To: "Frank Brockners (fbrockne)" <fbroc...@cisco.com><mailto:fbroc...@cisco.com>, Sofia Wallin <sofia.wal...@ericsson.com><mailto:sofia.wal...@ericsson.com> Cc: Fatih Degirmenci <fatih.degirme...@ericsson.com><mailto:fatih.degirme...@ericsson.com>, Jose Lausuch <jalaus...@suse.com><mailto:jalaus...@suse.com>, "Alec Hothan (ahothan)" <ahot...@cisco.com><mailto:ahot...@cisco.com>, 'Brandon Wick' <bw...@linuxfoundation.org><mailto:bw...@linuxfoundation.org> Subject: Re: Releasing OPNFV tools in Euphrates design is done by LF design team they create icons for the projects and create the global image the testing catalog page provides interactivity on the image to display the different test cases it is based on the cases descriptions created in the test databases http://testresults.opnfv.org/test/api/v1/projects/yardstick/cases http://testresults.opnfv.org/test/api/v1/projects/vsperf/cases changing the image => change the processing for the roll over, onclick, ... in a first step, regarding the time frame, we could 1) change the figure in the doc (static) => need LF update @Brandon possible to get visual for NFVBench and include it on the main picture? as far as I understand it is not connected to CI, Test API, ..so it should be somewhere in the Performance Testing box but without connection with Test API or Yardstick [Alec] I did read emails sent by Morgan regarding that testing landing page but was not sure how to position NFVbench. NFVbench also covers the operational side because it is designed from the get go to run in production. Today there is no solution for doing data plane performance measurement and monitoring on a production cloud because it is just too costly and too complex to do with traditional benchmarking methods (HW traffic generators in every cloud is just not reasonable). NFVbench fills that gap for practically no cost (you just need a NIC card - a tiny fraction of the cost of a pod). So yes it can be used as a testing tool (to test and measure data plane performance) but is also and foremost a tool for production. To make it simpler in euphrates we can just put it in the testing group for now. To make it even simpler, perhaps we should just add “NFVbench” next to vsperf in the diagram since they both tackle the data plane performance? No need to add a new logo, we can worry about a better arrangement for Fraser 2) reference NFVBench in the testing wiki page https://wiki.opnfv.org/display/testing [Alec] see suggestion above. 3) add a chapter on NFVbench in testing ecosystem documentation => patch https://git.opnfv.org/opnfvdocs/tree/docs/testing/ecosystem/overview.rst -- remove CPerf (I planned ot do it) -- add NFVBench section in the table after Functest (alphabetical order) [Alec] I can take care of this, will send out a gerrit review. -- reference your documentation (NFVBench User guide and NFVBench Configuration guide) if any under https://git.opnfv.org/opnfvdocs/tree/docs/testing/testing-user.rst [Alec] there is already a gerrit review for this (waiting for Sofia’s approval). https://gerrit.opnfv.org/gerrit/#/c/43521/ 4) keep current figure for testing catalog as NFVBench did not declare any case in the database and is not running in CI as far as I understand [Alec] fine with that, CI/XCI/CD for NFVbench requires some standardization in pod data plane wiring and can come in Fraser. As of now, I am testing NFVbench on my own openstack pods to make sure it is working. Thanks, Alec On 10/10/2017 18:04, Frank Brockners (fbrockne) wrote: Hi Morgan, NFVbench for now is stand-alone. Intention is to offer a “one button to hit” full stack qualification tool. Cc’ing Alec. So I’d agree that we should add it to the testing-tools world map picture. Do you have the source for that picture? Thanks, Frank From: morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> [mailto:morgan.richo...@orange.com] Sent: Dienstag, 10. Oktober 2017 17:58 To: Frank Brockners (fbrockne) <fbroc...@cisco.com><mailto:fbroc...@cisco.com>; Sofia Wallin <sofia.wal...@ericsson.com><mailto:sofia.wal...@ericsson.com> Cc: Fatih Degirmenci <fatih.degirme...@ericsson.com><mailto:fatih.degirme...@ericsson.com>; Jose Lausuch <jalaus...@suse.com><mailto:jalaus...@suse.com> Subject: Re: Releasing OPNFV tools in Euphrates +1 for operations BTW Frank, is NFV bench integrated in yardstick or independent if it is independent, it would have made sense to include it in the Testing overview picture... http://docs.opnfv.org/en/latest/testing/ecosystem/overview.html#the-opnfv-testing-ecosystem we could remove Cperf and add NFBench instead, but it means that the picture should also be updated and the testing catalog as well http://testresults.opnfv.org/testing/#!/select/visual<http://testresults.opnfv.org/testing/#%21/select/visual> I send the question several times on the mailing list/during testing meeting if any new project would like to be integrated but did not get any feedback Did I miss something? /Morgan On 10/10/2017 17:39, Frank Brockners (fbrockne) wrote: Hi Sofia, good idea – roping in Faith, Morgan, Jose. Folks, any thoughts on how to evolve the current docs structure. Key need is to evolve the structure so that we make a home for e.g. the Calipso documentation is which an operational tool. Thanks, Frank From: Sofia Wallin [mailto:sofia.wal...@ericsson.com] Sent: Dienstag, 10. Oktober 2017 17:37 To: Frank Brockners (fbrockne) <fbroc...@cisco.com><mailto:fbroc...@cisco.com> Subject: Re: Releasing OPNFV tools in Euphrates Hi Frank, Since I know that both the Test WG and Infra WG have done and are working on structuring and moving their documentation to docs.opnfv.org I think I would be fair to invite someone from each side. We have definitely come to the point where we need to review the bigger picture but the urgent question right now is, how much can we do for Euphrates. CI related documentation in added under ‘Infrastructure’. And I don’t know if it makes the most sense to add/change the Test Frameworks to OPNFV tools or add some kind of Operations section under Infrastructure. Or maybe Operations would be a new, stand-alone section. Maybe we can have a discussion with Fatih and Morgan? //Sofia From: Frank Brockners <fbroc...@cisco.com<mailto:fbroc...@cisco.com>> Date: Tuesday, 10 October 2017 at 14:22 To: Sofia Wallin <sofia.wal...@ericsson.com<mailto:sofia.wal...@ericsson.com>> Subject: RE: Releasing OPNFV tools in Euphrates Hi Sofia, thanks. We could include NFVbench under “test tools” indeed. Calispo is an operations support tool. It dynamically discovers and shows you the network topology of the system across all different layers (see e.g. calipso.io). Equally well, we need a home for XCI. How about we rename the “Test Frameworks” / “Testing user guides” page to “OPNFV Tools” – that way we can have all the different tools – for testing (functional, performance, compliance – i.e. we’d include Dovetail’s tools), operations, CI in one big bucket. That might be the lowest hanging fruit for Euphrates. We can create a more granular structure in the future. Thoughts? Thanks, Frank From: Sofia Wallin [mailto:sofia.wal...@ericsson.com] Sent: Dienstag, 10. Oktober 2017 14:02 To: Frank Brockners (fbrockne) <fbroc...@cisco.com<mailto:fbroc...@cisco.com>> Subject: Re: Releasing OPNFV tools in Euphrates Hi Frank, Ok, we will of course make necessary changes but just to sort a few things out. Currently we have ‘Release’, ‘Test Framework’, ‘Development’ and ‘Infrastructure’ on our landing page, http://docs.opnfv.org/en/latest/index.html And my understand is that NFVBench is a test tool? Which means that it should be included under ‘test framework’? And for the Calipso project, who is the targeted user or audience for this tool? Thanks, Sofia From: Frank Brockners <fbroc...@cisco.com<mailto:fbroc...@cisco.com>> Date: Tuesday, 10 October 2017 at 13:26 To: Sofia Wallin <sofia.wal...@ericsson.com<mailto:sofia.wal...@ericsson.com>> Subject: FW: Releasing OPNFV tools in Euphrates Hi Sofia, that is the latest, that I saw. I was assuming that Doc’s would reflect the structure for the new releases page. IMHO we could name the “other initiatives page” that Brandon mentions below “Tools”. In that we could refer to (at least) · Calipso: https://git.opnfv.org/calipso/tree/docs · NFVbench: https://git.opnfv.org/nfvbench/tree/docs · XCI: https://git.opnfv.org/releng-xci/tree/docs Could this be a possible starting point for you to implement things? Once we had a draft, we could ask for wider review. Thanks, Frank From: Brandon Wick [mailto:bw...@linuxfoundation.org] Sent: Mittwoch, 4. Oktober 2017 03:47 To: Heather Kirksey <hkirk...@linuxfoundation.org<mailto:hkirk...@linuxfoundation.org>> Cc: Fatih Degirmenci <fatih.degirme...@ericsson.com<mailto:fatih.degirme...@ericsson.com>>; Tim Irnich <tim.irn...@ericsson.com<mailto:tim.irn...@ericsson.com>>; Sofia Wallin <sofia.wal...@ericsson.com<mailto:sofia.wal...@ericsson.com>>; Frank Brockners (fbrockne) <fbroc...@cisco.com<mailto:fbroc...@cisco.com>>; Raymond Paik <rp...@linuxfoundation.org<mailto:rp...@linuxfoundation.org>>; David McBride <dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>> Subject: Re: Releasing OPNFV tools in Euphrates Thanks all, Appreciate the discussion as our team has been wondering how we can better package up and promote these important elements of OPNFV. Specifically in regard to the software downloads page for Euphrates (https://www.opnfv.org/software/downloads), we're asking the web team to come up with a mock up of a new version that does the following: · Highlights the link to Release Documentation more prominently · Changes format for "Installers" Header (it looks too much like a button) · Adds a Column for Daisy · Adds a section for OPNFV Testing where we link to the Test Project User Guides · Add a section where we can feature another initiative (like XCI, Calipso, etc) I'll share this with y'all when I'm back on 10/16 and we'll rev from there. Best, Brandon Wick OPNFV Head of Marketing, The Linux Foundation Mobile: +1.917.282.0960 Skype: wick.brandon Email / Google Talk: bw...@linuxfoundation.org<mailto:bw...@linuxfoundation.org> On Thu, Sep 28, 2017 at 10:30 AM, Heather Kirksey <hkirk...@linuxfoundation.org<mailto:hkirk...@linuxfoundation.org>> wrote: Frank, It's as if you were reading my mind (and of course expanding on it). I've been having a nascent idea that given that we're getting quite a nice set of test tools over all w/ all the performance and benchmarking ones + Yardstick and Functest, that we might want to start publishing all of them in a more obvious and consumable fashion (and perhaps even decouple their release from the "platform" releases). I think that set of test tools is in many ways our real "product" and that we might want to highlight that more. Some of this is probably more forward-looking that Euphrates specifically (since that is so close) but I'd really like to see our entire suite of testing tools packaged and documented in a way that makes it very obvious how to use them in a way that's separate from the scenario/installer builds. I am also thinking we might want a deep dive white paper on testing to highlight these capabilities. In terms of what we need to do specifically (perhaps of a minimum viable variety), let's work with Brandon and David and Ray to make sure we've got what we need to arm the web team when they're generating the artifacts page. It seems as though we've got high level needs identified. Heather On Thu, Sep 28, 2017 at 6:22 AM, Fatih Degirmenci <fatih.degirme...@ericsson.com<mailto:fatih.degirme...@ericsson.com>> wrote: Hi, I agree to Frank's points below. We only display installers on release page and it is important to display others in a similar structure as the docs.opnfv.org<http://docs.opnfv.org> on release page; Testing, Operations, Infrastructure perhaps. We will have docs.opnfv.org<http://docs.opnfv.org> obviously but first thing potential users see is the release page and it is important to highlight what else we are doing so users can directly go to documentation or whatever released as part of the release. /Fatih From: Sofia Wallin <sofia.wal...@ericsson.com<mailto:sofia.wal...@ericsson.com>> Date: Thursday, 28 September 2017 at 15:13 To: Frank Brockners <fbroc...@cisco.com<mailto:fbroc...@cisco.com>>, Raymond Paik <rp...@linuxfoundation.org<mailto:rp...@linuxfoundation.org>>, Fatih Degirmenci <fatih.degirme...@ericsson.com<mailto:fatih.degirme...@ericsson.com>>, David McBride <dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>> Cc: Heather Kirksey <hkirk...@linuxfoundation.org<mailto:hkirk...@linuxfoundation.org>>, Brandon Wick <bw...@linuxfoundation.org<mailto:bw...@linuxfoundation.org>> Subject: Re: Releasing OPNFV tools in Euphrates Hi everyone, From a documentation point of view, we have already prepared an “Infrastructure” section where XCI related stuff will be published. Work is ongoing to move material from the wiki to docs.opnfv.org<http://docs.opnfv.org>. I’m not fully up to date with what the Calipso project does, but if there is need to adjust the structure on the landing page for docs I’m happy to have a discussion. //Sofia From: Frank Brockners <fbroc...@cisco.com<mailto:fbroc...@cisco.com>> Date: Thursday, 28 September 2017 at 11:47 To: Raymond Paik <rp...@linuxfoundation.org<mailto:rp...@linuxfoundation.org>>, Fatih Degirmenci <fatih.degirme...@ericsson.com<mailto:fatih.degirme...@ericsson.com>>, "dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>" <dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>> Cc: Sofia Wallin <sofia.wal...@ericsson.com<mailto:sofia.wal...@ericsson.com>>, Heather Kirksey <hkirk...@linuxfoundation.org<mailto:hkirk...@linuxfoundation.org>>, Brandon Wick <bw...@linuxfoundation.org<mailto:bw...@linuxfoundation.org>> Subject: Releasing OPNFV tools in Euphrates Hi folks, just had a quick discussion over IRC with Fatih on the question of how we release artifacts in Euphrates which are not packaged with an installer, examples are · Testing tools (like Functest, Yardstick, NFVbench) · Operational tools (like Calipso) · XCI tooling Right now the download page only shows the different installers. If someone would want to run e.g. FuncTest as a standalone tool (which several customers I know of do), you’ll need to do a bit of digging. With Euphrates, things get even more diversified from a tooling perspective (see above), hence we should evolve how we present *all* our release artifacts to the world, as opposed to limit the view to installers/scenarios. The main impact would be on the software downloads page https://www.opnfv.org/software/downloads - and documentation http://docs.opnfv.org/ (cc’ing Sophia as well). IMHO we’d need dedicated sections on both, the releases page as well as the documentation page for the categories listed above (for testing we obviously already have a dedicated documentation section). Thoughts? Thanks, Frank -- Heather Kirksey Director, OPNFV Mobile: +1.512.917.7938<tel:%28512%29%20917-7938> Email/Google Talk: hkirk...@linuxfoundation.org<mailto:hkirk...@linuxfoundation.org> Skype: HeatherReneeKirksey IRC: HKirksey [FV_RGB.png] -- Morgan Richomme Orange/ IMT/ OLN/ CNC/ NCA/ SINA Network architect for innovative services Future of the Network community member Open source Orange community manager tel. +33 (0) 296 072 106 mob. +33 (0) 637 753 326 morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -- Morgan Richomme Orange/ IMT/ OLN/ CNC/ NCA/ SINA Network architect for innovative services Future of the Network community member Open source Orange community manager tel. +33 (0) 296 072 106 mob. +33 (0) 637 753 326 morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -- Morgan Richomme Orange/ IMT/ OLN/ CNC/ NCA/ SINA Network architect for innovative services Future of the Network community member Open source Orange community manager tel. +33 (0) 296 072 106 mob. +33 (0) 637 753 326 morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
