@Morgan, thank you for your clearly illustration. I am trying to write a demo to update the status and trust_indicator now.
But the definition of status and trust_indicator still to be discuss. So if you have any suggestions, please do not hesitate to comment here. Once we come to a conclusion, I will write a python demo script or an API for you to implement it. And then our landing page will finally works. Any ideas are all welcome~ ☺ Regards, Jack Chan 发件人: morgan.richo...@orange.com [mailto:morgan.richo...@orange.com] 发送时间: 2017年9月15日 22:35 收件人: Brattain, Ross B; Gaoliang (kubi); mark.bei...@emc.com; Cooper, Trevor; MORTON, ALFRED C (AL); Yuyang (Gabriel); jalaus...@suse.com; chenjiankun; test...@lists.opnfv.org 抄送: opnfv-tech-discuss@lists.opnfv.org 主题: [OPNFV] [testing group] landing page: score and trust indicator Hi we initiated the landing page some time ago: http://testresults.opnfv.org/testing/#!/landingpage/table front and back end are now ready Jack started doing an update for Yardstick It is now possible to call the Test API to set a score and see/modify trust indicator after a CI run. during the plugfest we saw that we were not fully aligned on what we want to show on such landing page the recent discussion on the reporting pages show also some differences on the way we report our results I think we need to agree on a minimum common view during a Testing working meeting to have a consistent page Concretely we have 2 scenario parameters for this landing page: - Scenario score - Scenario Trust Indicator Scenario score *************** Today there are 2 main views VSPERF, Yardstick and Bottleneck => report Jenkins status assuming that - SLA/threshold neither defined (VSPERF) nor taken into account (Yardstick) to set test suite criteria to PASS or FAIL - if test suite deals with lots of tests => only 1 aggregated result corresponding to the test suite "runnability" in CI In Functest we do a calculation based on the sum of all the individual cases (https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617) that is why in reporting page we have score like 14/27, 18/18, 1/9 .... The first figure represent the success (18/18 = max score; 1/9 = very poor results) the max score represent the richness of the test suite, i.e. /9 = 3 test cases in the suite; / 27 = 9 test cases. We have a x3 factor due to the criteria of 4 consecutive runs. In the evolution I had in mind to report only the scenario status In Storperf Mark defined and takes into the threshold to set the test to PASS or FAIL So we may have 2 approaches for the landing page - The Jenkins status (nb run OK / nb attempted over the last N days) - The testing status (tests are run and results are according to project success criteria e.g. for functest we expect the max score, 15/18 will be considered as FAIL) What is your view? What kind of score would you give to the CI runs currently running? shall we be all consistent or is it up to each project to define what it wants to show in such page? Trust Indicator ************** this parameter allows to consider the scenario evolution versus time in previous discussions we considered 3 states: - Silver (init) - Gold - Platinium this indicator is related to the scenario on which a test project is running the test suite A simple proposal for scenario promotion [cid:image001.png@01D33181.8CE0EED0] /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
