Although it took a while to go through all your slides and understand them all
(the 2 big example slides helped a lot), I agree in general with what you
explain. There are some confusing assumptions like "if a case(weekly) fails
=> still weekly loop but daily status" ... but the concept behind sounds like a
good approach to follow.
We should definitively move to different keywords to latest/daily/weekly.
Silver/Gold/Platinum sounds good enough, and congratulations for those
scenarios that get the Platinum badge from Functest because it comprises a lot
of long tests..
The promotions are performed in a logical way and everything is calculated and
stored in the central DB. Just some concerns come to my mind:
- if a couple scenarios (say 3 or 4) are promoted to "Gold", it means that
they will be running all the tests (weekly loop) in CI (unless there is a
regression on a daily test if I understand correctly) ==> each Funtest weekly
run might take 3-5 hours (say 8 hours for Deploy+Functest+Yardstick) ==> CI
needs more time to complete an entire installer loop going through all the
scenarios. Today, in some cases, a scenario takes ~2 days to run again in CI
(with only daily Functest tests) and some scenarios could be considered
nowadays as "Gold".. Are we prepared to run weekly tests on so many scenarios?
Will we be able to do this the weeks before a release?
- This new logic is very CI-oriented. We also need to bypass it for the case
that an end user just wants to run all the tests the same way we do now (i.e.
functest testcase run all). So, the logic has to be implemented in another
layer on top, not in the test project framework as such. I can imagine a layer
between CI (Jenkins) and the test project. This layer just tells Functest what
to execute according to the info in the DB.
From: morgan.richo...@orange.com [mailto:morgan.richo...@orange.com]
Sent: Thursday, October 13, 2016 17:12 PM
Cc: Fatih Degirmenci; email@example.com; serena feng;
yaohelan; Jose Lausuch
Subject: CI evolution on impact on test projects
I took the action point to describe the possible consequences of CI evolution
on test projects.
Please find attached a pdf detailing what I tried to explain yesterday :)
I will be off next Thursday and will not be able to detail it before Barcelona
but if you have comments/remarks
I have also the odp and yed files if you want to edit anything
@Kubi could you check Yardstick status info?
If you prefer I can put that on the wiki, but as there are many things to agree
first (to be sure we are sharing the same view), I prefer to initiate first
exchanges by mail to converge to a first level of stability.
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites
ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez
le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les
messages electroniques etant susceptibles d'alteration, Orange decline toute
responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law; they should not be distributed, used
or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
opnfv-tech-discuss mailing list