Although it took a while to go through all your slides and understand them all 
(the 2 big example slides helped a lot), I agree in general with what you 
explain. There are some confusing assumptions like   "if a case(weekly) fails 
=> still weekly loop but daily status" ... but the concept behind sounds like a 
good approach to follow.
We should definitively move to different keywords to latest/daily/weekly. 
Silver/Gold/Platinum sounds good enough, and congratulations for those 
scenarios that get the Platinum badge from Functest because it comprises a lot 
of long tests..

The promotions are performed in a logical way and everything is calculated and 
stored in the central DB. Just some concerns come to my mind:

   - if a couple scenarios (say 3 or 4) are promoted to "Gold", it means that 
they will be running all the tests (weekly loop) in CI (unless there is a 
regression on a daily test if I understand correctly)  ==> each Funtest weekly 
run might take 3-5 hours (say 8 hours for Deploy+Functest+Yardstick) ==> CI 
needs more time to complete an entire installer loop going through all the 
scenarios. Today, in some cases, a scenario takes ~2 days to run again in CI 
(with only daily Functest tests) and some scenarios could be considered 
nowadays as "Gold".. Are we prepared to run weekly tests on so many scenarios? 
Will we be able to do this the weeks before a release?

   - This new logic is very CI-oriented. We also need to bypass it for the case 
that an end user just wants to run all the tests the same way we do now (i.e. 
functest testcase run all). So, the logic has to be implemented in another 
layer on top, not in the test project framework as such. I can imagine a layer 
between CI (Jenkins) and the test project. This layer just tells Functest what 
to execute according to the info in the DB.


-----Original Message-----
From: morgan.richo...@orange.com [mailto:morgan.richo...@orange.com] 
Sent: Thursday, October 13, 2016 17:12 PM
To: test...@lists.opnfv.org
Cc: Fatih Degirmenci; opnfv-tech-discuss@lists.opnfv.org; serena feng; 
yaohelan; Jose Lausuch
Subject: CI evolution on impact on test projects


I took the action point to describe the possible consequences of CI evolution 
on test projects.

Please find attached a pdf detailing what I tried to explain yesterday :)

I will be off next Thursday and will not be able to detail it before Barcelona 
but if you have comments/remarks

I have also the odp and yed files if you want to edit anything

@Kubi could you check Yardstick status info?

If you prefer I can put that on the wiki, but as there are many things to agree 
first (to be sure we are sharing the same view), I prefer to initiate first 
exchanges by mail to converge to a first level of stability.



Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites 
ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez 
le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les 
messages electroniques etant susceptibles d'alteration, Orange decline toute 
responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law; they should not be distributed, used 
or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

opnfv-tech-discuss mailing list

Reply via email to