Apart from the basic issue you are facing being dependency on test frameworks 
(which can be a convenient tool, until they change out from under you without 
notice), the basic problem OPNFV faces here is that it needs to, and easily 
can, provide such notice automatically. For example, when there is a patch that 
will cause dependencies to break, the easiest case being that some function no 
longer exists, it should be straightforward to run a scan of all OPNFV repos to 
check for references to that function. Other cases, e.g. where a function 
changes in definition/input etc may be harder, but there should also be notice 
somehow, supported by a similar scan for references to that function.

In Models, VES, and Copper, I solve the basic problem you are facing by not 
depending upon any of the test frameworks/utils, and to the extent possible on 
none of the installer frameworks/utils either. But in your case I agree the 
community should support effective notice that something is likely to break.

Bryan Sullivan | AT&T

[] On Behalf Of Manuel Buil
Sent: Friday, February 02, 2018 4:00 AM
Subject: [opnfv-tech-discuss] Could ways of working with testing frameworks be 
improved when working with master?


As you know, the SFC project is currently testing openstack master, that means 
we need to use functest master in order to consume the lastest patches in the 
projects we use. Unfortunately, in the last weeks, we wasted quite some time 
trying to understand why suddenly the tests were not able to run and the cause 
was a change in the master branch of functest. For example, we are using the 
function "timethis" from functest_utils, which was removed by this patch:<>
 and suddenly our tests are not working and we don't know whether there is an 

Functest is the framework we use for our tests and ideally, we (SFC project) 
would like to get some heads up before that change is done, so that we are 
warned and we don't have to waste time investigating what changed. I guess the 
same could be applied to other core testing frameworks like Yardstick. However, 
this is complicated and I am not sure if there is a good solution to achieve 
that level of communication without impacting the efficiency of 
funcatest/yardstick/... development. I have some ideas:

A) Functest/Yarstick leave the old functionality for a week adding a log saying 
"This is going to be deprecated, please check this patch: xxxx"

B) Add gates in functest/yardstick projects which run tests of their customer 
projects (as in, SFC is a customer of functest). This way, projects could be 
warned on time

C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick 

D) ??

From what I heard in the TSC, Apex is going to join the XCI philosophy and 
allow working with the tip of master, so it seems to me that more functest and 
potentially yardstick users are going to hit this problem, that's why I believe 
it could be a good time to discuss possible solutions.

Please don't get me wrong, I am not trying to blame functest (or yardstick), I 
just want to share the problems we are having with the current ways of working 
and try to find a solution :)

opnfv-tech-discuss mailing list

Reply via email to