Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hi Manuel, I am sorry for the inconvenience that utils removal and some updates in Functest have brought in to sfc and other users. We understand your concern. Some internals out of Functest have been removed without warning, mainly because Functest prefers feature/companion projects to be in charge of their own code, scripts and scenarios, rather than providing some common functionalies. To be honest, we could not know exactly every method consumed by feature projects, but if we did, we must have informed you in advance, just like the change of openstack_utils. Even though we double-checked if they were used, we cared more about functest internal tests and Releng jobs. Unfortunately Releng-XCI has been a bit ignored, as we are not sure where to check the sfc jobs on XCI. However, it would not bother more if we could initiate a wiki page listing all the methods to be deprecated (as Cedric proposed ) and add all the potential users as reviewers ( a new group for reviewing could be created) when publishing the related patches, eg,. "TEST_DB_URL", creds rename and something like that. Nice weekend :) Best Regards, Linda -- 王戊林 Linda M: +86-18560046533 E: wangwu...@huawei.com<mailto:wangwu...@huawei.com> 产品与解决方案-云核心网NFV研究部 Products & Solutions-NFV Research Dept,CCN From:cedric.olliv...@orange.com To:Manuel Buil,OPNFV-TECH-DISCUSS OPNFV, Date:2018-02-03 00:33:27 Subject:Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? No, “blocking innovation” was simply related to a possible inability to change our internal methods (timethis()) which are out of our Framework. You highlighted a real problem: we need to remove obsolete utils which don’t meet our criteria and may be possibly reused by others. Cleaning our tree could create side effects in OPNFV projects even if we mostly double check anyway. Yes we should have warned about the next end user changes: TEST_DB_URL and creds (and we previously removed prepare_env). For the previous release, we have simply updated a wiki page (Running Alpine container) for all the changes from an end-users POV. https://wiki.opnfv.org/display/functest/Run+Alpine+Functest+containers Why not creating a new wiki page for master as well. But I consider gerrit as the best place simply because it would be delayed by mails or by wiki pages. That may be considered as the cost to select master (it’s also the key challenges of XCI ; for instance odl mechanism driver and ODL neutron API may be temporarily incompatible if both are master). The issue is raised because we simply updated Functest and Releng jobs to avoid breaking gate but that’s true that we forgot releng-xci. But such changes rarely happened: here we are integrating the first Kubernetes testcases (creds) and we are preparing Xtesting. Have a nice weekend, Cédric De : Manuel Buil [mailto:mb...@suse.com] Envoyé : vendredi 2 février 2018 16:25 À : OLLIVIER Cédric IMT/OLN; OPNFV-TECH-DISCUSS OPNFV Objet : Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi Cedric, I probably expressed myself wrongly since you understood that I intent to block innovation and host SFC functions in functest but that is not my intention. My objective with this mail is to check if there exists (perhaps not), an efficient way for testing frameworks to communicate in advance important changes which will break things in the projects using those frameworks. I have a very good example: functest warned us during the Euphrates cycle that openstack_utils was not going to be maintained anymore and that we should stop using that and migrate to an alternative during the Fraser development cycle. You guys even suggested us an alternative (SNAPs). That was really helpful and we were able to plan the task accordingly in our SFC backlog and right now we are 100% compatible with SNAPs. Note that as far as I know, you haven't removed the openstack_utils yet (or at least we were able to use it while doing the migration). However, during the last weeks, some functest changes broke our testing (apart from that function I mentioned as an example): * It seems like the openrc file changed the name in the functest container and /home/opnfv/functest/conf/openstack.creds does not work anymore * TEST_DB_URL is now an environment variable and cannot be consumed from the functest config file To understand how to fix those, we had to go into the IRC channel and ping you. If this is only SFC guys doing it, I guess we could continue with option C, but if now more users start to work on master (XCI and APEX), I suspect the number pings you will receive will increase. And I guess the same will happen with yardstick, that's why I don't think this is a SFC <--> Functest issue but a more general ways of working issue. Regards,
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
No, “blocking innovation” was simply related to a possible inability to change our internal methods (timethis()) which are out of our Framework. You highlighted a real problem: we need to remove obsolete utils which don’t meet our criteria and may be possibly reused by others. Cleaning our tree could create side effects in OPNFV projects even if we mostly double check anyway. Yes we should have warned about the next end user changes: TEST_DB_URL and creds (and we previously removed prepare_env). For the previous release, we have simply updated a wiki page (Running Alpine container) for all the changes from an end-users POV. https://wiki.opnfv.org/display/functest/Run+Alpine+Functest+containers Why not creating a new wiki page for master as well. But I consider gerrit as the best place simply because it would be delayed by mails or by wiki pages. That may be considered as the cost to select master (it’s also the key challenges of XCI ; for instance odl mechanism driver and ODL neutron API may be temporarily incompatible if both are master). The issue is raised because we simply updated Functest and Releng jobs to avoid breaking gate but that’s true that we forgot releng-xci. But such changes rarely happened: here we are integrating the first Kubernetes testcases (creds) and we are preparing Xtesting. Have a nice weekend, Cédric De : Manuel Buil [mailto:mb...@suse.com] Envoyé : vendredi 2 février 2018 16:25 À : OLLIVIER Cédric IMT/OLN; OPNFV-TECH-DISCUSS OPNFV Objet : Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi Cedric, I probably expressed myself wrongly since you understood that I intent to block innovation and host SFC functions in functest but that is not my intention. My objective with this mail is to check if there exists (perhaps not), an efficient way for testing frameworks to communicate in advance important changes which will break things in the projects using those frameworks. I have a very good example: functest warned us during the Euphrates cycle that openstack_utils was not going to be maintained anymore and that we should stop using that and migrate to an alternative during the Fraser development cycle. You guys even suggested us an alternative (SNAPs). That was really helpful and we were able to plan the task accordingly in our SFC backlog and right now we are 100% compatible with SNAPs. Note that as far as I know, you haven't removed the openstack_utils yet (or at least we were able to use it while doing the migration). However, during the last weeks, some functest changes broke our testing (apart from that function I mentioned as an example): * It seems like the openrc file changed the name in the functest container and /home/opnfv/functest/conf/openstack.creds does not work anymore * TEST_DB_URL is now an environment variable and cannot be consumed from the functest config file To understand how to fix those, we had to go into the IRC channel and ping you. If this is only SFC guys doing it, I guess we could continue with option C, but if now more users start to work on master (XCI and APEX), I suspect the number pings you will receive will increase. And I guess the same will happen with yardstick, that's why I don't think this is a SFC <--> Functest issue but a more general ways of working issue. Regards, Manuel On Fri, 2018-02-02 at 14:24 +, cedric.olliv...@orange.com<mailto:cedric.olliv...@orange.com> wrote: Hello Manuel, In case of Functest, we are simply improving our code that why we are removing uncovered obsolete (and unused internally) functions. As we take care of pylint output and coverage, we should stop maintaining modules/functions which are out of our standards and unused by Functest. https://git.opnfv.org/functest/tree/tox.ini It’s well known that we are getting rid of openstack utils to leverage on snaps and we are removing unused functest utils We have worked on SDNVPN about that point before and we should apply the same rules for SFC. I don’t understand why Functest should host functions needed by SFC. In fact this issue is on the client side (SFC) which still uses obsolete Functest functions and your email seems asking to block innovation. I don’t think it’s an issue regarding APEX master or XCI for which this global improvement initiated from E is very helpful. Be free to reuse this method in your tree as it could have belong to instead. Cédric De : opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] De la part de Manuel Buil Envoyé : vendredi 2 février 2018 13:00 À : OPNFV-TECH-DISCUSS OPNFV Objet : [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functe
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hi Georg, I think your proposal could be a good solution!! Having an API which guarantees us some stability will surely solve the problems I described. I am not sure if this will be too much effort for the testing frameworks though and how the API model could be developed. Regards, Manuel On Fri, 2018-02-02 at 14:42 +, Georg Kunz wrote: > > > Hi all, > > > > In the context of this discussion I am wondering if one of the official goals of Functest is to provide common functionality to support developers in OPNFV to write functional > > > tests? That was my understanding, but I might be wrong. In any case, the Functest team along with the broader community should have a clear common understanding of this. > > > > Assuming this is the case, then the functionality which Functest provides constitutes an API. Typically, APIs exhibit some kind of stability along with some kind of deprecation > > process. It would in this case be beneficial for the broader community if such processes are defined and known. > > > > If Functest is not considered a framework providing common functionality, well, then it is indeed a problem of SFC. However, I do not consider this a valuable approach because > > > > it might just result in every project creating similar code for the same tasks, e.g., measuring execution times as in Manuel’s case. Moreover, Brian’s statement that he is trying to avoid using Functest functionality in his projects doesn’t really feel right > from a community perspective… > > Just my two cents as an observer. > > Best regards > Georg > > > > > > > > From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-d iscuss-boun...@lists.opnfv.org] > On Behalf Of cedric.olliv...@orange.com > > Sent: Friday, February 02, 2018 3:25 PM > > > > To: Manuel Buil ; OPNFV-TECH-DISCUSS OPNFV > > > Subject: Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? > > > > > > Hello Manuel, > > > In case of Functest, we are simply improving our code that why we are removing uncovered obsolete (and unused internally) functions. > > > As we take care of pylint output and coverage, we should stop maintaining modules/functions which are out of our standards and > unused by Functest. > https://git.opnfv.org/functest/tree/tox.ini > > > It’s well known that we are getting rid of openstack utils to leverage on snaps and we are removing unused functest utils > > We have worked on SDNVPN about that point before and we should apply the same rules for SFC. > I don’t understand why Functest should host functions needed by SFC. > > > > > In fact this issue is on the client side (SFC) which still uses obsolete Functest functions and your email seems asking to block innovation. > > I don’t think it’s an issue regarding APEX master or XCI for which this global improvement initiated from E is very helpful. > > > Be free to reuse this method in your tree as it could have belong to instead. > > Cédric > > > > De : > > opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss -boun...@lists.opnfv.org] > De la part de Manuel Buil > > Envoyé : vendredi 2 février 2018 13:00 > > À : OPNFV-TECH-DISCUSS OPNFV > > > Objet : [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? > > > > > > > Hi, > > > > > > > > > > > As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite > > > > > > some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/o pnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff- b5f06ecfb223c80624f432ef33cf1fdd and > > suddenly our tests are not working and we don't know whether there is an alternative. > > > > > > > > > > > Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I > > > > > guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of commu
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hi Cedric, I probably expressed myself wrongly since you understood that I intent to block innovation and host SFC functions in functest but that is not my intention. My objective with this mail is to check if there exists (perhaps not), an efficient way for testing frameworks to communicate in advance important changes which will break things in the projects using those frameworks. I have a very good example: functest warned us during the Euphrates cycle that openstack_utils was not going to be maintained anymore and that we should stop using that and migrate to an alternative during the Fraser development cycle. You guys even suggested us an alternative (SNAPs). That was really helpful and we were able to plan the task accordingly in our SFC backlog and right now we are 100% compatible with SNAPs. Note that as far as I know, you haven't removed the openstack_utils yet (or at least we were able to use it while doing the migration). However, during the last weeks, some functest changes broke our testing (apart from that function I mentioned as an example): * It seems like the openrc file changed the name in the functest container and /home/opnfv/functest/conf/openstack.creds does not work anymore * TEST_DB_URL is now an environment variable and cannot be consumed from the functest config file To understand how to fix those, we had to go into the IRC channel and ping you. If this is only SFC guys doing it, I guess we could continue with option C, but if now more users start to work on master (XCI and APEX), I suspect the number pings you will receive will increase. And I guess the same will happen with yardstick, that's why I don't think this is a SFC <--> Functest issue but a more general ways of working issue. Regards, Manuel On Fri, 2018-02-02 at 14:24 +, cedric.olliv...@orange.com wrote: > > > Hello Manuel, > > > In case of Functest, we are simply improving our code that why we are removing uncovered obsolete (and unused internally) functions. > > > As we take care of pylint output and coverage, we should stop maintaining modules/functions which are out of our standards and > unused by Functest. > https://git.opnfv.org/functest/tree/tox.ini > > > It’s well known that we are getting rid of openstack utils to leverage on snaps and we are removing unused functest utils > > We have worked on SDNVPN about that point before and we should apply the same rules for SFC. > I don’t understand why Functest should host functions needed by SFC. > > > > > In fact this issue is on the client side (SFC) which still uses obsolete Functest functions and your email seems asking to block innovation. > > I don’t think it’s an issue regarding APEX master or XCI for which this global improvement initiated from E is very helpful. > > > Be free to reuse this method in your tree as it could have belong to instead. > > Cédric > > > > > > De : opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-di scuss-boun...@lists.opnfv.org] > De la part de Manuel Buil > > Envoyé : vendredi 2 février 2018 13:00 > > À : OPNFV-TECH-DISCUSS OPNFV > > > Objet : [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? > > > > > > > Hi, > > > > > > > > > > > As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying > > > > > > to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/opnfv/functest/commit/c6092 cb676363d89f366dc9a416ba6c53eeea33f#diff- b5f06ecfb223c80624f432ef33cf1fdd and > > suddenly our tests are not working and we don't know whether there is an alternative. > > > > > > > > > > > Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same > > > > could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have > some ideas: > > > > > > > > > > A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, p
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Yes, the “issue” is simply related that we all leverage on git master instead of released packages (see OpenStack Requirements’ upper-constraints) during the development cycle. More, this case is a little bit more complex as this function shouldn’t be considered as part of our framework and we shouldn’t expect this obsolete internal method to reused outside. Cédric De : opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] De la part de SULLIVAN, BRYAN L (BRYAN L) Envoyé : vendredi 2 février 2018 15:02 À : Manuel Buil; OPNFV-TECH-DISCUSS OPNFV Objet : Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Manuel, Apart from the basic issue you are facing being dependency on test frameworks (which can be a convenient tool, until they change out from under you without notice), the basic problem OPNFV faces here is that it needs to, and easily can, provide such notice automatically. For example, when there is a patch that will cause dependencies to break, the easiest case being that some function no longer exists, it should be straightforward to run a scan of all OPNFV repos to check for references to that function. Other cases, e.g. where a function changes in definition/input etc may be harder, but there should also be notice somehow, supported by a similar scan for references to that function. In Models, VES, and Copper, I solve the basic problem you are facing by not depending upon any of the test frameworks/utils, and to the extent possible on none of the installer frameworks/utils either. But in your case I agree the community should support effective notice that something is likely to break. Thanks, Bryan Sullivan | AT&T From: opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Manuel Buil Sent: Friday, February 02, 2018 4:00 AM To: OPNFV-TECH-DISCUSS OPNFV mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/opnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff-b5f06ecfb223c80624f432ef33cf1fdd<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_opnfv_functest_commit_c6092cb676363d89f366dc9a416ba6c53eeea33f-23diff-2Db5f06ecfb223c80624f432ef33cf1fdd&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=j8ivdxqB81FrnWMaB0iNxRfKwI3TQfVMEJhuGfLlZdY&s=RkBPe60Yxqt5Ez4G-fIQK-AzCOMITZZo6hVSQ3jLaZc&e=> and suddenly our tests are not working and we don't know whether there is an alternative. Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have some ideas: A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, please check this patch: " B) Add gates in functest/yardstick projects which run tests of their customer projects (as in, SFC is a customer of functest). This way, projects could be warned on time C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick master D) ?? From what I heard in the TSC, Apex is going to join the XCI philosophy and allow working with the tip of master, so it seems to me that more functest and potentially yardstick users are going to hit this problem, that's why I believe it could be a good time to discuss possible solutions. Please don't get me wrong, I am not trying to blame functest (or yardstick), I just want to share the problems we are having with the current ways of working and try to find a solution :) Thanks, Manuel _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploi
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hi all, In the context of this discussion I am wondering if one of the official goals of Functest is to provide common functionality to support developers in OPNFV to write functional tests? That was my understanding, but I might be wrong. In any case, the Functest team along with the broader community should have a clear common understanding of this. Assuming this is the case, then the functionality which Functest provides constitutes an API. Typically, APIs exhibit some kind of stability along with some kind of deprecation process. It would in this case be beneficial for the broader community if such processes are defined and known. If Functest is not considered a framework providing common functionality, well, then it is indeed a problem of SFC. However, I do not consider this a valuable approach because it might just result in every project creating similar code for the same tasks, e.g., measuring execution times as in Manuel’s case. Moreover, Brian’s statement that he is trying to avoid using Functest functionality in his projects doesn’t really feel right from a community perspective… Just my two cents as an observer. Best regards Georg From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of cedric.olliv...@orange.com Sent: Friday, February 02, 2018 3:25 PM To: Manuel Buil ; OPNFV-TECH-DISCUSS OPNFV Subject: Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hello Manuel, In case of Functest, we are simply improving our code that why we are removing uncovered obsolete (and unused internally) functions. As we take care of pylint output and coverage, we should stop maintaining modules/functions which are out of our standards and unused by Functest. https://git.opnfv.org/functest/tree/tox.ini It’s well known that we are getting rid of openstack utils to leverage on snaps and we are removing unused functest utils We have worked on SDNVPN about that point before and we should apply the same rules for SFC. I don’t understand why Functest should host functions needed by SFC. In fact this issue is on the client side (SFC) which still uses obsolete Functest functions and your email seems asking to block innovation. I don’t think it’s an issue regarding APEX master or XCI for which this global improvement initiated from E is very helpful. Be free to reuse this method in your tree as it could have belong to instead. Cédric De : opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] De la part de Manuel Buil Envoyé : vendredi 2 février 2018 13:00 À : OPNFV-TECH-DISCUSS OPNFV Objet : [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/opnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff-b5f06ecfb223c80624f432ef33cf1fdd and suddenly our tests are not working and we don't know whether there is an alternative. Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have some ideas: A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, please check this patch: " B) Add gates in functest/yardstick projects which run tests of their customer projects (as in, SFC is a customer of functest). This way, projects could be warned on time C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick master D) ?? From what I heard in the TSC, Apex is going to join the XCI philosophy and allow working with the tip of master, so it seems to me that more functest and potentially yardstick users are going to hit this problem, that's why I believe it could be a good time to discuss possible solutions. Please don't get me wrong, I am not trying to blame functest (or yardstick), I just want to share the problems we are having with the current ways of working
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hello Manuel, In case of Functest, we are simply improving our code that why we are removing uncovered obsolete (and unused internally) functions. As we take care of pylint output and coverage, we should stop maintaining modules/functions which are out of our standards and unused by Functest. https://git.opnfv.org/functest/tree/tox.ini It’s well known that we are getting rid of openstack utils to leverage on snaps and we are removing unused functest utils We have worked on SDNVPN about that point before and we should apply the same rules for SFC. I don’t understand why Functest should host functions needed by SFC. In fact this issue is on the client side (SFC) which still uses obsolete Functest functions and your email seems asking to block innovation. I don’t think it’s an issue regarding APEX master or XCI for which this global improvement initiated from E is very helpful. Be free to reuse this method in your tree as it could have belong to instead. Cédric De : opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] De la part de Manuel Buil Envoyé : vendredi 2 février 2018 13:00 À : OPNFV-TECH-DISCUSS OPNFV Objet : [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/opnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff-b5f06ecfb223c80624f432ef33cf1fdd and suddenly our tests are not working and we don't know whether there is an alternative. Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have some ideas: A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, please check this patch: " B) Add gates in functest/yardstick projects which run tests of their customer projects (as in, SFC is a customer of functest). This way, projects could be warned on time C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick master D) ?? From what I heard in the TSC, Apex is going to join the XCI philosophy and allow working with the tip of master, so it seems to me that more functest and potentially yardstick users are going to hit this problem, that's why I believe it could be a good time to discuss possible solutions. Please don't get me wrong, I am not trying to blame functest (or yardstick), I just want to share the problems we are having with the current ways of working and try to find a solution :) Thanks, Manuel _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. ___ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
Re: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Manuel, Apart from the basic issue you are facing being dependency on test frameworks (which can be a convenient tool, until they change out from under you without notice), the basic problem OPNFV faces here is that it needs to, and easily can, provide such notice automatically. For example, when there is a patch that will cause dependencies to break, the easiest case being that some function no longer exists, it should be straightforward to run a scan of all OPNFV repos to check for references to that function. Other cases, e.g. where a function changes in definition/input etc may be harder, but there should also be notice somehow, supported by a similar scan for references to that function. In Models, VES, and Copper, I solve the basic problem you are facing by not depending upon any of the test frameworks/utils, and to the extent possible on none of the installer frameworks/utils either. But in your case I agree the community should support effective notice that something is likely to break. Thanks, Bryan Sullivan | AT&T From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Manuel Buil Sent: Friday, February 02, 2018 4:00 AM To: OPNFV-TECH-DISCUSS OPNFV Subject: [opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master? Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.com/opnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff-b5f06ecfb223c80624f432ef33cf1fdd<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_opnfv_functest_commit_c6092cb676363d89f366dc9a416ba6c53eeea33f-23diff-2Db5f06ecfb223c80624f432ef33cf1fdd&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=j8ivdxqB81FrnWMaB0iNxRfKwI3TQfVMEJhuGfLlZdY&s=RkBPe60Yxqt5Ez4G-fIQK-AzCOMITZZo6hVSQ3jLaZc&e=> and suddenly our tests are not working and we don't know whether there is an alternative. Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have some ideas: A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, please check this patch: " B) Add gates in functest/yardstick projects which run tests of their customer projects (as in, SFC is a customer of functest). This way, projects could be warned on time C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick master D) ?? From what I heard in the TSC, Apex is going to join the XCI philosophy and allow working with the tip of master, so it seems to me that more functest and potentially yardstick users are going to hit this problem, that's why I believe it could be a good time to discuss possible solutions. Please don't get me wrong, I am not trying to blame functest (or yardstick), I just want to share the problems we are having with the current ways of working and try to find a solution :) Thanks, Manuel ___ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
[opnfv-tech-discuss] Could ways of working with testing frameworks be improved when working with master?
Hi, As you know, the SFC project is currently testing openstack master, that means we need to use functest master in order to consume the lastest patches in the projects we use. Unfortunately, in the last weeks, we wasted quite some time trying to understand why suddenly the tests were not able to run and the cause was a change in the master branch of functest. For example, we are using the function "timethis" from functest_utils, which was removed by this patch: https://github.co m/opnfv/functest/commit/c6092cb676363d89f366dc9a416ba6c53eeea33f#diff- b5f06ecfb223c80624f432ef33cf1fdd and suddenly our tests are not working and we don't know whether there is an alternative. Functest is the framework we use for our tests and ideally, we (SFC project) would like to get some heads up before that change is done, so that we are warned and we don't have to waste time investigating what changed. I guess the same could be applied to other core testing frameworks like Yardstick. However, this is complicated and I am not sure if there is a good solution to achieve that level of communication without impacting the efficiency of funcatest/yardstick/... development. I have some ideas: A) Functest/Yarstick leave the old functionality for a week adding a log saying "This is going to be deprecated, please check this patch: " B) Add gates in functest/yardstick projects which run tests of their customer projects (as in, SFC is a customer of functest). This way, projects could be warned on time C) Do nothing. Sorry, this is the consequence of consuming functest/yardstick master D) ?? >From what I heard in the TSC, Apex is going to join the XCI philosophy and allow working with the tip of master, so it seems to me that more functest and potentially yardstick users are going to hit this problem, that's why I believe it could be a good time to discuss possible solutions. Please don't get me wrong, I am not trying to blame functest (or yardstick), I just want to share the problems we are having with the current ways of working and try to find a solution :) Thanks, Manuel___ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
