Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
On 06/17/2015 06:54 AM, Ken'ichi Ohmichi wrote: 2015-06-17 12:38 GMT+09:00 Yuiko Takada yuikotakada0...@gmail.com: Then, as you and Matt and Dimitry talked about this on IRC few days ago, We can add Ironic/Ironic-inspector tests into Tempest still, right? So that I've started to implement a test in Tempest, but I'm facing another issue. As you know, Ironic API has microversions, and Ironic-inspector can run with microversion 1.6. But currently there is no feature testing specific Ironic API microversions on Tempest, right? So that, we have to think about some solutions. (1) Make testing specific Ironic API microversions on Tempest possible adam_g is posting this patch set. https://review.openstack.org/166386 (2)Using tempest_lib instead of adding tests into Tempest Is tempest_lib available already? Or do we need to wait for something will be merged? I guess the above question seems multiple factors are mixed. You want to test ironic-inspector behaviors by * using ironic-inspector REST APIs directly without Ironic * using Ironic REST APIs which need newer microversion right? Hi, thanks for clarifying, let me jump in :) The former is more or less covered by functional testing, so I'd like us to concentrate on the latter, and run it voting on inspector repo and non-voting on Ironic for the time being. For the first test, you can implement without considering microversion. The test just calls ironic-inspector REST APIs directly and checks its behavior. You can implement the test on Tempest/ironic-inspector repository. Current tempest-lib seems enough to implement tests in ironic-inspector repository as features, but it is better to wait for Tempest's external interface spec[1] approval. It is trying to define directory structure of Tempest-like tests on each project repository and Tempest will discover tests based on the directory structure and run them. So if implementing tests on ironic-inspector repository before the spec approval, you will need to change the directory structure again in the future. This wait part bothers me to some extend, because gate absence badly affects us for some time, but fine. Thanks for heads up anyway. For the second test, microversions support is necessary on Tempest side and adam_g's patch seems good for implementing it. My main concern of microversions tests is how to run multiple microversions on the gate. We have discussed that in Nova design session of Vancouver summit and the conclusion is * Minimum microversion * Maximum microversion * Interesting microversions as the gate test. Facepalm. That's what I was talking about (and what we actually ended up in Ironic with): we're introducing a ton of non-tested (and thus presumably broken) microversions, because it's cool to do. Ok, that's another thread :) IMO Interesting microversions would be the last microversions of each release(Kilo, Liberty, ..) I feel. With Ironic intermediate release, it will be more, I estimate it to be 5-6 per year, but of course I can't tell for sure. I have qa-spec[2] for testing microversions on the gate, but that is not complete yet. That will affect how to specify/run microversions tests on Tempest. So now I'm not sure yet the way to specify microversion on current adam_g's patch is the best. So my recommendation/hope is that we concentrate on Tempest's external interface spec[1] and make it better together, then we can implement Tempest-like tests on each repository after that. As the next step, we will test microversions on the same way between projects based on conclusion of the spec[2]. What I'd prefer us to start with is a gate test, which just sets devstack with our plugin and runs a shell script testing a couple of basic things. This will be a HUGE leap forward for inspector, compared to only limited functional testing we have now. So maybe we should start with it, and keep an eye on the tempest-lib stuff, wdyt? (3)Make Ironic-inspector available even if microversion 1.6 Dmitry is posting this patch set. https://review.openstack.org/192196 # I don't mean asking you to review this, don't worry :p I've reviewed it already :) Thanks Ken Ohmichi --- [1]: https://review.openstack.org/#/c/184992/ [2]: https://review.openstack.org/#/c/169126/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
2015-06-17 12:38 GMT+09:00 Yuiko Takada yuikotakada0...@gmail.com: Then, as you and Matt and Dimitry talked about this on IRC few days ago, We can add Ironic/Ironic-inspector tests into Tempest still, right? So that I've started to implement a test in Tempest, but I'm facing another issue. As you know, Ironic API has microversions, and Ironic-inspector can run with microversion 1.6. But currently there is no feature testing specific Ironic API microversions on Tempest, right? So that, we have to think about some solutions. (1) Make testing specific Ironic API microversions on Tempest possible adam_g is posting this patch set. https://review.openstack.org/166386 (2)Using tempest_lib instead of adding tests into Tempest Is tempest_lib available already? Or do we need to wait for something will be merged? I guess the above question seems multiple factors are mixed. You want to test ironic-inspector behaviors by * using ironic-inspector REST APIs directly without Ironic * using Ironic REST APIs which need newer microversion right? For the first test, you can implement without considering microversion. The test just calls ironic-inspector REST APIs directly and checks its behavior. You can implement the test on Tempest/ironic-inspector repository. Current tempest-lib seems enough to implement tests in ironic-inspector repository as features, but it is better to wait for Tempest's external interface spec[1] approval. It is trying to define directory structure of Tempest-like tests on each project repository and Tempest will discover tests based on the directory structure and run them. So if implementing tests on ironic-inspector repository before the spec approval, you will need to change the directory structure again in the future. For the second test, microversions support is necessary on Tempest side and adam_g's patch seems good for implementing it. My main concern of microversions tests is how to run multiple microversions on the gate. We have discussed that in Nova design session of Vancouver summit and the conclusion is * Minimum microversion * Maximum microversion * Interesting microversions as the gate test. IMO Interesting microversions would be the last microversions of each release(Kilo, Liberty, ..) I feel. I have qa-spec[2] for testing microversions on the gate, but that is not complete yet. That will affect how to specify/run microversions tests on Tempest. So now I'm not sure yet the way to specify microversion on current adam_g's patch is the best. So my recommendation/hope is that we concentrate on Tempest's external interface spec[1] and make it better together, then we can implement Tempest-like tests on each repository after that. As the next step, we will test microversions on the same way between projects based on conclusion of the spec[2]. (3)Make Ironic-inspector available even if microversion 1.6 Dmitry is posting this patch set. https://review.openstack.org/192196 # I don't mean asking you to review this, don't worry :p I've reviewed it already :) Thanks Ken Ohmichi --- [1]: https://review.openstack.org/#/c/184992/ [2]: https://review.openstack.org/#/c/169126/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
Hi, Dmitry, thank you for putting it up, and also Ken'ichi, thank you for your reply. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? Exactly. All we want to do is run Ironic+Ironic-inspector tests on gate. Then, as you and Matt and Dimitry talked about this on IRC few days ago, We can add Ironic/Ironic-inspector tests into Tempest still, right? So that I've started to implement a test in Tempest, but I'm facing another issue. As you know, Ironic API has microversions, and Ironic-inspector can run with microversion 1.6. But currently there is no feature testing specific Ironic API microversions on Tempest, right? So that, we have to think about some solutions. (1) Make testing specific Ironic API microversions on Tempest possible adam_g is posting this patch set. https://review.openstack.org/166386 (2)Using tempest_lib instead of adding tests into Tempest Is tempest_lib available already? Or do we need to wait for something will be merged? (3)Make Ironic-inspector available even if microversion 1.6 Dmitry is posting this patch set. https://review.openstack.org/192196 # I don't mean asking you to review this, don't worry :p Could you please think about the best and fast solution together? Best Regards, Yuiko Takada 2015-06-10 17:57 GMT+09:00 Ken'ichi Ohmichi ken1ohmi...@gmail.com: 2015-06-10 16:48 GMT+09:00 Dmitry Tantsur dtant...@redhat.com: On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Yeah, right. Neutron is already doing that. maybe neutron/tests/api/ of Neutron repository will be a hint for it. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec for https://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/...;? Yeah, will be. The idea of this external interface is based on DevStack's one. I think we will be able to use it on the gate like that. Thanks Ken'ichi Ohmichi --- On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
Dmitry, If you chose to use Rally framework for testing there are 3 opportunities: - Keep Rally plugins (tests) in separated tree - Keep Rally plugins (tests) in your project tree - Keep Rally plugins (tests) in Rally repo Rally plugins can be used for all kinds of testing: (perf, scalability, load...) so you are killing two birds with one stone. P.S. I would imho prefer to keep all high quality plugins inside Rally repo to simplify operators life.. Best regards, Boris Pavlovic On Wed, Jun 10, 2015 at 11:57 AM, Ken'ichi Ohmichi ken1ohmi...@gmail.com wrote: 2015-06-10 16:48 GMT+09:00 Dmitry Tantsur dtant...@redhat.com: On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Yeah, right. Neutron is already doing that. maybe neutron/tests/api/ of Neutron repository will be a hint for it. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec for https://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/...;? Yeah, will be. The idea of this external interface is based on DevStack's one. I think we will be able to use it on the gate like that. Thanks Ken'ichi Ohmichi --- On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added one BP to [4], so please take a look. https://blueprints.launchpad.net/ironic-inspector/+spec/delete-db-api Looks good, though I don't think it's a big priority for 2.0.0. Definitely worth doing for 2.1.0. Thanks for assigning for tempest work, that's definitely a priority right now. BTW, how do you think about Ironic-inspector's release model? You wrote Version released with Ironic Liberty as Ironic-inspector Version 2.1.0 in etherpad [3], but as you know, Ironic's release model has changed to feature releases.(right?) Should we make our release model same as Ironic? I guess the whole idea of new release models is NOT to tie projects to each
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
On 06/10/2015 11:57 AM, Boris Pavlovic wrote: Dmitry, If you chose to use Rally framework for testing there are 3 opportunities: - Keep Rally plugins (tests) in separated tree - Keep Rally plugins (tests) in your project tree - Keep Rally plugins (tests) in Rally repo Rally plugins can be used for all kinds of testing: (perf, scalability, load...) so you are killing two birds with one stone. P.S. I would imho prefer to keep all high quality plugins inside Rally repo to simplify operators life.. Hi, that sounds interesting, I'll have a look. Note, however, that Inspector integration testing highly depends on Ironic one, so unless Ironic adapts/agrees to adapt Rally, it will be hard to Inspector to do it. Best regards, Boris Pavlovic On Wed, Jun 10, 2015 at 11:57 AM, Ken'ichi Ohmichi ken1ohmi...@gmail.com mailto:ken1ohmi...@gmail.com wrote: 2015-06-10 16:48 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Yeah, right. Neutron is already doing that. maybe neutron/tests/api/ of Neutron repository will be a hint for it. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec forhttps://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/...;? Yeah, will be. The idea of this external interface is based on DevStack's one. I think we will be able to use it on the gate like that. Thanks Ken'ichi Ohmichi --- On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
Dmitry, We introduced recently dsvm rally ironic job: https://review.openstack.org/#/c/187997/ Now we are working on Rally tests for Ironic: https://review.openstack.org/#/c/186064/ Don't hesitate to join us=) Best regards, Boris Pavlovic On Wed, Jun 10, 2015 at 1:23 PM, Dmitry Tantsur dtant...@redhat.com wrote: On 06/10/2015 11:57 AM, Boris Pavlovic wrote: Dmitry, If you chose to use Rally framework for testing there are 3 opportunities: - Keep Rally plugins (tests) in separated tree - Keep Rally plugins (tests) in your project tree - Keep Rally plugins (tests) in Rally repo Rally plugins can be used for all kinds of testing: (perf, scalability, load...) so you are killing two birds with one stone. P.S. I would imho prefer to keep all high quality plugins inside Rally repo to simplify operators life.. Hi, that sounds interesting, I'll have a look. Note, however, that Inspector integration testing highly depends on Ironic one, so unless Ironic adapts/agrees to adapt Rally, it will be hard to Inspector to do it. Best regards, Boris Pavlovic On Wed, Jun 10, 2015 at 11:57 AM, Ken'ichi Ohmichi ken1ohmi...@gmail.com mailto:ken1ohmi...@gmail.com wrote: 2015-06-10 16:48 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Yeah, right. Neutron is already doing that. maybe neutron/tests/api/ of Neutron repository will be a hint for it. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec forhttps://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/.. .? Yeah, will be. The idea of this external interface is based on DevStack's one. I think we will be able to use it on the gate like that. Thanks Ken'ichi Ohmichi --- On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad
[openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
Hi, QA folks! As ironic-inspector is joining the openstack/* crows, we're faces with integration testing question. At the summit we discussed with Devananda that it makes sense for inspector + ironic integration test to live in tempest with other bare metal tests. However, judging by https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent even Ironic integration tests will no longer be welcome in Tempest itself, to say nothing about pretty new Inspector. Are there any hints now on where we can start with our integration tests? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. Cheers, Dmitry On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added one BP to [4], so please take a look. https://blueprints.launchpad.net/ironic-inspector/+spec/delete-db-api Looks good, though I don't think it's a big priority for 2.0.0. Definitely worth doing for 2.1.0. Thanks for assigning for tempest work, that's definitely a priority right now. BTW, how do you think about Ironic-inspector's release model? You wrote Version released with Ironic Liberty as Ironic-inspector Version 2.1.0 in etherpad [3], but as you know, Ironic's release model has changed to feature releases.(right?) Should we make our release model same as Ironic? I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. Best Regards, Yuiko Takada(Inspector team member) 2015-06-08 20:38 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: Hello, Inspector team! The renaming process is going pretty well, the last thing we need to do is to get Infra approval and actual rename [1][2]. I'd like to allow people (e.g. myself) to start packaging inspector under it's new name, so I'd like to make 2.0.0 release as soon as possible (as opposed to scheduling it to particular date). All breaking changes should land by this release - I don't expect 3.0.0 soon :) I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I would like 2.1.0 to be released with Ironic Liberty and be properly supported. Let me know what you think. Cheers, Dmitry [1] https://review.openstack.org/#/c/188030/ [2] https://review.openstack.org/#/c/188798/ [3] https://etherpad.openstack.org/p/liberty-ironic-discoverd [4]
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: Hi Dmitry, 2015-06-10 16:11 GMT+09:00 Dmitry Tantsur dtant...@redhat.com: Hi, QA folks! As ironic-inspector is joining the openstack/* crows, we're faces with integration testing question. At the summit we discussed with Devananda that it makes sense for inspector + ironic integration test to live in tempest with other bare metal tests. However, judging by https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent even Ironic integration tests will no longer be welcome in Tempest itself, to say nothing about pretty new Inspector. Yeah, right. On big-tent trend, it is difficult to cover/review test of all projects by a single Tempest-team. It is easy to imagine Tempest-team will be bottleneck of developments if continuing current way. +1 To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec for https://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/...;? Thanks Ken'ichi Ohmichi On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added one BP to [4], so please take a look. https://blueprints.launchpad.net/ironic-inspector/+spec/delete-db-api Looks good, though I don't think it's a big priority for 2.0.0. Definitely worth doing for 2.1.0. Thanks for assigning for tempest work, that's definitely a priority right now. BTW, how do you think about Ironic-inspector's release model? You wrote Version released with Ironic Liberty as Ironic-inspector Version 2.1.0 in etherpad [3], but as you know, Ironic's release model has changed to feature releases.(right?) Should we make our release model same as Ironic? I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. Best Regards, Yuiko Takada(Inspector team member) 2015-06-08 20:38 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: Hello, Inspector team! The renaming process is going pretty well,
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
2015-06-10 16:48 GMT+09:00 Dmitry Tantsur dtant...@redhat.com: On 06/10/2015 09:40 AM, Ken'ichi Ohmichi wrote: To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. So in my case we can start with putting test code to ironic-inspector tree using tempest-lib, right? Yeah, right. Neutron is already doing that. maybe neutron/tests/api/ of Neutron repository will be a hint for it. Will it be possible to run tests on Ironic as well using plugin from ironic-inspector? Yeah, it will be possible. but I'm guessing ironic-inspector is optional and Ironic should not depend on the gate test result of ironic-inspector. So maybe you just need to run Ironic tests on ironic-inspector gate tests, right? After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec for https://review.openstack.org/#/c/184992/ Cool, thanks! Does it mean that devstack-gate will also be updated to allow something like DEVSTACK_GATE_TEMPEST_PLUGINS=https://github.com/...;? Yeah, will be. The idea of this external interface is based on DevStack's one. I think we will be able to use it on the gate like that. Thanks Ken'ichi Ohmichi --- On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added one BP to [4], so please take a look. https://blueprints.launchpad.net/ironic-inspector/+spec/delete-db-api Looks good, though I don't think it's a big priority for 2.0.0. Definitely worth doing for 2.1.0. Thanks for assigning for tempest work, that's definitely a priority right now. BTW, how do you think about Ironic-inspector's release model? You wrote Version released with Ironic Liberty as Ironic-inspector Version 2.1.0 in etherpad [3], but as you know, Ironic's release model has changed to feature releases.(right?) Should we make our release model same as Ironic? I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. Best Regards, Yuiko Takada(Inspector team member) 2015-06-08 20:38 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: Hello, Inspector team! The renaming process is going pretty well, the last thing we need to do is to get Infra approval and actual rename [1][2]. I'd like to
Re: [openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
Hi Dmitry, 2015-06-10 16:11 GMT+09:00 Dmitry Tantsur dtant...@redhat.com: Hi, QA folks! As ironic-inspector is joining the openstack/* crows, we're faces with integration testing question. At the summit we discussed with Devananda that it makes sense for inspector + ironic integration test to live in tempest with other bare metal tests. However, judging by https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent even Ironic integration tests will no longer be welcome in Tempest itself, to say nothing about pretty new Inspector. Yeah, right. On big-tent trend, it is difficult to cover/review test of all projects by a single Tempest-team. It is easy to imagine Tempest-team will be bottleneck of developments if continuing current way. To solve it, we have decided the scope of Tempest as the etherpad mentioned. Are there any hints now on where we can start with our integration tests? For the other projects, we are migrating the test framework of Tempest to tempest-lib which is a library. So each project can implement their own tests in each repository by using the test framework of tempest-lib. After a quick look at devstack-gate I got an impression that it's expecting tests as part of tempest: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L600 Our final goal is to have devstack gate test for Ironic and Inspector projects working together. We have discussed external interfaces of Tempest on the summit, so that Tempest gathers tests from each project repository and runs them at the same time. There is a qa-spec for https://review.openstack.org/#/c/184992/ Thanks Ken'ichi Ohmichi On 06/10/2015 08:07 AM, Yuiko Takada wrote: Hi, Dmitry, I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. OK, I see. One more concern, about Tempest integration test which I will implement in V2.1.0, it seems like that we cannot add Ironic-inspector's tests into Tempest even if integration tests. Please see: https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent Good catch. I guess the answer depends on where Ironic integration tests are going to live - we're going to live with them. Let me retarget this thread to a wider audience. But I heard from you that Devananda thinks we need this in tempest itself. [3] Do you know something like current situation? Best Regards, Yuiko Takada 2015-06-09 15:59 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com: On 06/09/2015 03:49 AM, Yuiko Takada wrote: Hi, Dmitry, Thank you for notifying. I've updated our summit etherpad [3] with whatever priorities I remembered, please have a look. I've also untargeted a few things in launchpad [4] (and will probably untarget more later on). Please assign yourself, if you want something done in this release time frame. I've assigned one item to myself in [3], and also I added one BP to [4], so please take a look. https://blueprints.launchpad.net/ironic-inspector/+spec/delete-db-api Looks good, though I don't think it's a big priority for 2.0.0. Definitely worth doing for 2.1.0. Thanks for assigning for tempest work, that's definitely a priority right now. BTW, how do you think about Ironic-inspector's release model? You wrote Version released with Ironic Liberty as Ironic-inspector Version 2.1.0 in etherpad [3], but as you know, Ironic's release model has changed to feature releases.(right?) Should we make our release model same as Ironic? I guess the whole idea of new release models is NOT to tie projects to each other any more except for The Big Release twice a year :) So I think no, we don't need to. We still can do it, if we have something to release by the time Ironic releases, but I suggest deciding it on case-by-case basis. Best Regards, Yuiko Takada(Inspector team member) 2015-06-08 20:38 GMT+09:00 Dmitry Tantsur dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com mailto:dtant...@redhat.com: Hello, Inspector team! The renaming process is going pretty well, the last thing we need to do is to get Infra approval and actual rename [1][2]. I'd like to allow people (e.g. myself) to start packaging inspector under it's new name, so I'd like to make 2.0.0 release as soon as possible (as opposed to scheduling it to particular date). All
