Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/08/2015 03:58 AM, Dmitry Tantsur wrote: On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Tempest shouldn't be the *only* test for a project API. The projects themselves need to take some ownership for their API getting full coverage with in tree testing, including whatever microversion strategy they are employing. My original thoughts about this would be that Tempest tests without versions (so the low end), plus there could be specific Tempest tests added for new microversions. Regression of 'latest' should be a low probability event, caught by the project in tree. So I think that Tempest on every run for that is excessive. Instead I'd say that should go into periodic changes instead. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/08/2015 12:53 PM, Sean Dague wrote: On 04/08/2015 03:58 AM, Dmitry Tantsur wrote: On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Tempest shouldn't be the *only* test for a project API. The projects themselves need to take some ownership for their API getting full coverage with in tree testing, including whatever microversion strategy they are employing. Agreed, but in-tree testing is also not feasible with too many version. Even now we have 7 (1.0-1.6), if it continues, we'll have not less than 12 after L, 18 after M, etc. And we have to test every one of them for regressions at least occasionally, provided that we don't start to aggressively deprecated microversions. If we do start, then we'll start breaking people even more often, than we should. E.g. if someone writes a tool targeted at 1.1, and we deprecated 1.1 in M cycle, the tool will break, though maybe it can actually work with new API. My original thoughts about this would be that Tempest tests without versions (so the low end), plus there could be specific Tempest tests added for new microversions. Regression of 'latest' should be a low probability event, caught by the project in tree. So I think that Tempest on every run for that is excessive. Instead I'd say that should go into periodic changes instead. -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Thanks Ken Ohmichi --- [1]: https://review.openstack.org/#/c/166386/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/08/2015 07:38 AM, Dmitry Tantsur wrote: On 04/08/2015 12:53 PM, Sean Dague wrote: On 04/08/2015 03:58 AM, Dmitry Tantsur wrote: On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Tempest shouldn't be the *only* test for a project API. The projects themselves need to take some ownership for their API getting full coverage with in tree testing, including whatever microversion strategy they are employing. Agreed, but in-tree testing is also not feasible with too many version. Even now we have 7 (1.0-1.6), if it continues, we'll have not less than 12 after L, 18 after M, etc. And we have to test every one of them for regressions at least occasionally, provided that we don't start to aggressively deprecated microversions. If we do start, then we'll start breaking people even more often, than we should. E.g. if someone writes a tool targeted at 1.1, and we deprecated 1.1 in M cycle, the tool will break, though maybe it can actually work with new API. I do not understand how in tree testing is not feasible. In tree you have insights into all the branching that occurs within code so can very clearly understand what paths aren't possible. It should be a lot more straight forward than external black box testing where that can't be assume. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/08/2015 05:24 AM, Sean Dague wrote: On 04/08/2015 07:38 AM, Dmitry Tantsur wrote: On 04/08/2015 12:53 PM, Sean Dague wrote: On 04/08/2015 03:58 AM, Dmitry Tantsur wrote: On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Tempest shouldn't be the *only* test for a project API. The projects themselves need to take some ownership for their API getting full coverage with in tree testing, including whatever microversion strategy they are employing. Agreed, but in-tree testing is also not feasible with too many version. Even now we have 7 (1.0-1.6), if it continues, we'll have not less than 12 after L, 18 after M, etc. And we have to test every one of them for regressions at least occasionally, provided that we don't start to aggressively deprecated microversions. If we do start, then we'll start breaking people even more often, than we should. E.g. if someone writes a tool targeted at 1.1, and we deprecated 1.1 in M cycle, the tool will break, though maybe it can actually work with new API. I do not understand how in tree testing is not feasible. In tree you have insights into all the branching that occurs within code so can very clearly understand what paths aren't possible. It should be a lot more straight forward than external black box testing where that can't be assume. Exactly. The whole *point* of microversions was to allow the APIs to evolve in a backwards-compatible, structured and advertised way. The evolution of the APIs response and request payloads should be tested fully for each microversion added to the codebase -- in tree. -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
FWIW the Ironic microversion test patch mentioned on gerrit is only targeted at Tempest because thats where the API tests currenty live and from which our infra is setup to run. The eventual goal is to move all of tempest.api.baremetal.* to the Ironic tree, there's no reason why those proposed new tests wouldn't either. Those tests were designed to allow running against all available microversions or some configured subset, and to ensure tests for previous microversions run against newer. I think its perfectly feasible to test many microversions in tree or out, provided test coverage is kept sufficiently up to date as the APIs evolve. Adam On Wed, Apr 8, 2015 at 7:45 AM, Jay Pipes jaypi...@gmail.com wrote: On 04/08/2015 05:24 AM, Sean Dague wrote: On 04/08/2015 07:38 AM, Dmitry Tantsur wrote: On 04/08/2015 12:53 PM, Sean Dague wrote: On 04/08/2015 03:58 AM, Dmitry Tantsur wrote: On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Hi! I've already stated this point in #openstack-ironic and I'd like to reiterate: if we test only the lowest and the highest microversions essentially means (or at least might mean) that the other are broken. At least in Ironic only some unit tests actually touch code paths for versions 1.2-1.5. As we really can't test too many versions, I suggest we stop producing a microversion for every API feature feature change in L. No idea what to do with 1.2-1.5 now except for politely asking people not to use them :D Tempest shouldn't be the *only* test for a project API. The projects themselves need to take some ownership for their API getting full coverage with in tree testing, including whatever microversion strategy they are employing. Agreed, but in-tree testing is also not feasible with too many version. Even now we have 7 (1.0-1.6), if it continues, we'll have not less than 12 after L, 18 after M, etc. And we have to test every one of them for regressions at least occasionally, provided that we don't start to aggressively deprecated microversions. If we do start, then we'll start breaking people even more often, than we should. E.g. if someone writes a tool targeted at 1.1, and we deprecated 1.1 in M cycle, the tool will break, though maybe it can actually work with new API. I do not understand how in tree testing is not feasible. In tree you have insights into all the branching that occurs within code so can very clearly understand what paths aren't possible. It should be a lot more straight forward than external black box testing where that can't be assume. Exactly. The whole *point* of microversions was to allow the APIs to evolve in a backwards-compatible, structured and advertised way. The evolution of the APIs response and request payloads should be tested fully for each microversion added to the codebase -- in tree. -jay
Re: [openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
On 04/07/2015 10:23 PM, Ken'ichi Ohmichi wrote: Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. snip Any thoughts? Makes sense, I think. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][nova][ironic] How to run microversions tests on the gate
Hi, Now Nova and Ironic have implemented API microversions in Kilo. Nova's microversions are v2.1 - v2.3. Ironic's microversions are v1.1 - v1.6. Now Tempest is testing the lowest microversion on the gate, and Ironic's microversions test patch[1] is on the gerrit. Before merging the patch, I'd like to propose consistent test way for microversions of Nova and Ironic. My suggestion is the test target microversions are: * the lowest microversion * the biggest microversion, but don't use the keyword latest on a header and these microversions tests are operated on different gate jobs. The lowest microversion is already tested on check-tempest-dsvm-full or something, so this proposes just to add the biggest microversion job like check-tempest-dsvm-full-big-microversion. [background] In long-term, these microversions continue increasing and it is difficult to run Tempest for all microversions on the gate because of test workload. So I feel we need to select microversions which are tested on the gate for efficient testing. [the lowest microversion] On microversion mechanism, if a client *doesn't* specify favorite microversion in its request header, a Nova/Ironic server considers the request as the lowest microversion. So the lowest microversion is default behavior and important. I think we need to test it at least. [the biggest microversion] On microversion mechanism, if a client specify the keyword latest in its request header instead of microversion, a Nova/Ironic server works on the biggest microversion behavior. During the development, there is time lag between each project dev and Tempest dev. After adding a new API on a project, corresponding tests are added to Tempest in most cases. So if specifying the keyword latest, Tempest would not handle the request/response and fail, because Tempest can not catch the latest API changes until corresponding Tempest patch is merged. So it is necessary to have the target microversion config option in Tempest and pass specific biggest microversion to Tempest with openstack-infra/project-config. Any thoughts? Thanks Ken Ohmichi --- [1]: https://review.openstack.org/#/c/166386/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
