On 06/17/2015 06:54 AM, Ken'ichi Ohmichi wrote:
2015-06-17 12:38 GMT+09:00 Yuiko Takada <yuikotakada0...@gmail.com>:
Then, as you and Matt and Dimitry talked about this on IRC few days ago,
We can add Ironic/Ironic-inspector tests into Tempest still, right?
So that I've started to implement a test in Tempest,
but I'm facing another issue.
As you know, Ironic API has microversions, and Ironic-inspector can run
with microversion > 1.6.
But currently there is no feature testing specific Ironic API microversions
on Tempest, right?
So that, we have to think about some solutions.
(1) Make testing specific Ironic API microversions on Tempest possible
adam_g is posting this patch set.
https://review.openstack.org/166386
(2)Using tempest_lib instead of adding tests into Tempest
Is tempest_lib available already?
Or do we need to wait for something will be merged?
I guess the above question seems multiple factors are mixed.
You want to test ironic-inspector behaviors by
* using ironic-inspector REST APIs directly without Ironic
* using Ironic REST APIs which need newer microversion
right?
Hi, thanks for clarifying, let me jump in :)
The former is more or less covered by functional testing, so I'd like us
to concentrate on the latter, and run it voting on inspector repo and
non-voting on Ironic for the time being.
For the first test, you can implement without considering microversion.
The test just calls ironic-inspector REST APIs directly and checks its behavior.
You can implement the test on Tempest/ironic-inspector repository.
Current tempest-lib seems enough to implement tests in
ironic-inspector repository as features, but it is better to wait for
Tempest's external interface spec[1] approval.
It is trying to define directory structure of Tempest-like tests on
each project repository and Tempest will discover tests based on the
directory structure and run them.
So if implementing tests on ironic-inspector repository before the
spec approval, you will need to change the directory structure again
in the future.
This "wait" part bothers me to some extend, because gate absence badly
affects us for some time, but fine. Thanks for heads up anyway.
For the second test, microversions support is necessary on Tempest
side and adam_g's patch seems good for implementing it.
My main concern of microversions tests is how to run multiple
microversions on the gate.
We have discussed that in Nova design session of Vancouver summit and
the conclusion is
* Minimum microversion
* Maximum microversion
* Interesting microversions
as the gate test.
Facepalm. That's what I was talking about (and what we actually ended up
in Ironic with): we're introducing a ton of non-tested (and thus
presumably broken) microversions, because it's cool to do. Ok, that's
another thread :)
IMO "Interesting microversions" would be the last microversions of
each release(Kilo, Liberty, ..) I feel.
With Ironic intermediate release, it will be more, I estimate it to be
5-6 per year, but of course I can't tell for sure.
I have qa-spec[2] for testing microversions on the gate, but that is
not complete yet.
That will affect how to specify/run microversions tests on Tempest.
So now I'm not sure yet the way to specify microversion on current
adam_g's patch is the best.
So my recommendation/hope is that we concentrate on Tempest's external
interface spec[1] and make it better together, then we can implement
Tempest-like tests on each repository after that.
As the next step, we will test microversions on the same way between
projects based on conclusion of the spec[2].
What I'd prefer us to start with is a gate test, which just sets
devstack with our plugin and runs a shell script testing a couple of
basic things. This will be a HUGE leap forward for inspector, compared
to only limited functional testing we have now.
So maybe we should start with it, and keep an eye on the tempest-lib
stuff, wdyt?
(3)Make Ironic-inspector available even if microversion < 1.6
Dmitry is posting this patch set.
https://review.openstack.org/192196
# I don't mean asking you to review this, don't worry :p
I've reviewed it already :)
Thanks
Ken Ohmichi
---
[1]: https://review.openstack.org/#/c/184992/
[2]: https://review.openstack.org/#/c/169126/
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
