On Thu, 1 Jun 2017, Matthew Treinish wrote:
On Thu, Jun 01, 2017 at 11:09:56AM +0100, Chris Dent wrote:
A lot of this results, in part, from there being no single guiding
pattern and principle for how (and where) the tests are to be
managed.
It sounds like you want to write a general testing guide for openstack.
Have you started this effort anywhere? I don't think anyone would be opposed
to starting a document for that, it seems like a reasonable thing to have.
But, I think you'll find there is not a one size fits all solution though,
because every project has their own requirements and needs for testing.
No, I haven't made any decisions about what ought to happen. I'm
still trying to figure out if there is a problem, a suite of
problems, or everything is great. Knowing what the problems are
tends to be a reasonable thing to do before proposing or
implementing solutions, especially if we want those solutions to be
most correct.
So have you read the documentation:
https://docs.openstack.org/developer/tempest/ (or any of the other relevant
documentation
and filed bugs about where you think there are gaps? This is something that
really bugs me sometimes (yes the pun is intended) just like anything else this
is all about iterative improvements. These broad trends are things tempest
and (every project hopefully) have been working on. But improvements don't
just magically occur overnight it takes time to implement them.
This is a huge part of the colllaboration issues I was identifying
in my previous message. Somebody says "there seems to be some
confusion here" and somebody else comes along and asks "have you
filed bugs?" or "have you proposed a solution?".
Well, "no" because like I said above I don't know what (or even _if_)
there's something to fix or the relevant foundations of the confusion.
I have some suspicions or concerns that the implicit hierarchy of
some tempests tests being in plugins and some not creates issues
with discovery, management and identification of responsible parties
and _may_ imply a lack of "level playing field".
But:
* if other people don't have those concerns it's not worth
pursuing
* until we reach some kind of shared understanding and agreement
about the concerns, speculating about solutions is premature
Just compare the state of the documentation and tooling from 2 years ago (when
tempest started adding the plugin interface) to today. Things have steadily
improved over time and the situation now is much better. This will continue and
in the future things will get even better.
Yes, it's great. If you feel like I was suggesting otherwise, then
my apologies for not being clear. As a general rule tempest and
other QA tools have consistently done great work in terms of
documentation and tooling. That there are plugins at all is
fantastic; that we are having discussions about how to make the most
effective and fair use of them is a sign that they work.
--
Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/
freenode: cdent tw: @anticdent
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
