On 08/03/18 19:31, Doug Hellmann wrote:
If these new plugins might contain "candidate" tests and all tests
are potentially candidates, how are these new repos different from
the existing repos that already contain all of the tests?

We have to allow candidate tests because otherwise there's a chicken-and-egg type problem where we can never add tests to an existing trademark program. But the definition of 'candidate' that I'm thinking of is along the lines of "under active discussion to be added to the next iteration of refstack". Perhaps that should be documented.

To answer your question directly, the main way they're different is that by putting tests in one of these repos, teams are committing to not really change them, and certainly to ~never break backwards compat.

It seems
like at least part of the problem with the current system was
triggered by confusion about when to move tests around to satisfy
the policy. Can we avoid that problem with the new system? If we're
not going to move the tests into Tempest itself and have the QA
team manage them, why not simply take the tests from the repos where
they already live?

I think a lesson we've learned from Tempest is that there are two parts to successfully reviewing a change:

1) Determine whether the change affects a test that is part of the trademark program.
2) Don't make the change if it does.

2 is easy, but 1 is hard. By separating the trademark tests into a separate repo, we make 1 easy as well. This reduces risk and increases review throughput.

I thought the QA team no longer wanted to be responsible for these
extra tests. Has that changed again? I've lost track of everyone's
positions, I'm afraid.

Their position as I understand it is:

* They're not going to write tests
* They're happy to document the process, offer advice, and review tests as time allows * They don't want tests to be thrown over the transom and made their problem for the rest of time

That doesn't conflict with anybody's goals here, so it's mostly a matter of documenting it clearly so that somebody reading without all this context won't get the wrong idea.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Reply via email to