*** First off Adam: it's great to see you working on this stuff, the
testing stuff in general needs a LOT more work and hasn't seen much
attention, so yeah... this is great.
On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:
The service engine tests define several different test cases. The
second test case does a data load. Then each subsequent test
modifies
the data state of the previous test. They all have to run in
sequence, for them to pass.
This means that they are *not* separate test cases, but a single
test
case. A test case is supposed to be self-contained, not depending
on
the state of anything else. It's supposed to start from empty, do
whatever mutative changes it needs to, without any external
dependencies.
Who says?
Why?
The change you made doesn't change this, it just creates something
new
called a "test group" that I'm guessing is supposed to group tests
together than depend on eachother. However, it doesn't make the test
cases independent.
Also, why introduce a test group concept when we already have the
test-suite concept? Why not just use that? Run one suite at a time
and
only allow dependencies within that suite.
Well, TestRunContainer supports running a singly-defined test case
from some testdef.xml file. If done against service tests, then
you'll get false positives.
I'm fine with not having this <test-group>, so long as we don't allow
for running separate tests, that TestRunContainer currently allows.
Currently, we can run all tests suites inside a component, or one
separate test in a component, in whatever test suite it resides in.
We have no way to restrict the lookup code to just a single test
suite.
What does all of this have to do with it?
Are you just saying that we need a way to run one test suite at a
time? If we're going to move toward the practice of having
dependencies only within a single test suite, then yes, I'd agree this
is necessary.
At the minute the practice is to allow dependencies within a component.
Still, this hasn't been discussed. The only thing we've discussed and
somewhat decided on so far (including the ApacheCon discussions I was
involved in, which was the same as what was discussed before) is to
only
allow dependencies between test cases within components.
Well, in some cases, multiple tests defined inside some test-suite.xml
can *not* run in series, or in parallel. Each defined test case needs
a *clean* database. In other cases, the test cases inside the
test-suite.xml *do* require running each in series.
Oh yeah? Where?
Here's the scary thing that I'm starting to believe to be the case:
because you didn't read and research before starting to change things
you didn't try running the tests for a component all at once. You made
incorrect assumptions about how they were written and how they were
meant to be run.
Your rants and complaints about the tests may be incorrect, and the
"fixes" you've been making for them may be unnecessary. They are only
true if you adopt your assertion that all test cases should be
independent. Otherwise, they are not true and you're heading down a
road that others may not appreciate.
Could we maybe step back and talk about this before you cruise along
too much more?
-David
