On Mar 4, 2009, at 8:03 PM, Adam Heath wrote:
David E Jones wrote:
Why not just use different test-suite elements for this? Why have an
hierarchy in this?
Actually... I'm not even sure what "this" is, ie what is the point
of a
group? Maybe we should discuss that first...
The service engine tests define several different test cases. The
second test case does a data load. Then each subsequent test modifies
the data state of the previous test. They all have to run in
sequence, for them to pass.
This means that they are *not* separate test cases, but a single test
case. A test case is supposed to be self-contained, not depending on
the state of anything else. It's supposed to start from empty, do
whatever mutative changes it needs to, without any external
dependencies.
Who says?
Why?
The change you made doesn't change this, it just creates something new
called a "test group" that I'm guessing is supposed to group tests
together than depend on eachother. However, it doesn't make the test
cases independent.
Also, why introduce a test group concept when we already have the test-
suite concept? Why not just use that? Run one suite at a time and only
allow dependencies within that suite.
Still, this hasn't been discussed. The only thing we've discussed and
somewhat decided on so far (including the ApacheCon discussions I was
involved in, which was the same as what was discussed before) is to
only allow dependencies between test cases within components.
We could restrict that more by only allowing dependencies between test
cases within a suite (which I think is the more normal practice), and
the terms "test case" and "test suite" are more known as those are
used in JUnit and other things.
The service test cases did not follow this.
Of course not. It's never been discussed or defined or established
even as a recommended practice, let alone one we'll adopt... possibly
because it doesn't make much sense?
Backing up a little bit, you can make the assertion that each test
case should be 100% independent, but that is only the beginning of the
discussion, not the end of it. We do need to decide on what
dependencies will be allowed, but I'm pretty sure won't end up with an
agreed on answer of "no dependencies period". That leads to excessive
redundancy.
I guess one thing to keep in mind is that while we use JUnit
underneath we're actually talking about something higher level than
unit tests. These are general test cases and test suites, not
necessarily just unit tests as those are really of limited utility. In
fact, we're mostly testing processes with many steps and in the case
of the applications tests these are meant to call a series of services
that would normally be called through the UI, but we're dropping down
to the service layer for them to test on that level.
I've got a script that runs each defined test case separately; it
stops/starts ofbiz for each individual test case, and restores a
previous saved copy of the derby data folder between each test.
Yep, I saw that, looks cool.
Either way, unless there is a good reason to have these test-groups,
let's just stick with test-suite. I still haven't seen anything you've
written about what the test-group will be used for, so at least that
would be helpful.
-David
