David E Jones wrote: > *** First off Adam: it's great to see you working on this stuff, the > testing stuff in general needs a LOT more work and hasn't seen much > attention, so yeah... this is great.
Completely automatic tests with no side-effects is what I am striving for. I believe we are close. > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: > >>>> The service engine tests define several different test cases. The >>>> second test case does a data load. Then each subsequent test modifies >>>> the data state of the previous test. They all have to run in >>>> sequence, for them to pass. >>>> >>>> This means that they are *not* separate test cases, but a single test >>>> case. A test case is supposed to be self-contained, not depending on >>>> the state of anything else. It's supposed to start from empty, do >>>> whatever mutative changes it needs to, without any external >>>> dependencies. >>> >>> Who says? >>> >>> Why? >>> >>> The change you made doesn't change this, it just creates something new >>> called a "test group" that I'm guessing is supposed to group tests >>> together than depend on eachother. However, it doesn't make the test >>> cases independent. >>> >>> Also, why introduce a test group concept when we already have the >>> test-suite concept? Why not just use that? Run one suite at a time and >>> only allow dependencies within that suite. >> >> Well, TestRunContainer supports running a singly-defined test case >> from some testdef.xml file. If done against service tests, then >> you'll get false positives. >> >> I'm fine with not having this <test-group>, so long as we don't allow >> for running separate tests, that TestRunContainer currently allows. >> >> Currently, we can run all tests suites inside a component, or one >> separate test in a component, in whatever test suite it resides in. >> We have no way to restrict the lookup code to just a single test suite. > > What does all of this have to do with it? > > Are you just saying that we need a way to run one test suite at a time? > If we're going to move toward the practice of having dependencies only > within a single test suite, then yes, I'd agree this is necessary. > > At the minute the practice is to allow dependencies within a component. > >>> Still, this hasn't been discussed. The only thing we've discussed and >>> somewhat decided on so far (including the ApacheCon discussions I was >>> involved in, which was the same as what was discussed before) is to only >>> allow dependencies between test cases within components. >> >> Well, in some cases, multiple tests defined inside some test-suite.xml >> can *not* run in series, or in parallel. Each defined test case needs >> a *clean* database. In other cases, the test cases inside the >> test-suite.xml *do* require running each in series. > > Oh yeah? Where? > > Here's the scary thing that I'm starting to believe to be the case: > because you didn't read and research before starting to change things > you didn't try running the tests for a component all at once. You made > incorrect assumptions about how they were written and how they were > meant to be run. But I did do some. The service tests require running everything in the test suite. That I am absolutely certain of. The other thing we know, is that running some tests together causes problems. I'm not entirely certain if these same bad tests together exist in a single test suite, or if different test suites conflict. > Your rants and complaints about the tests may be incorrect, and the > "fixes" you've been making for them may be unnecessary. They are only > true if you adopt your assertion that all test cases should be > independent. Otherwise, they are not true and you're heading down a road > that others may not appreciate. > > Could we maybe step back and talk about this before you cruise along too > much more? I still stand by the <test-group> addition; it's a new feature, that seems usefull. It allows combining separately defined test cases into a single whole, without giving them each a separate name. It allows for code reuse, without having to write some java code, or a simple-method, to combine them. However, what we are heading towards is making the granularity be a test suite. In that case, the service test definition should be changed back, as well as changing my recent build.xml and TestListContainer policy. Just to restate, I will change the policy to be each suite gets run separately.
