David E Jones wrote: > > On Mar 4, 2009, at 8:03 PM, Adam Heath wrote: > >> David E Jones wrote: >>> >>> Why not just use different test-suite elements for this? Why have an >>> hierarchy in this? >>> >>> Actually... I'm not even sure what "this" is, ie what is the point of a >>> group? Maybe we should discuss that first... >> >> The service engine tests define several different test cases. The >> second test case does a data load. Then each subsequent test modifies >> the data state of the previous test. They all have to run in >> sequence, for them to pass. >> >> This means that they are *not* separate test cases, but a single test >> case. A test case is supposed to be self-contained, not depending on >> the state of anything else. It's supposed to start from empty, do >> whatever mutative changes it needs to, without any external >> dependencies. > > Who says? > > Why? > > The change you made doesn't change this, it just creates something new > called a "test group" that I'm guessing is supposed to group tests > together than depend on eachother. However, it doesn't make the test > cases independent. > > Also, why introduce a test group concept when we already have the > test-suite concept? Why not just use that? Run one suite at a time and > only allow dependencies within that suite.
Well, TestRunContainer supports running a singly-defined test case from some testdef.xml file. If done against service tests, then you'll get false positives. I'm fine with not having this <test-group>, so long as we don't allow for running separate tests, that TestRunContainer currently allows. Currently, we can run all tests suites inside a component, or one separate test in a component, in whatever test suite it resides in. We have no way to restrict the lookup code to just a single test suite. > Still, this hasn't been discussed. The only thing we've discussed and > somewhat decided on so far (including the ApacheCon discussions I was > involved in, which was the same as what was discussed before) is to only > allow dependencies between test cases within components. Well, in some cases, multiple tests defined inside some test-suite.xml can *not* run in series, or in parallel. Each defined test case needs a *clean* database. In other cases, the test cases inside the test-suite.xml *do* require running each in series. > We could restrict that more by only allowing dependencies between test > cases within a suite (which I think is the more normal practice), and > the terms "test case" and "test suite" are more known as those are used > in JUnit and other things. > >> The service test cases did not follow this. > > Of course not. It's never been discussed or defined or established even > as a recommended practice, let alone one we'll adopt... possibly because > it doesn't make much sense? > > Backing up a little bit, you can make the assertion that each test case > should be 100% independent, but that is only the beginning of the > discussion, not the end of it. We do need to decide on what dependencies > will be allowed, but I'm pretty sure won't end up with an agreed on > answer of "no dependencies period". That leads to excessive redundancy. > > I guess one thing to keep in mind is that while we use JUnit underneath > we're actually talking about something higher level than unit tests. > These are general test cases and test suites, not necessarily just unit > tests as those are really of limited utility. In fact, we're mostly > testing processes with many steps and in the case of the applications > tests these are meant to call a series of services that would normally > be called through the UI, but we're dropping down to the service layer > for them to test on that level. You and I are on the same wave length. >> I've got a script that runs each defined test case separately; it >> stops/starts ofbiz for each individual test case, and restores a >> previous saved copy of the derby data folder between each test. > > Yep, I saw that, looks cool. > > Either way, unless there is a good reason to have these test-groups, > let's just stick with test-suite. I still haven't seen anything you've > written about what the test-group will be used for, so at least that > would be helpful. I had a later commit that changed the service tests over to it; however, that was just an automatic-type change, so that my script wouldn't break, by trying to run each one separately.
