K, so.... On 28 February 2011 11:27, Peter <[email protected]> wrote:
> > I'm not yet ready to buy modules as being a decent divide point for > tests. > > Tests are typically associated with packages, not modules IMHO. > > > > Feels like an impedance mismatch or maybe I'm not liking how much the > > build/deploy/package mechanism is a factor in a discussion about testing. > > > Currently we have: > > Unit tests: junit > Integration tests: qa test harness. > Jini platform compliance tests: qa test harness > Regression tests: jtreg > > On top of that we've got the discovery and join test kit. > > We're very lucky to have such a vast array of tests. > > We are indeed. > It would be nice if I could just work on a small part of what is River, > improve it using test driven development by only running the tests needed. > Yep - that's a requirement for sure. But the tests you run for that probably don't include all the tests available. Probably just unit and service level or similar. Probably not performance, maybe some compliance stuff etc. > > It seems that you must first build before you can test, sometimes it is > wise to find the right questions before looking for the answer. > > Well, yes you must first build but the tests you run aren't always directly connected to that build. I might: (1) Be just building some module which maps closely to some service and thus want to run service-specific tests. (2) Be running some set of performance tests against nitty-gritty implementation details within a single module. (3) Be running some global set of tests across a number of services, only one of which I've changed. Sounds strange right? Nope, if I'm halfway through making a global change and I've updated some of the services, I want to see how I'm doing and if I broke anything non obvious. > We don't build junit or jtreg every time we test, why can't the qa harness > be separate too? > > It can, but separation is merely a small element of the overall problem. Some tests span, some tests run against APIs, some run against nitty-gritty implementation bits. In essence, tests don't always nicely separate with modules IMHO, sometimes they do, sometimes not. > Why can't a component such as Outrigger be kept separate from the jini > platform build? Why can't it have the tests that must be run to test it be > known in advance, or compiled in advance for that matter? > > Why isn't it easy to determine the tests that need to be run to test a > component? > We're not always talking a component IMHO. Or at least, not a component that is relevant at the build system level and herein lies the problem. > > Is there a better solution? > > Dunno, but I'm thinking a clean separation of build/module/packaging from test might be the way to go. Testing lifecycle and the things it touches are unrelated to modules and such the build system yields and we want to account for that. > If not I'll have to face facts: my development windows of opportunity don't > allow enough time to adequately test changes. Maybe modularity is not the > answer, maybe I'm grasping at straws. I don't know, these things take time > to figure out. > > Can anyone else see the problems? > > Anyone here know the right questions? > What are all our testing scenarios and their associated lifecycles (when are they run and against what)? > > With the right questions, answers are easier to find.
