On 3/5/08, Tim Ellison <[EMAIL PROTECTED]> wrote: > Stepan Mishura wrote: > > Hi folks, > > > > I've moved 2.0 build-and-test's branch to the trunk - the branch was > > used during M2-M5 testing and IMO it proved to be very useful (also I > > tagged the former trunk as 1.0.) > > Hurray! > > > I think that this is a good time to collect ideas and hear opinions > > what should be improved and possible directions for test infra > > development. > > > > I have to admit that the current testing infra is far from being > > perfect – there a lot for work to make is more usable, cleaning up > > current scripts, sort out all know issues with tests/scenarios and so > > on. But I'd like to address problems I believe of different level. I > > have an impression that Harmony tests suites are still in shadow (IOW, > > IMHO they are undervalued) of testing automation (i.e. CruiseControl > > and support scripts) – and most of folks prefer to contribute to the > > code development. And I believe that it is possible to make tests > > development also interesting too. IMO the missed point that most of > > Harmony test suites can be used as stand alone suites to verify/test > > different implementations – in other words, the suites are not tightly > > coupled with DRL VM, for example, it is possible to run them against > > IBM VME (BTW, may it be makes sense to run full set of suites against > > IBM VME for each milestone candidate as for DRL VM. Any volunteers?) > > > > I'm going to update the test infra home page [1] with info related to > > the renewed trunk. Also I think the page should contain some words > > about project's testing strategy: i.e. how Harmony QA and release > > process are organized - integrity/snapshot testing, milestones, code > > freeze and etc., what suites are there (their description and > > purpose). I think this may encourage people to invest efforts into > > developing tests. > > > > Other ideas and thoughts? > > Thanks for your investment in the test suites, and taking time to think > about the improvements. > > The thing I struggle with is understanding the current system well > enough to make a valuable contribution. >
Yes, to automate the suite runs you have to study how a whole system (i.e. BTI) works. And my point was that you don't need to understand BTI to contribute to a particular suite. You should only grab the suite and start running it. This is true for classlib unit tests, drlvm suites ... , isn't it? But the same applies for the rest of them: Geronimo unit tests, Eclipse unit tests, functional suite, VTS VM suite and so on. > It would be really good to have a newbies' guide to the test suites > document on the website with step-by-step instructions (like the quick > help for contributors document). > Sure, I'm going to improve the docs. Thanks, Stepan. > While the architecture documentation would be interesting, I think many > people work by running the simple case, then get dragged into > investigating the complex cases when their needs require it. > > Regards, > Tim > > > [1] http://harmony.apache.org/subcomponents/buildtest/index.html > > > > Thanks, > > Stepan. > > >
