Stepan Mishura wrote:
Hi folks,
I've moved 2.0 build-and-test's branch to the trunk - the branch was
used during M2-M5 testing and IMO it proved to be very useful (also I
tagged the former trunk as 1.0.)
Hurray!
I think that this is a good time to collect ideas and hear opinions
what should be improved and possible directions for test infra
development.
I have to admit that the current testing infra is far from being
perfect – there a lot for work to make is more usable, cleaning up
current scripts, sort out all know issues with tests/scenarios and so
on. But I'd like to address problems I believe of different level. I
have an impression that Harmony tests suites are still in shadow (IOW,
IMHO they are undervalued) of testing automation (i.e. CruiseControl
and support scripts) – and most of folks prefer to contribute to the
code development. And I believe that it is possible to make tests
development also interesting too. IMO the missed point that most of
Harmony test suites can be used as stand alone suites to verify/test
different implementations – in other words, the suites are not tightly
coupled with DRL VM, for example, it is possible to run them against
IBM VME (BTW, may it be makes sense to run full set of suites against
IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
I'm going to update the test infra home page [1] with info related to
the renewed trunk. Also I think the page should contain some words
about project's testing strategy: i.e. how Harmony QA and release
process are organized - integrity/snapshot testing, milestones, code
freeze and etc., what suites are there (their description and
purpose). I think this may encourage people to invest efforts into
developing tests.
Other ideas and thoughts?
Thanks for your investment in the test suites, and taking time to think
about the improvements.
The thing I struggle with is understanding the current system well
enough to make a valuable contribution.
It would be really good to have a newbies' guide to the test suites
document on the website with step-by-step instructions (like the quick
help for contributors document).
While the architecture documentation would be interesting, I think many
people work by running the simple case, then get dragged into
investigating the complex cases when their needs require it.
Regards,
Tim
[1] http://harmony.apache.org/subcomponents/buildtest/index.html
Thanks,
Stepan.
