"build test" is necessary but not sufficient for where we need to go. In
a sense, committing code with only "build test" to prevent regression is the equivalent to flying in the fog without instrumentation.
Right. Here I see 2 aspects - creating VM/JIT regression tests and running various tests regularaly / tracking regressions timely. 1. VM/JIT regression test development. We can start with these guidelines: Who creates/integrates a regression test? - committer. How? - typically, from bug's reproducer code. Where? - drlvm\trunk\src\test\vm\<component>\harmony-XXXX\ or drlvm\trunk\src\test\jit\<component>\harmony-XXXX\ Format: Junit, Jasmin When? - Consider creating a regression test for each fixed bug against Harmony DRLVM or JIT, if reasonable and technically possible. Creation of regression test may be omitted if: - Bug in documentation, build, test, code comment, code style, enhancement is fixed. - Performance-related bug is fixed. - Bug is found by existing Harmony test (Harmony regression or unit test). // to avoid duplication of tests Try to use public API to provide implementation independence and stability of the tests. This test suite can/should be used in pre-commit testing. 2. Timely regression tracking via test execution. Now we have some solution for code integrity testing via Cruise Control (buildtest/trunk/CC + HARMONY-995) - good start point to track regressions hourly using classlib unit tests and "build test" on your specific platform. I see HARMONY-2038 which is about automation of Ecilpse Unit Tests execution in context of buildtest/ infrastructure - to start running EUT regularly using buildtest/ (say, nightly) and reporting timely about new test failures (and causing commits). I think, it would be great if we continue adding more automation scripts into buildtest/ for various public unit test suites and scenarios (Derby, Tomcat, etc.), so that one could take automation scripts from buildtest/ and run them regularly on his specific platform, report regressions timely (what Nina, I suppose, is going to do with EUT). On 11/4/06, Weldon Washburn <[EMAIL PROTECTED]> wrote:
Folks, I have spent the last two months committing patches to the VM. While we have added a ton of much needed functionality, the stability of the system has been ignored. By chance, I looked at thread synchronization design problems this week. Its very apparent that we lack the regression testing to really find threading bugs, test the fixes and test against regression. No doubt there are similar problems in other VM subsystems. "build test" is necessary but not sufficient for where we need to go. In a sense, committing code with only "build test" to prevent regression is the equivalent to flying in the fog without instrumentation. So that we can get engineers focused on stability, I am thinking of coding the JIRAs that involve new features as "later" or even "won't fix". Please feel free to comment. We also need to restart the old email threads on regression tests. For example, we need some sort of automated test script that runs Eclipse and tomcat, etc. in a deterministic fashion so that we can compare test results. It does not have to be perfect for starts, just repeatable and easy to use. Feel free to beat me to starting these threads :) -- Weldon Washburn Intel Enterprise Solutions Software Division
