On 8/7/06, Alexey Varlamov <[EMAIL PROTECTED]> wrote:
And you forgot to mention cases when tested VM hangs :)
Still, almost any of those situations may be tested automatically,
just fork a tested JVM and setup timeout watchdog. You may examine
1) return value (via j.lProcess.exitValue());
2) out/err streams of the forked process (j.lProcess.getErrorStream()
& j.lProcess.getInputStream ());
Though I'm unsure this is readily possible with Ant+junit, likely more
functional harness is needed.
I thought about testing using exec() and testing using jasmin-written tests
and management of tests via metadata and here what came into my mind as
requirements to the harness which would solve current and future issues we
will face as we have more and more tests of various types:
General
· Multi-platform (Java-written)
· Easy to install, configure and run tests.
· Fully automated test execution (batch mode with no user interaction)
· Support of two modes – each test in separate VM, all tests in one VM.
- Each test in separate VM - to avoid test influence or VM crash
should not lead to test run stop.
- All tests in one VM – useful mode to test runtime for
stability/reliability and allows reducing test suite execution time.
· Be independent of test directory structure or test naming conventions.
Managing test execution
· Selection of tests for execution:
- By some property (platform-specific tests only, Harmony
implementation tests only, do not run buggy tests)
- By sub-directory and/or using list of tests to execute and/or
fitting test name pattern.
· Harness should control test execution timeout & kill test process when
timeout expires.
· Unexpected abnormal test/runtime completion does not stop test
execution.
· Test run environment should be configurable (best – all settings are in
one place – in some configuration file, for example)
Logging and reporting
· Harness should provide good reporting capabilities – easy to understand
and navigate.
Do we have in mind test harness which meets majority of requirements?
Thanks, Vladimir
--
Regards,
Alexey
[snip]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
