Geir, I've tried to put this proposal to the site. Please see the patch at http://issues.apache.org/jira/browse/HARMONY-511 - hope I did it correctly.
-- Regards, Anton Luht, Intel Middleware Products Division On 5/23/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote:
How about a patch for the website, including instructions on what to do? Fedotov, Alexei A wrote: > Alex, > >> Please, find the new version (0.2) and all previous versions here: >> http://issues.apache.org/jira/secure/ManageAttachments.jspa?id=12340105 > > Good. I like it (everyone likes when his suggestions are implemented > :-). So let me try to outline where we are in Harmony stress testing. > > === TEST DESIGN === > > * Stress tests are built from simple building blocks according to > configuration strings. > > * Tests have junit interface. > [Case study] Imagine someone puts tests into SVN which implements > different test interface. To reuse them we can add another generator to > convert these tests to junit interface. > > * Configuration string list is maintained manually. If we plan to use > junit runner to launch a sequent of the stress tests, then the most > straightforward model is to wrap configuration strings into junit test > cases and put documentation into javadoc for these test cases. > > === FURTHER STEPS === > > * You wrote, "stress test suite should generate relevant bugs". Since > usually stress behavior is unspecified, we need to introduce something > measurable instead of pass/fail result for the stress tests. See my > thoughts about a comparative approach below. > > * I will continue code reviews. > > * All should create tests and run them against Harmony VM and RI. > This would be a real-life testing for our approach. > > === COMPARATIVE APPROACH === > > The simplest example of comparative apporach is the following. > Tester: My test fails on Harmony VM and passes on RI. Please, > fix Harmony VM. > > This usually does not work for stress tests. > Developer: Who told you that OutOfMemoryError should be thrown > in your thread? My finalizer thread is just a normal java thread, like > yours, and it can fail as well. You have a bug in your test. > > There are multiple reasons why we always will have such bugs in the > tests. > * These bugs keep showing up. The time to fix all these bugs > regularly is too high. > * Stress testing reuses tests which are usually not designed for > stress execution, for example, multithread execution. > * These bugs are dependent on VM internal structure. Test authors do > not posess sufficient knowledge of the problem and the structure. > * Sometimes Java is not rich enough. > > How can we have a maintainable test product takung all this limitation > into account? We need to learn how to live with occasional failures of > the stress tests. This means, instead of fail, the test should better > report how good it is on Harmony VM compared to RI: > * Failures with the worst relative metric can be evaluated first. > * We can detect that a relative metric for a test worsened on the > recent build. > > Developers are better convinsed to fix "the worst issue" or > "dergadation" instead of "some issue". > > Now let me list here several metrics for each test. > * Pass rate: assuming the test is 100% reliable on RI we can > calculate a percentage of failures. > * Number of times the test can be executed sequentionally before a > fail. > * Memory consumption: a generator can preallocate more and more > memory before launching the test in a loop. > * Max threads supported: a generator can exponentially increase > number of threads launching the test in parallel. > * Here is your metric. > * Execution time: have you noticed that all this apparatus is quite > close to performance testing methodology? There is no need to compete > with them in their field though. :-) > > The thing I like most about this approach is it can be introduced on a > stress test generator level. > > With best regards, > Alexei Fedotov, > Intel Middleware Products Division > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
