I think (d) is mis-characterized. If a test fails but it isn't a VM or Classpath bug, then >>surely<< it must be a fault in the testcase and/or the Mauve framework. Please let's be clear about this.
For example:
1) Tests for behavior that is beyond the scope of "the standard".
For example, testcases that test the names of classes returned by certain factory methods, or that test exceptions message strings
are fundamentally broken.
2) Tests that have been designed to match (Sun acknowledged) buggy JDK behavior or buggy specs / javadocs.
3) Tests that assume some VM-specific functionality or behavior. For example, almost any test for GC-related behavior is inevitably VM-specific, since there is no VM-independent way to force the GC to run.
4) Tests that depend on environmental conditions. For example,
some existing tests assume that the host machine can open socket connections to random machines/ports on other network. These
break for me because I am behind a firewall.
IMO, all of the above are actually bugs in the Mauve testcases, and/or problems in the way that they are organized / classified.
Yep, that's a good classification. My point is simply that even with these problems, the tests are still useful, and people want to run them, so we need some way of handling them.
-Archie
__________________________________________________________________________ Archie Cobbs * CTO, Awarix * http://www.awarix.com
_______________________________________________ Classpath mailing list [email protected] http://lists.gnu.org/mailman/listinfo/classpath
