Thomas Zander wrote: >>>Huh? Why is adding broken tests the right thing to do? And besides, >>>if a broken test is added, this way there will be motivation to >>>resolve the discrepancy. With a whitelist, a broken test can get >>>added but no one will notice and then it just sits there getting >>>stale. >> >>Its common practise to add new code to one implementation, e.g GNU >>classpath or libgcj, and test it for a while and later merge it to >>kaffe. According to you the mauve tests don't need to be added before >>it's included in all implementations because nothing may be broken. > > > Ehm; just being a bystander; a broken test in Archies email is a test that > does not work properly (harness.check(1 ==2)). > A broken test we are talking about, and what Michael seems to imply; is a > test that is fully correct, and will (probably) run correctly on Suns JVM, > but fails on another. > > Lets call the former a broken, and the latter a failing test, please :) > > Mauve is not suppost to hold _any_ broken tests, right?
Thanks, I was misreading the original comment. We all agree Mauve may contain "failing" tests but should never contain "broken" tests. Now, back to the original point... I've made a proposal for cleaning up the Mauve mess. For those who don't like it, please explain your proposal and most importantly exactly how this "whitelist" (or whatver) will be maintained. Who is going to do the maintenance work? When are they going to do it? Etc. I don't really care how we do it, but these seem to be reasonable requirements. Maybe we should try to agree on these first: - It should be possible to test any JVM using some "official" set of tests which a Classpath JVM should pass and see clearly and obviously any failures; a "perfect" JVM would have no failures. - When a new test is added to Mauve, by default it is automatically added to the set of Classpath tests. I.e., it's not possible for newly added tests to not actually run against Classpath without explicitly configuring things that way. The point is that every test has a known state, one of: (a) don't even bother trying it (doesn't compile or we know it fails) (b) it is a known failure (i.e., a Classpath bug) (c) it should pass (i.e., if it fails, it must be a JVM bug) (d) it should pass but there can be false negatives depending on the JVM and most importantly the maintenance of these states is simle enough that we don't lose track and get back into the mess we're in now. By the way, saying "fix ./batch_run to respect xfails" or "fix the normal mauve run to not ignore tests that don't compile" is fair game (as long as you are willing to do the work :-) -Archie __________________________________________________________________________ Archie Cobbs * CTO, Awarix * http://www.awarix.com * Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. * _______________________________________________ Classpath mailing list [email protected] http://lists.gnu.org/mailman/listinfo/classpath
