Archie Cobbs wrote: > Jeroen Frijters wrote: > >>The problem with that approach is that if someone adds a new test > >>to Mauve, it doesn't automatically get added to our "white list". > > > > That's a feature, not a bug! In practice new tests often > get added that > > don't yet run without failures (and this is the right thing > to do). So I > > strongly believe we should work with a white list. > > Huh? Why is adding broken tests the right thing to do?
The test isn't broken, it just exposes a bug (or missing feature) in Classpath. Often when you run into a Classpath bug (or missing feature) the right thing to do is to add a test, later Classpath gets fixed. > And besides, if a broken test is added, this way there will be > motivation to resolve the discrepancy. With a whitelist, a broken > test can get added but no one will notice and then it just sits > there getting stale. The alternative is worse, it's the situation we're in now. There are lots of broken tests and we have no idea if the failures are due to Classpath, VM or test bugs. There is obviously no silver bullet, it a matter of making the right trade-off. I believe I white list is the right trade-off, because it offers a list that is known to be good at all times, so when someone checks out Classpath and starts hacking, she will be comfortable to know that the white list will contain tests that run correctly and can use that fact to test if Classpath and/or the VM were build correctly and any changes made didn't break anything (subject to test coverage, of course). The downside of a white list is that we need to be extra vigilant in keeping it up to date, but I don't think this is a problem. It might be helpful to have an option in Mauve to list all the successful tests in a particular run, that way that list can be compared against the standard white list to see if additional tests can be added to the white list. Regards, Jeroen _______________________________________________ Classpath mailing list [email protected] http://lists.gnu.org/mailman/listinfo/classpath
