On Saturday 12 January 2008 06:16:14 James E Keenan wrote: > We can overcome this difficulty to a considerable extent by refactoring > that mocks the conditions found on various OS/platform/compiler > combinations. Even if we're not on a Windows box, we can write tests > that ask, "Suppose that we were on Windows and were provided with > information as to whether we had a VC++ compiler or not. Is the code > which uses that information sound? Have we considered what happens at > all diversion points in the control flow after we have received that > information? If we were on Windows, would our tests exercise every nook > and cranny of the code?"
Sure, but then you're testing the mock and not the configuration system, because the configuration system doesn't actually get tested because it never applies to any other platform. At best, the mock can only answer the question "If MSVC ever ran on this platform, would this test likely run?" MSVC's not going to run on any platform but Windows in the near future. It doesn't answer any interesting questions about the functional behavior of this part of the configuration system because there's only one interesting question about the functional behavior of the configuration system, and that's "Does MSVC run on this platform?" and we already know the answer to that. Further, if the test fails, what useful information does it give? Is the mock wrong? Did it get called correctly? Now I have at least two things to debug, possibly three, and the third is the details of a platform I don't have and can't actually test. > And that's what these tests do. The fact that they achieve high test > coverage of auto::msvc > (http://thenceforward.net/parrot/coverage/configure-build/config-auto-msvc- >pm.html) when run on a non-Windows system gives me considerable confidence > that auto::msvc will do the right thing when run on Windows. It also gives > anyone in the future doing refactoring on auto::msvc a framework with which > to test the soundness of that refactoring. High test coverage is a good thing only if it adds value to the project. I think what you're testing here is your ability to make the configuration system look like someone's run it on Windows with MSVC. This is not a useful configuration for building or testing Parrot. The only thing that gives me confidence that auto::msvc will do the right thing when run on Windows is if the test passes on Windows and the MSVC compilation produces a working binary. -- c
