On Monday, September 26, 2011 11:37:30 Don Clugston wrote: > I think we need to have a strategy for managing the amount of time > required for running the unittests. Currently, on Windows, the time > for all compiler tests + phobos tests is one hour. (The druntime tests > are only 30 seconds). > By contrast, running all compiler tests + phobos tests on D1, takes > about four minutes. > > I used to be able to use test-driven development extensively -- > running all tests after every change. But this is no longer viable on > D2. It's quite terrible to have to wait for one hour before finding > that you broke something. > A very large fraction of the time is used in testing only a tiny part > of the Phobos API. So I have some anxiety over what may happen in the > long term -- if current trends continue, we could easily have ten > hours of Phobos tests eventually. > > How about defining a version, eg: > version=ExtendedPhobosTests; > which contains the more exhaustive, black-box tests, which take almost > all of the time. So that the standard tests would consist of > (1) regression tests, which should have a corresponding bugzilla > entry, unless they were discovered during development of the library > (The feature of these tests, is that at some point, they failed); > and > (2) code coverage tests. > > Both of these execute quickly, and since they are linear with the > number of reported bugs + the number of lines of code, they should > always remain manageable. The black-box tests, on the other hand, are > potentially unlimited.
In principle, I think that it's a solid idea, though I'm not sure that it's necessarily at all obvious where the line should be drawn between the normal and extended tests, and that could be a bit tricky. We wouldn't want the normal unit tests to become not extensive enough simply in an effort to reduce the amount of time that it takes to run tests, but on the other hand, we don't want to the tests to take forever, so it's a bit of a balancing act. For whatever it's worth, on the whole, I believe that the main culprit for the length of time that it takes to run the unit tests is the amount of time that it takes to compile them. Actually running the tests is generally quite quick. And Windows does that much worse then, because it compiles all of the modules together for unit tests instead of separately like Posix does - though that has the advantage of helping to find module interdependencies with module constructors which don't get found otherwise, so I'm not sure that we necessarily want to change how Windows does the unit tests. Maybe one way to deal with that would be to adjust the makefiles so that the modules are normally tested separately on _all_ of the OSes, but there's make target on all OSes to build them all as one if we want to test for module constructor issues - though then that target risks not every being called. The other question is what the autotester should be running. Should it run the normal unit tests or the extend unit tests? Should it run then the normal ones normally and the extended ones periodically? It'll be easier to miss platform- specific bugs if the extended tests are only ever run on an individual's machine and never the autotester. On the whole though, I think that this is a good idea. - Jonathan M Davis _______________________________________________ phobos mailing list [email protected] http://lists.puremagic.com/mailman/listinfo/phobos
