On Tue, 6 Dec 2005, Bryan Stearns wrote:
It's important to run the tests against a consistent "base" repository, whether it's empty or not, and it's important to build that repository using the same sources you're using for Chandler (so, saving off built repositories to use across Chandler versions would scare me).
Yes, definitely, consistency is very important or else all comparisons are bogus. My point had to do with the potential expendability of non-real end user situations when compared to real end user situations. As for the across chandler versions issue, we sure have to be able to attain that goal some day.
There's "dynamic state" in the repository that strongly influences performance: collection indexes and cached block trees come to mind immediately, and there's probably more; block tree caching has already caused confusion in interpreting performance measurements.
Until that is resolved, we ought to be very careful with comparisons.
It might mean that running the suite means building a specific-sized repository using current sources, then running Chandler against a fresh copy of it for each test (or batch of related tests where we'll accept the possibility of the tests influencing each other). We might even build several test repositories this way (empty, 100, 1000, 10000 items?) and choose an appropriate-sized one for each test batch so that the execution time is acceptable (say, shooting for a test runtime of less than a minute to keep the whole suite manageable).
Or start with building the repository to some consistent specifications of data, then back it up and restore it before each test is run. Some of our calendar tests, for instance, are dependant on the date they are run since, depending on that date, different events are relevant.
Andi.. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
