Benji York wrote:

Tarek has started some very interesting work on adding performance testing to the Zope 3 testing infrastructure and it so happens that Jim and I were discussing something very similar last week, so I'd like to suggest some functionality we might want to have (which I should be able to help implement).

1) warn about regressions: the test runner will keep per-test, machine-independent records of how long tests take and will report regressions larger than a predefined percentage. These records will be checked in so that if someone else makes changes (in a fresh checkout) that causes a particular test to slow down drastically, they will be warned.

-> About machine-independence: Stephan brought the pystone idea to take care of it

-> some thaught about percentage

this percentage (let's call it the tolerance), may vary a lot, depending on the test complexity. So i think we can do the performance regression testing in two steps i would like to suggest:

1/ a first step would be to run the tester in a special mode to render an ordered list for each non-marked test:

   + a measure of the average complexity, including the number of calls
and when possible the type of complexity (linear, exponential, logarithmic, etc...)

   + a measure of pystones

2/ This stats would then be used to mark all hot spots with a "max pystones allowed".
   The decorator we implemented last week fits well for this.

2) testbrowser should keep up with a (machine-independent) metric of how long the previous request took so performance assertions can be made inside tests. E.g.

    >>> browser.last_request_time < 0.5

3) the functional testing framework should be extended to allow the collection of total time (again, machine-independent) per request and the test runner should have an option to display the top "n" slowest requests.


For the same reasons, it would be nice to have the same kind of regression test for memory taken by objects:

In the webmail i am coding, if i suddenly change the code and by doing this, i double the size of a mail object in the ZODB, that can be quite bad, as i have thousands of instances of them.

So i would like to be able to do the same kind of marker for memory.



Zope3-dev mailing list

Reply via email to