Benji York wrote:
Tarek has started some very interesting work on adding performance
testing to the Zope 3 testing infrastructure and it so happens that
Jim and I were discussing something very similar last week, so I'd
like to suggest some functionality we might want to have (which I
should be able to help implement).
1) warn about regressions: the test runner will keep per-test,
machine-independent records of how long tests take and will report
regressions larger than a predefined percentage. These records will
be checked in so that if someone else makes changes (in a fresh
checkout) that causes a particular test to slow down drastically, they
will be warned.
-> About machine-independence: Stephan brought the pystone idea to take
care of it
-> some thaught about percentage
this percentage (let's call it the tolerance), may vary a lot, depending
on the test complexity.
So i think we can do the performance regression testing in two steps i
would like to suggest:
1/ a first step would be to run the tester in a special mode to render
an ordered list for each non-marked test:
+ a measure of the average complexity, including the number of calls
and when possible the type of complexity (linear, exponential,
+ a measure of pystones
2/ This stats would then be used to mark all hot spots with a "max
The decorator we implemented last week fits well for this.
2) testbrowser should keep up with a (machine-independent) metric of
how long the previous request took so performance assertions can be
made inside tests. E.g.
>>> browser.last_request_time < 0.5
3) the functional testing framework should be extended to allow the
collection of total time (again, machine-independent) per request and
the test runner should have an option to display the top "n" slowest
For the same reasons, it would be nice to have the same kind of
regression test for memory taken by objects:
In the webmail i am coding, if i suddenly change the code and by doing
this, i double the size of a mail object
in the ZODB, that can be quite bad, as i have thousands of instances of
So i would like to be able to do the same kind of marker for memory.
Zope3-dev mailing list