Maybe it's an overly simplistic question, but: what makes the tests
slow currently? It's not simply the volume of them. It's more than
possible for Python to race through hundreds of tests per second under
the right conditions.
Some of Django's test modules obviously zip along, but others seem to
take a rather long time to complete individual tests. Has anyone
identified which tests are the slowest and what is making them so
slow? Is it the length of the tests? The parsing? The fixtures that
are loaded? The data generated in the test itself?
Without knowing what gives the current test suite its performance
characteristics aren't we all just guessing at how a rewrite of it
will compare?
Perhaps a good test would be to take one test module and convert it
from doc tests to unit tests and profile the before and after versions
to see how they compare. While the micro-scale may not ultimately
compare to the macro, it would at least be a starting benchmark.
I don't have any answers here, but that's what occurs to me about
performance...
All the best,
- Gabriel
On Apr 9, 12:51 pm, Paul McMillan <[email protected]> wrote:
> I've revised the proposal, but am inlining more complete answers.
>
> > If someone needs to fix a bug, they will essentially need to
> > write the test twice; once for the old framework, and once for the
> > new. Do you have any suggestions on what we should be doing to
> > mitigate this complexity?
>
> Unfortunately some mismatch seems inevitable for major changes like
> the proposed one. I will include comments indicating where
> corresponding test code in the previous branches can be found to help
> with this problem. If the test code is a completely new chunk of code,
> it will be written in the unittest style which should not be too hard
> to add to the old test suite. Many of the bits of code that are
> currently doctests are fairly well vetted by now, and so should
> hopefully have few changes necessary for backports. If it's just a
> minor couple line testing change, that shouldn't be too hard to make
> into a doctest if that's more appropriate. Better ideas are, of course
> welcome.
>
> > We have a buildbot currently in development; if you're have the
> > resources to contribute, it would be good if these could be integrated
> > permanently into the build farm.
>
> That might be possible. I was intending to pull up vmware instances on
> my development machine, but we should discuss how best I could
> contribute to that.
>
> > The #python IRC channel needs to think about the problem a little more.
>
> I figured that was probably the case, given that you included that
> aspect in the project suggestion...
>
> > While this project is *very* appealing from a project point of view --
> > we get the benefit of individually failing and isolated tests -- but
> > the risk of test runtime blowouts is huge. Any suggestions on how to
> > tackle this problem (and speed up test runs in general) would be
> > almost as valuable as the doctest->unittest conversion itself.
>
> This is a bit of a sticky problem. I'll be thinking about it between
> now and when the project (potentially) starts, but my first thoughts
> are as follows:
>
> We know that the existing doctests work when run sequentially. There
> may be bugs that are revealed as they're run individually, but to a
> first order, we know that some tests can safely be run sequentially
> without the setup/teardown each time. Since we're worried that this
> process is going to greatly inflate runtime, why not investigate
> options that let us avoid it where appropriate? Specifically, why not
> structure things such that we can lump tests together into groups that
> can safely be run sequentially, but also allow a "full" test run that
> runs each tested item individually as is currently the case for
> unittests?
>
> There are a couple problems with this approach. The first is that
> there are now 2 different ways to "pass the tests" - the quick run and
> the full run. Developers will still have to run the "full" run
> sometimes, negating at least some time saved. On the other hand, the
> proposed "quick" run doesn't provide LESS coverage than the existing
> doctests, so it might be good enough, particularly for tests which are
> currently in the regression test category. Unfortunately, it's
> probably not broadly appropriate for a majority of existing unit
> tests. We could make the "quick" option the standard for unit tests,
> with the "full" option reserved for assistance in bug hunting when
> there's an identified error.
>
> It needs more concerted thought, but I'd be interested in feedback.
> Maybe it's a terrible idea. This subject should probably have a
> dedicated thread here for better exposure and more specific
> discussion. Hopefully others have good ideas.
>
> In any case, I should do some profiling work as I'm doing the
> conversion to identify particularly inefficient areas and clean them
> up. This is part of the advantage of doing it by hand rather than as
> an automated process - another set of eyes to look for problems.
>
> I reworked the proposal timeline slightly to move a week from the end
> to the beginning to implement major changes (such as the above idea)
> to improve performance before I get deeply into the conversion.
>
> -Paul
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
