I've revised the proposal, but am inlining more complete answers. > If someone needs to fix a bug, they will essentially need to > write the test twice; once for the old framework, and once for the > new. Do you have any suggestions on what we should be doing to > mitigate this complexity?
Unfortunately some mismatch seems inevitable for major changes like the proposed one. I will include comments indicating where corresponding test code in the previous branches can be found to help with this problem. If the test code is a completely new chunk of code, it will be written in the unittest style which should not be too hard to add to the old test suite. Many of the bits of code that are currently doctests are fairly well vetted by now, and so should hopefully have few changes necessary for backports. If it's just a minor couple line testing change, that shouldn't be too hard to make into a doctest if that's more appropriate. Better ideas are, of course welcome. > We have a buildbot currently in development; if you're have the > resources to contribute, it would be good if these could be integrated > permanently into the build farm. That might be possible. I was intending to pull up vmware instances on my development machine, but we should discuss how best I could contribute to that. > The #python IRC channel needs to think about the problem a little more. I figured that was probably the case, given that you included that aspect in the project suggestion... > While this project is *very* appealing from a project point of view -- > we get the benefit of individually failing and isolated tests -- but > the risk of test runtime blowouts is huge. Any suggestions on how to > tackle this problem (and speed up test runs in general) would be > almost as valuable as the doctest->unittest conversion itself. This is a bit of a sticky problem. I'll be thinking about it between now and when the project (potentially) starts, but my first thoughts are as follows: We know that the existing doctests work when run sequentially. There may be bugs that are revealed as they're run individually, but to a first order, we know that some tests can safely be run sequentially without the setup/teardown each time. Since we're worried that this process is going to greatly inflate runtime, why not investigate options that let us avoid it where appropriate? Specifically, why not structure things such that we can lump tests together into groups that can safely be run sequentially, but also allow a "full" test run that runs each tested item individually as is currently the case for unittests? There are a couple problems with this approach. The first is that there are now 2 different ways to "pass the tests" - the quick run and the full run. Developers will still have to run the "full" run sometimes, negating at least some time saved. On the other hand, the proposed "quick" run doesn't provide LESS coverage than the existing doctests, so it might be good enough, particularly for tests which are currently in the regression test category. Unfortunately, it's probably not broadly appropriate for a majority of existing unit tests. We could make the "quick" option the standard for unit tests, with the "full" option reserved for assistance in bug hunting when there's an identified error. It needs more concerted thought, but I'd be interested in feedback. Maybe it's a terrible idea. This subject should probably have a dedicated thread here for better exposure and more specific discussion. Hopefully others have good ideas. In any case, I should do some profiling work as I'm doing the conversion to identify particularly inefficient areas and clean them up. This is part of the advantage of doing it by hand rather than as an automated process - another set of eyes to look for problems. I reworked the proposal timeline slightly to move a week from the end to the beginning to implement major changes (such as the above idea) to improve performance before I get deeply into the conversion. -Paul -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
