On Fri, Apr 9, 2010 at 8:59 AM, Paul McMillan <[email protected]> wrote: > I've written an application to improve Django's Test Suite. > http://code.djangoproject.com/wiki/SummerOfCode2010#Testingupdates > > My application is here (also pasted below for convenience): > http://socghop.appspot.com/gsoc/student_proposal/show/google/gsoc2010/paulmcmillan/t127077396156 > > I'm hoping for constructive feedback particularly in the area of > maintaining test integrity. I realize this is a fairly late > application, but the task itself is relatively non-controversial. > > Thanks! > -Paul > > ========================= > The problem > > Django's testing suite has become fragmented over time. The modeltests > and regressiontests directories serve similar purposes and should be > joined. Nobody wants to do concerted work to improve testing because > it isn't a showstopping bug or a new feature. GSOC provides an > opportunity to support this kind of work. > > Proposal > > I will merge the code in modeltests and regressiontests, turning > doctests into unit tests and cleaning up existing code as I go. I will > use coverage.py to verify that the re-written tests cover the same > areas of code as the previous ones did, as well as identifying areas > without coverage. > > Many GSOC proposals are for large feature improvements that will > involve a major code branch and merge. This project can instead be > done as a series of incremental patches, reducing the impact of a > failure to complete and providing more immediate results.
This is one of the appealing aspect to housekeeping style projects. However, in this case, you may want to give some thought to the backporting complexity that you will be introducing. Yes, when you have finished converting the "basic" doctest into a unittest suite, we can commit that to trunk. However, at the point we do this, there is a massive disconnect between trunk and the stable branches. If someone needs to fix a bug, they will essentially need to write the test twice; once for the old framework, and once for the new. Do you have any suggestions on what we should be doing to mitigate this complexity? > In order to avoid introducing new bugs, I will configure a variety of > virtual machines for testing. At a minimum, I will test in Ubuntu, > Windows XP, and Mac OS X, with the most recent versions of Python 2.4, > 2.5, and 2.6. On each of these systems, I will test against the > supported database backends (PostgreSQL, MySQL, SQLite, Oracle). I > will build scripts to automate the test running in these environments > so that testing can continue prior to patches being merged to trunk. We have a buildbot currently in development; if you're have the resources to contribute, it would be good if these could be integrated permanently into the build farm. > Since Django 1.2 is dropping support for Python 2.3, there is extra > code in the test suite to support 2.3 that can be removed. This should > improve the speed of test running. If it does, it will be by a trivial fraction. The extra overheads cause by Py2.3 requirements are fairly minor. > Internally, when a doctest is run, Python converts it into a unittest > before actually executing it. It might be possible to use this code to > automate the conversion. Converting to unittests should improve test > run time because it eliminates the parsing necessary to convert each > doctest into a unittest. The #python IRC channel anecdotally agrees on > this point. The #python IRC channel needs to think about the problem a little more. Yes - a simple unittest will be faster than a comparable doctest. The problem is that this isn't what will be happening. In a doctest, there is one prolonged setup process; as the doctest runs, the test setup evolves with the test. In a unittest, each testable item needs to be isolated, which means running setUp and tearDown multiple times. Each setup/teardown cycle means creating a bunch of database objects, and then rolling back the database to destroy those objects. The setup/teardown process is effectively done once in a doctest. It is done N times on a unittest with N tests. This is a potentially expensive operation, and it *will* slow down test runs. While this project is *very* appealing from a project point of view -- we get the benefit of individually failing and isolated tests -- but the risk of test runtime blowouts is huge. Any suggestions on how to tackle this problem (and speed up test runs in general) would be almost as valuable as the doctest->unittest conversion itself. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
