Once #17365 (as per https://github.com/jezdez/django-discover-runner) is implemented it will be a lot cleaner, since that splits out 1 & 2 from 3. Lets call them Django, Django api compatible, and Django project api
Three things hammered home for me that Carl's approach is the correct way to do tests: 1.) Deploying to Heroku which encourages a small virtualenv size to as small as possible for deployment. Including test code in deployment is daft. 2) The usefulness of 3rd party application test suites can be limited especially when developing against trunk or varying versions. I don't want to fork just to make their test suite pass if there's nothing wrong with the code base itself - just the tests. 3.) My project provided a custom middleware and auth backend which extended the auth functionality by adding sites to users, and authenticating against that (along with email usernames). It broke the django tests in all sorts of places. That's when I discovered Carl's talk and was an instant convert. If I'm developing django itself or to be compatible with the django api then I need to run the django test suite and ensure it passes with my custom app, but there are any number of ways that a project or even other applications can break the test suite so project tests should never be run against the django test suite. On Saturday, 25 August 2012 18:15:07 UTC+10, Russell Keith-Magee wrote: > > Hi all, > > So, I've been working on a Django branch [1] to implement the approach > to pluggable user models that was decided upon earlier this year [2] > > [1] https://github.com/freakboy3742/django/tree/t3011 > [2] https://code.djangoproject.com/wiki/ContribAuthImprovements > > The user-swapping code itself is coming together well. However, I've > hit a snag during the process of writing tests that may require a > little yak shaving. > > The problem is this: With pluggable auth models, you lose the > certainty over exactly which User model is present, which makes it > much harder to write some tests, especially ones that will continue to > work in an unpredictable end-user testing environment. > > With regards to pluggable Users, there are 5 types of tests that can be > run: > > 1) Contrib.auth tests that validate that the internals of > contrib.auth work as expected > > 2) Contrib.auth tests that validate that the internals of > contrib.auth work with a custom User model > > 3) Contrib.auth tests that validate that the currently specified user > model meets the requirements of the User contract > > The problem is that because of the way syncdb works during testing > some of these tests are effectively mutually exclusive. The test > framework runs syncdb at the start of the test run, which sets up the > models that will be available for the duration of testing -- which > then constrains the tests that can actually be run. > > This doesn't affect the Django core tests so much; Django's tests will > synchronise auth.User by default, which allows tests of type 1 to run. > It can also provide a custom User model, and use @override_settings to > swap in that model as required for tests of type 2. Tests of type 3 > are effectively integration tests which will pass with *any* interface > compliant User model. > > However, if I have my own project that includes contrib.auth in > INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests > from contrib.auth. If I have a custom User model in play, that means > that the tests of type 1 *can't* pass, because auth.User won't be > synchronised to the database. I can't even use @override_settings to > force auth.User into use -- the opportunity for syncdb to pick up > auth.User has passed. > > We *could* just mark the affected tests that require auth.User as > "skipUnless(user model == auth.User)", but that would mean some > projects would run the tests, and some wouldn't. That seems like an > odd inconsistency to me -- the tests either should be run, or they > shouldn't. > > In thinking about this problem, it occurred to me that what is needed > is for us to finally solve an old problem with Django's testing -- the > fact that there is a difference between different types of tests. > There are tests in auth.User that Django's core team needs to run > before we cut a release, and there are integration tests that validate > that when contrib.auth is deployed in your own project, that it will > operate as designed. The internal tests need to run against a clean, > known environment; integration tests must run against your project's > native environment. > > Thinking more broadly, there may be other categories -- "smoke tests" > for quick sanity check that a system is working; "Interaction tests" > that run live browser tests; and so on. > > Python's unittest library contains the concept of Suites, which seems > to me like a reasonable analog of what I'm talking about here. What is > missing is a syntax for executing those suites, and maybe some helpers > to make it easier to build those suites in the first place. > > I don't have a concrete proposal at this point (beyond the high level > idea that suites seem like the way to handle this). This is an attempt > to feel out community opinion about the problem as a whole. I know > there are efforts underway to modify Django's test discovery mechanism > (#17365), and there might be some overlap here. There are also a range > of tickets relating to controlling the test execution process (#9156, > #11593), so there has been plenty of thought put into this general > problem in the past. If anyone has any opinions or alternate > proposals, I'd like to hear them. > > Yours, > Russ Magee %-) > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/oOjPaFQoeFwJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
