On Fri, Nov 5, 2010 at 3:46 AM, David Cramer <[email protected]> wrote: > A common behavior I seem to have is the need to tweak the settings > object for certain test cases. The other case is that in many cases we > were (are?) relying on settings being configured a certain way for the > Django tests to even work. I brought this up in #django-dev a while > back, but wanted to open it up to further discussion. > > Let me start with an example test: > > def test_with_awesome_setting(self): > _orig = getattr(settings, 'AWESOME', None) > settings.AWESOME = True > > # do my test > ... > > settings.AWESOME = _orig > > So the obvious problem for me here is that I'm repeating this same > code flow in a lot of situations. Ignoring the fact that it's ugly, > it's just not fun mangling with settings like this (at least, not fun > having to reset the values). > > My proposal is to include some kind of utility within the test suite > which would make this easier. There's a couple ways I could see this > working: > > 1. The settings object could be copied and reset on each case. > 2. The settings object could be replaced with a Proxy which stores a > copy of any value changed since reset, and returns that value if > present. It could then simply just be reset (clear the proxy's dict) > on each setUp. > > Anyways, I'd love to hear how others have dealt with this and any > other possible solutions.
Love the idea in general. Django's own test suite is full of this pattern (or buggy partial implementations of it), which is a prime indication that we should providing something at the framework level to make this easy to do. The devil is in the detail. If you look at the ways Django uses this pattern, there are lots of edge cases that need to be handled. For example: * Some settings are lists, and the test case needs to append/prepend/insert into that existing list, rather than overwriting a list. An example of this is adding an extra context processor for test purposes. Yes, this could be handled by manually specifying the full list, but it's a fairly common pattern, so it would be nice to have a quick way to represent the pattern. * Settings that are internally cached. For example, anything that modifies INSTALLED_APPS. * Settings that need to make call to reset state affected by loading new new group of settings. For example, if you change TZ, you need to call "time.tzset()" on teardown to reset the timezone. Similarly for deactivating translations if you change USE_I18N. * Settings that need to be removed, rather that set to None. Again, TZ is an example here -- there is a difference between "TZ exists and is None" and "TZ doesn't exist". I've probably missed a couple of other edge cases; it would be worth doing an audit of Django's test suite to see all the places that we've used this pattern, and the workarounds we've had to use to clean up after it. There's also the matter of making this system easy to use in the practical sense. The context manager approach that have been given so far in this thread is a nice syntactic fits, but miss one big use case -- modifying settings for *every* test in a TestCase. This is the way that settings changes are used right now in Django's test suite. The provided decorator has the same limitation, but it shouldn't be too hard to modify it to be a class-or-function decorator. Lastly, a persistent source of bugs in Django's own test suite is having complete settings isolation. If you're doing true unit tests, it's not enough to just replace one or two settings -- you have to clear the decks to make sure that the user's settings file doesn't provide an environment that breaks your test. This manifests itself as tests for contrib apps that pass fine in Django's own suite, but fail when an end user deployes the app -- for example, you deploy contrib.auth in an environment that doesn't have the login URLs deployed, and your tests fail; or you deploy contrib.flatpages but don't deploy contrib.sites, and the tests fail. Over time, we've cleaned up these issues as we've found them, but the real fix is to make sure a test runs in a completely clean settings environment. That is, we reset settings to a baseline (global_settings.py would be a an obvious candidate) rather than just tinkering with the one or two settings that we think are important. The "Clear the decks" approach is a different use case to "just change these three settings", but it's closely related, and worth tackling at the same time, IMHO. So - tl;dr. love the idea. However, we need to handle the edge cases if it's going to be added to Django trunk, and replacing Django's own usage of the ad-hoc pattern is as good a test as any that we've tackled the edge cases. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
