I'm really pleased to see coverage.py being used this way, thanks for doing the work. But I would strongly recommend starting coverage before importing the modules to avoid this effect of module-level statements not being measured. You can add exclude regexes to mitigate the problem, but it's a losing game. You'll inevitably either under- or over-exclude statements.
One of the things I tried hard to do with coverage was remove noise from the workflow. Excludes let developers move known not-executed code out of their attention, and lets coverage.py focus the developers on potential problem. Using excludes to "fix" module-level statements feels like a mis-use. I'd be glad to lend a hand if I can help. --Ned. http://nedbatchelder.com Kevin Kubasik wrote: > It's statement coverage, so it largely depends on what is called, and > how. > > http://nedbatchelder.com/code/coverage/faq.html > > The above explains why we get that effect. the problem is that we don't want > to start it much earlier, since we > are doing identification of apps. It's never perfect, but its more to > give us an idea of where we have totally missed a codepath, imports > are easy to identify. > > http://nedbatchelder.com/blog/200710/flaws_in_coverage_measurement.html > > Just my decision making process, but I am completely open to trying to > move things around if this is a serious problem. But what's generally > a bit easier is just expanding our exclude statements. I've added some > import excludes which eliminates most of them, I'll commit tomorrow! > > On Sat, Jun 27, 2009 at 12:36 AM, Alex Gaynor <[email protected] > <mailto:[email protected]>> wrote: > > > > On Sat, Jun 27, 2009 at 1:22 AM, Kevin Kubasik <[email protected] > <mailto:[email protected]>> wrote: > > So, up until now, most of the work on windmill hasn't exactly > been 'tester ready' in that, if you didn't you my exact > incantation of settings, luck, module versions and love, you > didn't get very far. While I still haven't gotten real docs > done (my original plan for the week) I have learned a valuable > lesson in assumptions. I foolishly assumed that splitting the > WSGI handler into its own thread, > while introducing another server (windmill), and doing my fixture > loading in the primary thread would just work. Surprisingly, it almost does, > except for the sqlite3+:memory: instance. > Obviously, as this is how 99% of django tests are run, I had > hit a snag. I believe that the problem is > solved, but it did put me a bit behind, as I have had to totally > reconsider how the loaders and databases > are handled. Namely, the following questions arose: > > 1. Should windmill tests default to > non-isolated/non-transactional DB behavior? Basically, > we are providing the means for functional tests, these > should be tests like 'Register and Complete a Profile', > then 'Edit Profile'. We don't want this as one massive > test, and it seems like that would be the expected > behavior most of the time, and still allowing for the > option of specific tests being run in isolation seems > like the best move. However, this could be confusing, or > just bad practice, so I wanted to get some feedback. > 2. Scratch my #2 I caught one of the Windmill guys on the > IRC and got some good direction on detection stuff. > 3. What is the general interest in test-only models as a > public api? The mechanics of it have been worked out for > the regression suite, but the debate falls to one of the > following systems. > 1. A class used instead of db.Model (db.TestModel) > 2. A module in the app (test_models.py) > 3. Similar to fixtures (a property on tests) > 4. A settings option > 4. I am assuming that code coverage of windmill tests isn't > that useful of a number, given the specialized execution > paths etc. But I wanted to double check that people > wouldn't be surprised by that. > > Overall, there has been marked improvements in the runner > state, and I've added some more tests as well. However, I am > holding off on the really js intense tests until the framework > is rock solid. But I wanted to get moving on documentation > sooner rather than later, so I expect a bit more cleanup next > week to make sure that the elaborate charade we play (conning > windmill to play with us) is reliable for 3rd party > applications as well. > > The branch isn't really ready for testing yet, but it has been > known to work. And Eric has kindly thrown up a coverage report! > > http://media.ericholscher.com/django_coverage/ > -- > Kevin Kubasik > http://kubasik.net/blog > > > > Just a small note, but there seems to be an issue with the > coverage, in that any module level statements aren't reported as > being executed, such as imports, function definitions, or class > definitions. That might be an issue with whatever does the > coverage report though. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your > right to say it." --Voltaire > "The people's good is the highest law."--Cicero > > > > > > -- > Kevin Kubasik > http://kubasik.net/blog > > > -- Ned Batchelder, http://nedbatchelder.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
