Forgot to include Django-Dev in this update, oops! (Sorry those who will get it twice!)
Original Message: Ok, so not showing the proposal is silly, below is the original, but we have been restructuring the prioritization to maximize the awesomeness guaranteed to be accomplished. I left the original intact so everyone will know what was accepted, and what the overall scope is, but here is what our discussion this week has narrowed down, and what I have coded/experimented with so far. First, we have come to a strong distinction regarding the 3 areas of testing, namely: * Unit -- Usually testing a specific piece of action or logic * Integration -- Cross application * Functional -- UI level (This isn't entirely accurate, but its the terminology we will use, given how django is structured.) I also want to note that 95% of my time will spent on the Functional/UI testing. We just spent some time making the distinction. The only real thing I plan on adding to Unit and Integration testing will be code coverage (and maybe cleaned up versions of some existing assertions). I know these three will need to be separate testing components or suites, but there is still debate on how to approach that. However, concrete things we do know are: 1) First step will be tacking code coverage. I plan on integrating this with the django test runner, and our regression suite runner. Generated reports need to be ported to use Django templates, and I plan on making a sample coverage report for the regression suite, so we have a better idea of how django itself is covered. A secondary goal will be allowing aggregation of results over multiple runs (aka. smoke tests) so we can run against all database types and see coverage for each of those runs, and overall. However, this capability is harder to test, and will be based on time availability. `Sample Report <http://test.55minutes.com/test_report/>`_ 2) Test-Only models: I will make a system by which models can be declared for tests, but not regular runs. This is basically the implementation of this ticket: `TestOnly Models <http://code.djangoproject.com/ticket/7835>`_ However, I don't plan on porting the current django regression suite to take advantage of this new functionality. I will use them myself in the admin tests, and hopefully there will be enough examples there that the rest of the regression suite can be migrated to use this feature as appropriate. 3) Windmill integration in tests. There are a few goals here. * They should be in django.contrib.admin.tests and selectively runnable (Not everyone will want to run the windmill tests each time, as they are much more time consuming then unittests.) * This framework should be reusable by any django app developer, and provide reasonable examples in the admin tests. * AJAX compatibility. This is a tough cookie, given that the django development server is single threaded. Meaning that we need to run the windmill tests from a separate process. This is probably less than ideal, since part of the windmill integration will be making assertions in the windmill context based on information available in Django. There is still probably some discussion needed regarding how to best approach this problem, and if its worth worrying ourselves with django-specific windmill conveniences. One idea has been to provide a 'recommend testing setup' against a multi-threaded server. Another possibility I explored was using CherryPy's wsgi capabilities to host django (I have a working Proof of Concept of this one). Overall, its a matter of deciding which of the following we want: a) Vanilla Windmill tests in one process, django development server on another b) Integrated Windmill tests and server in same process, server updated to be multi-threaded c) Integrated Windmill tests and context in process, only runnable from hosted server. Either way, there needs to be input on this one. How to test AJAX capabilities, while (ideally) still providing access to the django context for assertions. `` Example: Model BlogPosts is declared and added to admin. Windmill checks that model is available in admin windmill goes to the add BlogPosts page Windmill creates a BlogPost and saves it using the admin interface Assert (via django unittest API's) that model is now in database. Windmill assert that model is in BlogPosts list Windmill assert that searches/filters work (by performing expected filters and comparing results.) `` This integration would make for some very powerful testing options, but obviously presents with a host of problems in the current model. 4) Consider making this Windmill integration pluggable to allow writers who like Selenium to use that too. But realistically this depends on how integrated we make these capabilities. I was not planning on giving Selenium much thought, (but good design dictates that I will make sane abstractions where possible, but I don't want to concern myself with 2 massive UI frameworks). However, I wanted to know if there was a large body of django developers who already knew and wanted to continue using Selenium. If this is the case, we might change priorities. Realistically, those are the priority tasks, I have a few handy assertions and convenience functions that I found as tickets and would like to get patches in for if I have time, but listed above are the priorities. For those who haven't seen it, here is the original proposal. Upgrade the Awesomness Quotient of the Django Test Utils and Regression Suite ============================================================================= Abstract ******** To fix and test, so cover the Admin site/ with Windmill use, avoiding the great fright. At moment be, we cover not, popular/ is the admin but tested is not. Great addition tests would be/ the key in promoting greater Django harmony. The Problem *********** Django has a fantastic set of regression tests which cover much of the codebase, but the famous Admin interface isn't covered by any sort of automated tests. In addition, the tools for testing a site under the Django framework are weak, even if the API's available are quite powerful. Proposal -------- Overall Django has some great testing tools, but the addition of a few key integrations and tools could make it a much more comprehensive and easy to use testsuite. Namely, nose runners and extensions, Windmill integration, and some expansions to the core testing API's. Namely, I want to accomplish the following: 1. Windmill support already works/exists! Add anything that might be missing. 2. Expand the Django test runner to provide Code Coverage statistics. There are already some efforts towards this goal underway, and I don't plan on reinventing the wheel. My hope is to integrate their work, and potentially expand it based on Django's community requests. Most likely the base of this work will be figleaf, but I plan on researching other options. Lastly, I want to evaluate the coverage of Django's current regression suite and provide a wiki page/writeup detailing any major deficiencies so that they can be addressed. (Similar to this ticket: `CodeCoverage < http://code.djangoproject.com/ticket/4501>`_) 3. Utilize the new Windmill test support to provide coverage of the Admin interface. This is an extremely large task overall, but given how easy Windmill has made the creation of tests, I hope that I will be able to provide coverage of a majority of the admin's functionality. This will not include support for the django.contrib.gis namespace, but I will try and test all other namespaces that have Admin integration or functionality (namely auth, admin and comments). 4. **New** Cleanup/Expand the current testing API to include a few new super-awesome features/functions. Namely, I want to implement/integrate the following tickets (listed in order of priority): 1. `Skipping Tests <http://code.djangoproject.com/ticket/4788>`_ 2. `assertNumQueries <http://code.djangoproject.com/ticket/5416>`_ 3. `assertContext <http://code.djangoproject.com/ticket/5333>`_ 4. `assertNoBrokenLinks <http://code.djangoproject.com/ticket/5418>`_ 5. `Mock Request Creation Outside of urls < http://code.djangoproject.com/ticket/9002>`_ 6. `Fixtures with doctests <http://code.djangoproject.com/ticket/5624>`_ 7. `TestOnly Models <http://code.djangoproject.com/ticket/7835>`_ 8. `reStructured Text Checker <http://code.djangoproject.com/ticket/7474 >`_ 9. `fuzz or datagen testing <http://code.djangoproject.com/ticket/5419 >`_ Timeline -------- Weekly I should be able to put in between 20-30 hours a week, but that will be focused on the weekends. During the week I will have classes, but will be available during many of them for discussion, but will avoid coding. (I have 2 lab classes that require me to be in a room for 4 hours, but most of the time is watching something settle or spin or do nothing at all). A breakdown of my projected timing: - Week 1: Gain familiarity with Windmills capabilities and limitations, scope of each addition and research into other implementations. - Week 2: Get code coverage working for both Django Regression Suite and under a manage command. - Week 3-7: This is the time for as much of item 4 as is feasible. Absolute no-doubt is to get a-e implemented. f might not have a clever solution, h is easy to implement and hard to test completely (for not that much benefit in my mind), i just isn't that important, and j is very nebulous, and generally not all that helpful. - Week 8-10: Write first Windmill test for admin. Determine the 'environment' or set of Models we will use to test the Admin. The goal here is not to have the entire admin interface covered, it is to encapsulate ChangeList (possibly with actions, def with filter/drilldown/search) and basic object adding/deleting (including all Field types, but probably not Inline edits). Then its just some Test mania! Windmill test writing fury. My plan is to develop a best-practice for Windmill tests, as well as a strong repository of examples. - Week 10-12: Discuss missing tests or other cases to check. Possibly provide branch of tests for new admin work (if that project is accepted for GSOC). Implement any of item 4[a:f] that didn't get implemented, as well as inline edits in the admin tests (if I'm on/ahead of schedule). On Fri, Apr 24, 2009 at 11:05 AM, Jacob Kaplan-Moss < [email protected]> wrote: > > Hi folks -- > > It's come to my attention that Google doesn't make the accepted > proposals readable by the general public. That's annoying to folks who > are just now tuning in to SoC. > > So please in today's update post your full proposal, modulo any > modifications made based on our comments during the proposal process. > You can just post the whole thing, but better would be a link to some > permanent home for it elsewhere on the web. > > If you've already posted something -- Alex, Honza, y'all rock -- > please do follow up with the proposal. > > If you haven't... clock's ticking, guys :) > > Thanks! > > Jacob > > > -- Kevin Kubasik http://kubasik.net/blog -- Kevin Kubasik http://kubasik.net/blog --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
