Let me start by saying I'm very sorry I didn't post my proposal on django-devs earlier - attending a local conference and finishing my master thesis sucked most of my awake time during the last 2 weeks.
This proposal is written in the reStructuredText format, but I created a HTML version for convenience at http://gsoc2009.petarmaric.com/ ---------------------------------------------------------------------------------------- Abstract ======== My name is Petar Marić, a long time Djangonaut with an idea how to improve a somewhat neglected part of Django - distributing static (media) files with reusable Django apps and problems with serving those media files. This is done by introducing new conventions, changing parts of the Django development Web server and creating a new ``syncmedia`` management command. All of the proposed changes should be backwards compatible. Why? ==== Currently there are some problems with distributing and using media files that come with reusable Django apps: * Lack of conventions and official documentation on how to distribute media files with reusable Django apps. * Admin app automagically serves its media files during development, while ignoring the same issue for other apps. * Lack of web directory index for the admin app media files during development, because of the way it's special-cased. * Inconsistency in serving admin apps media files during development and in the production environment (primarily because of the special-casing). * The app media has to be placed manually into the ``settings.MEDIA_ROOT`` creating an opportunity for user errors. * No clear convention how to do the project level override of an app distributed media file. The goal of this proposal is to greatly simplify the end-developer experience when working with reusable Django apps which distribute their own default media. Use cases ~~~~~~~~~ Static file serving during development -------------------------------------- Uroš, a long time Django developer, created a reusable blogging Django app named ``teslablog`` which uses a JavaScript powered WYSIWYM editor for creating posts. It also relies on the Django admin app for most of its functions. Dejan, a seasonal Python developer, is fed up with writing "Hello world" Django blog apps and wants to reuse a tested one for his own personal blog. ``teslablog`` app is just about right, so Dejan installs it to his Django blog project. Dejan was really excited to try out the blogging app, but when he tried to create a new blog post the awesome WYSIWYM editor seen on the screenshots is nowhere to be found. Strange, thought Dejan, lemme check the server logs. It seems that the HTTP requests for the JavaScript and CSS files required by the WYSIWYM editor all resulted in 404s. Dejan went on to the `#django IRC channel`_ for help, where he was told Django development Web server `can serve media files`_ and given a tip to add the following to his root URLconf:: from django.conf import settings # ... if settings.DEBUG: tesla_dir = os.path.dirname(__import__(teslablog).__file__) urlpatterns += patterns('', ( r'^media/tesla_media/(?P<path>.*)$', 'django.views.static.serve', {'document_root': os.path.join(telsa_dir, 'media')} ), ) After his first successful post Dejan took a moment to wonder, but how does the admin media get served when there's no entry for it in the URLconf? While this interested him, he was itching to get his personal blog up and running. Soon he forgot all about the magic admin app media serving... .. _#django IRC channel: irc://irc.freenode.net/django .. _can serve media files: http://docs.djangoproject.com/en/dev/howto/static-files/ Deploying to a \*nix based production server -------------------------------------------- After some time Dejan was happy with his personal blog, so he decided to upload it to his \*nix powered production server. He already knew that serving ``teslablog`` media files could be a problem, so he asked the community what is the best way to do it. Although they gave him several solutions, there was one he really liked - just create a symbolic link from ``teslablog/media`` to ``settings.MEDIA_ROOT/tesla_media``. That way I won't have problems when some ``teslablog`` media file changes, he thought. Satisfied, he went to login and create his first post - but when the login form loaded he noticed that now the admin was stripped naked! What happened now, he pondered, and checked the logs. It was clear as day the server was coughing out 404s for media files used by admin. But it worked in development, Dejan yelled, why is this happening? After seeking help from the community he learned that he must manually point the Web server to admins app media files. So he created another symbolic link, this time from ``django/contrib/admin/media`` to ``settings.ADMIN_MEDIA_PREFIX``. While doing this Dejan realised he had to do this for **every** reusable app which is distributing default media files and on **every** production project - so he felt annoyed. Deploying to a Windows based production server ---------------------------------------------- With his blog up and running Dejan was happily posting away, learning Django and forgetting the media files drudgery he encountered some time ago. And then Sonja, his friend, saw the blog and thought it was so cool that she asked if he could create a similar blog for her. Of course I'll do that for you, Dejan said, just get some hosting that supports Django, and he started looking for recommendations. Sonja then interrupted him, and said she already bought hosting a while ago. So they contacted her hosting's support and found that Django is indeed supported. But at the time there was a catch Dejan knew nothing about - the server ran Windows. Bah, Windows/\*nix - Python doesn't care, he thought and continued deploying. And when the media issue arose he pondered, does Windows have symbolic links? After some reading he learned of `NTFS reparse points`_, but the hosting company banned their use on their servers. Agitated he decided to copy the media files instead of linking them, and hoped that he'll remember to overwrite target media files when those distributed with apps he used, change. .. _NTFS reparse points: http://en.wikipedia.org/wiki/NTFS_reparse_point Proposal ======== .. _proposed-conventions: Proposed conventions ~~~~~~~~~~~~~~~~~~~~ Django apps not distributing their own default media files keep their directory structure as-is. To distribute default media files with your Django app place the files inside of the app's ``media`` subdirectory. Your Django app should expect its media files accessible under the ``settings.MEDIA_URL/<app_label>`` URL, where: * ``<app_label>`` is only the *last* part of the full Python path to your app. For example, if your app's Python import path is ``django.contrib.admin``, ``<app_label>`` would be ``admin``. For example, media declaration of the ``django.contrib.admin.widgets.FilteredSelectMultiple`` widget:: class FilteredSelectMultiple(forms.SelectMultiple): class Media: js = (settings.ADMIN_MEDIA_PREFIX + "js/core.js", settings.ADMIN_MEDIA_PREFIX + "js/SelectBox.js", settings.ADMIN_MEDIA_PREFIX + "js/SelectFilter2.js") # ... ... would change to:: class FilteredSelectMultiple(forms.SelectMultiple): class Media: js = ("admin/js/core.js", "admin/js/SelectBox.js", "admin/js/SelectFilter2.js") # ... And the ``django/contrib/admin/templates/admin/change_form.html`` template: .. sourcecode:: html+django {% extends "admin/base_site.html" %} {% load i18n admin_modify adminmedia %} {# ... #} {% block extrastyle %} {{ block.super }} <link rel="stylesheet" type="text/css" href="{% admin_media_prefix %}css/forms.css" /> {% endblock %} {# ... #} ... would change to: .. sourcecode:: html+django {% extends "admin/base_site.html" %} {% load i18n admin_modify %} {# ... #} {% block extrastyle %} {{ block.super }} <link rel="stylesheet" type="text/css" href="{{ MEDIA_URL }}admin/css/forms.css" /> {% endblock %} {# ... #} In effect this removes the need for the ``ADMIN_MEDIA_PREFIX`` setting. If there's a need for the project level override of an app media file just place it under the ``settings.MEDIA_ROOT/<app_label>/<override-file-path>`` directory and Django will use your version instead off that distributed with the app. For example, if you wish to override the ``django/contrib/admin/media/js/core.js`` file you should place your version under ``settings.MEDIA_ROOT/admin/js/core.js``. Simpler static file serving during development ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instead of treating media file serving for the admin app as a special case I propose that the Django development Web server should try to automagically serve media files for all Django apps as per :ref:`proposed-conventions`. For example if the Django development Web server got a HTTP request for ``settings.MEDIA_URL/admin/js/core.js`` it would (simplified): #. Try to find the file ``settings.MEDIA_ROOT/admin/js/core.js`` and serve it. If the file couldn't be found it would proceed to the next step. #. Try to find the file ``django/contrib/admin/media/js/core.js`` and serve it. If the file couldn't be found it would proceed to the next step. #. Raise a ``django.http.Http404`` exception. I would completely remove the ``django.core.servers.basehttp.AdminMediaHandler`` WSGI middleware as there would no longer be any need for it and I'd probably create a new view (or a URLconf, haven't decided yet) that automagically serves static files placed under ``settings.MEDIA_ROOT`` as well as media files for all Django apps. There are `already`_ `some`_ `attempts`_ at this, but they have some flaws I hope to fix: #. Don't cache the list of apps that distribute their own media files. #. Either don't support app media overriding or doing it badly. #. No option to display file listings for directories. #. Not a part of Django. Thus, very few people know about them. .. _already: http://code.djangoproject.com/ticket/2557 .. _some: http://www.djangosnippets.org/snippets/943/ .. _attempts: http://www.djangosnippets.org/snippets/985/ Django ``syncmedia`` management command ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `Arne Brodowski`_ has an interesting idea, and the article is very good - so I strongly advise you to read it because I'm basing a part of my work on his article. Now, there are some things I don't like about his approach: * Works only on \*nix based operating systems. I mostly use Windows for my work and I don't like the "screw you" (metaphorically) exception I get. * It only handles symbolic links. Some version of Windows don't do that well. Also, user may be denied the right to create NTFS reparse points - command should fail and notify the user to either do it manually or use different sync options. * Is fired by the ``post_syncdb`` signal. I believe that a Django management command is better suited for this task because ``syncdb`` and ``syncmedia`` are basically orthogonal. * It shouldn't be run during development. It can just create problems for people using version control - i.e. you don't want admin media files together with your project in svn? Right? There are 2 ways to place app's media files in ``settings.MEDIA_ROOT``: #. (Symbolic) linking. Windows is somewhat problematic, but if NTFS is the underlying file system an appropriate NTFS reparse point can *maybe* be used. Otherwise, fail and notify the user to either do it manually or use different sync options. Also, it should check if any of the target directories already exist and are not links - should fail and notify the user linking is not possible and to either do it manually or use different sync options. #. Copying. The problem with this one is file overwriting, so multiple overwrite strategies have to be provided: #. Delete the target directory first and then copy everything – removes stale files. Could destroy project level overrides of app media files (no way of knowing), should come with a big warning and a "Are you sure?" prompt. #. Copy and overwrite all files #. Copy only new files, no overwriting #. Copy all files, overwrite only files with an older timestamp .. _Arne Brodowski: http://www.arnebrodowski.de/blog/distributing-mediafiles-with-django-apps.html Timeline ======== I should be able to work on this project about 20-30 hours a week, and this is my plan so far: April 21st - May 23rd: * Discussing the conventions, features, ponies and edge cases in detail with the community to improve the quality and probability of inclusion into Django. Week 1-3 - simplify static file serving during development: * Remove the special-cased magic admin app media serving from the Django development Web server. * Add support in the Django development Web server to automagically serve media files for all Django apps. * Add caching of the list of apps that distribute their own media files. * Add support for app media overriding. Week 4-6 - Django ``syncmedia`` management command: * Creating a basic version of the ``syncmedia`` management command - only the basic "nuke and then copy" strategy implemented. * Add other copy strategies. * Add generic linking support - implement only for \*nix. * Add sane automatic fallbacks. * Improve error handling. * Tune the ``syncmedia`` default settings and options. Week 7-10 - improve Windows support: * A more detailed research of NTFS reparse points and their availability under different Windows versions and editions. * Implement linking in Windows for the ``syncmedia`` command, create a separate Python library if needed. * Tune the ``syncmedia`` default settings and options for Windows. Week 11 - documentation: * Document the new conventions and update relevant documentation. Week 12 - ponies: * Implement remaining ponies, if any. About me ======== I'm a 24 year old applied computer science graduate student at the `Faculty of Technical Sciences`_, `University of Novi Sad`_, Serbia. I'm in the final stages (review, fix, repeat) of my `Django powered master thesis`_ (alpha code), and expect to have a 9.11 average grade (scale 5-10) upon my graduation. I've lead student team projects on *Software Design*, *Business Information Systems*, *Net-Centric Computing*, *Software Specification and Modelling*, *Computer Graphics and Multimedia Systems* and all team members got maximum credits on them. I'm a member of `EESTEC LC Novi Sad`_, `IEEE`_ Student Branch Novi Sad and a frequent attender of local conferences (where I gave a few lectures). I'm also an active member of the `DevProTalk forum`_ and the Django developers/i18n mailing lists. Over the years I've learned, used and dropped many programming languages until I heard of Python - which made programming **fun again**. I still learn new languages, mostly for the fun of it - next in line are C#, Lisp and mastering JavaScript. I've been using Django for a long time now, since August 2005. The trunk always was and will continue to be the only true version for me. I track Django code changes on a daily basis, and try to track the community feed and the developers mailing list at least twice a week. Due to my studies I was unable to contribute to Django and participate in the community as much as I'd like. Hope this changes once I graduate. My areas of interest include the `open-source movement`_, software engineering, web development, model driven development, data/information visualisation, OLAP-like systems, AI and game programming. .. _Faculty of Technical Sciences: http://www.ftn.ns.ac.rs/ .. _University of Novi Sad: http://www.ns.ac.rs/ .. _Django powered master thesis: http://code.google.com/p/opinion-extractor/ .. _EESTEC LC Novi Sad: http://www.eestec-ns.org.yu/ .. _IEEE: http://www.ieee.org/ .. _DevProTalk forum: http://www.devprotalk.com/ .. _open-source movement: https://www.ohloh.net/accounts/petar -- Petar Marić *e-mail: petar.ma...@gmail.com *mobile: +381 (64) 6122467 *icq: 224720322 *jabber: petar.ma...@gmail.com *web: http://www.petarmaric.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
