Django Template Compilation =========================== About Me ~~~~~~~~
I'm a sophomore computer science student at Rensselaer Polytechnic Institute. I'm a frequent contributor to Django (including last year's successful multiple database GSoC project) and other related projects; I'm also a commiter on both `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_ and `PyPy <http://codespeak.net/pypy/dist/pypy/doc/>`_. Background ~~~~~~~~~~ I've spent quite a bit of time reviewing alternate template implementations, particularly Spitfire and Jinja, as well as speaking with people like Armin Ronacher (the author of Jinja). I'm also involved in multiple alternative VMs for Python, so I'm extremely familiar with the performance characteristics of various Python operations. Plan ~~~~ Compile Django templates into Python functions. Rationale ~~~~~~~~~ Currently the Django template language exists at a level above the Python interpreter, and interprets Django templates. This leads to Django templates being particularly slow compared to implementations of template languages that compile templates into Python code. Method ~~~~~~ Templates will be compiled by turning each template into a series of functions, one per block (note that the base template from which other templates extend is a single function, not one per block). This is accomplished by taking the tree of ``Node`` objects which currently exist for a template and translating it into an alternate representation that more closely mirrors the structure of Python code, but which still has the semantics of the template language. For example, the new tree for a loop using the ``{% for %}`` tag would become a for loop in Python, plus assignments to set up the ``{{ forloop }}`` variable that the ``{% for %}`` tag provides. The semantics of Python code is that variables assigned in a for loop exist beyond the loop itself, including the looping variable. Django templates, however, pop the top layer from the context stack at the end of a for loop. This intermediate representation uses the scoping of Django templates. After an intermediate representation is created a compiler is invoked which translates the IR into Python code. This handles the details of Django template scoping, spilling variables in the event of conflicts, and calling template tag functions. An important feature of Django templates is that users can write template tags which have access to the full context, including the ability to modify the context. In order to maintain backwards compatibility with existing template tags, we must create a template context object whenever an uncompilable template tag is used, and mirror any changes made to the context in the function's locals. This presents a complication, as we forfeit the speed benefits of a compiled template (lookups of a Python local are a single index in a C array) and must perform a dictionary lookup for every variable. Unfortunately, mirroring a context dictionary back into local variables requires maintaining a dictionary of arbitrary names to values, which can't be efficiently implemented with Python's locals (use of ``exec`` causes locals to degrade to dictionary lookups). Furthermore, constructing a dictionary of the full context requires additional effort. To provide an optimal solution we must know which variables a given template tag needs, and which variables it can mutate. This can be accomplished by attaching a new class attribute to ``Nodes`` and passing only those values to the class, (instead of the full context dictionary). Subsequently, we would only need to mirror a few given values into the locals, and since these are known names, we avoid degrading the local lookups into dictionaries. Old-style ``Nodes`` will continue to work, but in a less efficient manner. Alternate ideas --------------- James Bennett has suggested making template compilation something that is opt-in by use of a custom template loader. The major advantage of this approach is that we can require all ``Nodes`` to specify the names they use, and thus not have to deal with mirroring the locals back and forth, which drastically reduces the possibility of performance regressions. I am in favor of going with this proposal as it gives us a clear transition between the current implementation and the future (see next section). Long Term --------- In the short term almost all template tags will be old-style (consider the number of tags using the "as X" idiom to modify the context). However, a long term goal should be to move to exclusively compiled template tags. This has several benefits: * Fewer codepaths to maintain * No chance for differing semantics (this follows from the first) * More consistant performance. One of the critical goals of this project will therefore be to develop the API for template tags in a manner where old-style tags can easily migrate to the new form. Examples ~~~~~~~~ The following are some examples of what I'd expect a compiled template to look like: .. sourcecode:: html+django {% for i in my_list %} {% if i|divisibleby:2 == 0 %} {{ i }} {% endif %} {% endfor %} .. sourcecode:: python def templ(context, divisibleby=divisibleby): my_list = context.get("my_list") _loop_len = len(my_list) result = [] for forloop, i in enumerate(my_list): forloop = { "counter0": forloop, "counter": forloop+1, "revcounter": _loop_len - i, "revcounter0": _loop_len - i - 1, "first": i == 0, "last": (i == _loop_len - 1), } if divisibleby(i, 2) == 0: result.append(force_unicode(i)) return "".join(result) For comparison here is the performnace of these 2:: >>> %timeit t.render(Context({"my_list": range(1000)})) 10 loops, best of 3: 38.2 ms per loop >>> %timeit templ(Context({"my_list": range(1000)})) 100 loops, best of 3: 3.63 ms per loop That's a 10-fold improvement! Timeline ~~~~~~~~ * 1 week -- develop a benchmark suite of templates, composed of real world templates as well as microbenchmarks, as well as simple scripts for regular testing of compatibility and performance. On this particular point I'm hoping the community will be able to help with providing example templates and fixture data for benchmarking and compatibility testing. * 3 weeks -- develop the frontend portion of this, code which translates Django's included template tags into the IR. * 1 week -- developing the internal IR generation API. * 2 weeks -- hooking up all of Django's template tags to actually use it. * 5 weeks -- develop the backend code generator. This takes the IR and translates it into Python, including handling the semantic changes. * 2 weeks -- basic code generation support. Does nothing but generate code that looks exactly like what's already executed, this means variable lookups are still lookups in a ``Context`` dictionary. * 3 weeks -- optimize known names into local variables at the python level, based on speaking with Armin Ronacher this is critical to getting good performance. This basically involves implementing something similar to a compiler's register allocator, except we aren't constrained by number of names, but rather by what the names are. * 2 weeks -- time set aside for dealing with bugs, corner cases, and anything else. * 1 week -- Explore possibility for additional optimizations, eliminating duplicate values (for example removing unused ``{{ forloop }}`` variables), allowing an external app to provide "type" data to IR nodes such that variable lookups could be resolved as indexing vs attribute lookup at compile time. Goals ~~~~~ As with any good project we need some criteria by which to measure success: * Successfully compile complete (real world) templates. * Speed up templates. For reference purposes, Jinja2 is about 10-20x faster than Django templates. My goal is to come within a factor of 2-3 of this. * Complete backwards compatibility. * Develop a complete porting guide for old-style template tags to minimize any pain in the transition. Any thoughts, feedback, cometary, Nobel prize nominations, and death threats are welcome! Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
