Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

Current status: The freezing of free variables into a context
dictionary for includes and extension is currently not working as
expected.  The reason is obvious but the solution is not entirely
clear to me yet.  I need to track what variables were assigned to
until a specific statement, not only what variables will be assigned
or referenced for a block.  I don't like to do that while transforming
the AST, but walking that another time sounds like a horrible idea was
well.  A bit torn, I might just use the current identifier tracking
code and also record the id of the node that triggers the assignment.
Then I could reconstruct the state before each statement I think.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

Status of this week: I spend the last week mostly trying to figure out how 
to debug a bunch of annoying issues and due to the slow progress I decided 
on doing an AST -> Python code translation for debugging purposes. The 
identifier tracking method I naively started using initially (which is 
assuming that there are no branches and recording identifiers for each scope 
while the transformer is creating code) turns out to be by far not good 
enough as it misses identifiers that are referenced after blocks. I will 
need to heavily improve on that.

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/mITAVQia2fEJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

Progress update.  This week I was starting the work on the compiler
and I have at least for Django's use case most compilation things
ironed out.  However it turns out that compiling AST nodes made from
scratch is a painful process as the interpreter will randomly segfault
or fail compiling with useless error messages.  For debugging the AST
I wrote a bunch of small helpers but the current HEAD revision
exhibits some weird behavior I have not found the cause for yet.
Generally however good progress for a week full of EuroPython :)

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

On Jun 17, 2:18 pm, Anatoly Vostryakov  wrote:
>   I've thought today. May be it will possible to add support of
> parameters for functions or object methods in django templates.
> I mean, something like this in django template:
The compiler is able to handle this case, but the syntax of the Django
templating language does not.  I will not change any features in the
Django template engine during this proposal and actually reused most
of the parser toolchain until fall.  If someone wants to step up and
continue improving the features on the language after GSOC I am happy
to assist.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

I'm in between two conferences right now and university and a bunch of
other stuff needs attention as well so progress is slowed down quite a
bit right now.  Will however catch up next week with what I cannot do
this one.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template inheritance and duplicated content.

[Armin Ronacher]

Hi,


On May 24, 10:58 am, Jonathan Slenders 
wrote:
> I guess this is a flaw in the inheritance algorithm, only the most
> outer blocks should be used during the resolving of inheritance.
How else would it work?  Think of a block like a function plus the
execution of that function. By placing a new block within a block you
want that behavior and if you override a block and in that another
block that already existed both Jinja2 and Django will have to also
immediately call it. There is only one rule: if you don't like that,
make the block toplevel. The only alternative sane semantic would be
raising a runtime error, but I don't think that is a good idea.

Regards,
That Jinja guy

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

Weekly progress report: I started on documenting some of the behavior
that the current django template engine has and how it affects the
compilation. The nodes are mostly implemented and some of them can be
interpreted already. I want to finish the interpreter first because
that way we can easily figure out if it's compatible with the existing
interpreter and if it is, start on compiling it down to AST nodes and
from that into actual Python bytecode.

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

The progress went slower than expected because I had to analyze some
more runtime differences between Django and Jinja2 which I missed at
first. The main problem are changes in the safe-string behavior but I
think I found a possible solution to counter that. I had to diverge
from my previous plan to keep the nodes unchanged from Jinja2 so at
that point it's pretty sure that Jinja2 will not survive this GSOC
without breaking backwards compat on it's extension interface either
which now simplify further plannings.

First pieces of currently meaningless code are checked in, so is an
OBSERVATIONS file which has some notes on internals. I will update
this file with more internal notes as the project progresses.

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

The repository with my changes will be here:
  https://github.com/mitsuhiko/django-template-compilation

As this proposal implements a 'newtemplates' module in Django instead
of modifying the existing one until it's proven to be compatible in
every single regard, I will have it as a separate repository. The idea
is that it can then easily be tested with other changes in Django
itself.

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

[GSOC] Template Engine Compilation and Runtime Refactoring Progress

[Armin Ronacher]

Hi,

This is the first weekly status update of many to come. Just a quick
overview of this project and myself.

My nickname on freenode is 'mitsuhiko' and this is the preferred form
to reach me in general. Alternatively my twitter account with the same
name is a good way of reaching me even when I'm not at my computer, so
is mail.

As the template system in Django is already a pretty isolated system
by itself it shouldn't be too hard to keep the work nicely merge-able.
I have not yet decided if I will keep this branch on github or
somewhere else. The Jinja2/Django maintenance aspect of this will be
decided at a late point, probably even just after GSOC so it can be
ignored for the duration of GSOC.

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal for Template Engine Refactoring

[Armin Ronacher]

Hi,

On Mar 29, 3:54 pm, Jacob Kaplan-Moss  wrote:
> Can you speak a little more about how you'd see the workflow working
> with this new library? It seems a bit complex to have a library that's
> *copied* between Django and Jinja: I'd worry about patches getting
> lost.
I think that problem can mostly be avoided. The interpreter /
compiler / ast combo library is probably very minimal in size. I don't
see a big problem having to copy them into the Django svn every once
in a while. That could even be automated.

> For context, this has happened a few times with the simplejson and
> elementtree libraries in core Python: patches made to the upstream got
> "lost" before merging into Python's stdlib, and patches made against
> the stdlib didn't make it upstream.
stdlib is a more complex problem because it's tightly tied into the
Python release cycle and has a separate set of developers. Even if I'm
not a django developer I am never far from the project so it will be
the same developer doing it during GSOC and afterwards. The extra work
involved in making this work with both Jinja2 and Django and merging
every once in a while should be minimal. The only problem would be
other people doing changes to the codebase directly in the Django SVN.
Which is why I would recommend having a dedicated separate repository
for this thing (might as well be subversion if you want) where I and
all Django developers have access for bugfixes if necessary. And also,
once that thing is stable I don't think there is a lot to be done on
the actual compiler. The Django specific stuff will not end up in that
backend anyways.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal for Template Engine Refactoring

[Armin Ronacher]

Hi,

On Mar 29, 2:23 pm, Jonathan S  wrote:
> Are you planning on keeping the API for registering template tags
> compatible?  (django.template.Library.register.tag)
It will be supported because otherwise upgrading won't be possible,
but how well this works in detail would have to be investigated and
what a cleaner implementation looks like. Jinja2 has an expression
parser and encourages people to create a AST that does what they want
in extensions. However in Jinja2 I never encouraged people to write
custom tags so the API is very limited there.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal for Template Engine Refactoring

[Armin Ronacher]

Great. Google groups decided to reflow my mail. Here in actually
readable version: http://paste.pocoo.org/show/362024/

Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Proposal for Template Engine Refactoring

[Armin Ronacher]

Hi,

I would like to participate in this year's Google Summer of Code by
introducing a new backend for the Django (and Jinja2) templating
language.
As the author of Jinja2 I recently decided to rewrite the code
generation
of the engine to better support alternative Python implementations
like
PyPy and restricted environments like Google's Appengine.

The Jinja2 backend is already usable for other templating engines as
well
for as long as the sematics are compatible with the ones from Jinja2.
However with a better backend design the Django semantics should be
possible to implement in there as well.

As such as I am proposing to keep this backend independent of either
Django and Jinja2.  I would maintain inside a separate repository and
with
each release of Jinja2 and Django port fixes into their respective
code
bases.  This could also be used as a basis for other template engines
or
domain specific languages if necessary.

The reason I don't want to keep this a separate package but actually
copy
the code over to Django and Jinja2 are twofold.  First of all I don't
want
to introduce a new dependency for the template engines themselves.
Secondly I really want to keep this an implementation detail for
Jinja2 at
the very least.  In Django we don't want to see dependencies either.

Reasons (coming from the current Jinja2 backend) for a new version:

1.  implement a drop-in interpreter as alternative to the template
compilation for better error reporting during development.
2.  implement an ast.py based compiler in order to avoid the debug
traceback hackery that is currently in place to support debugging
of
templates with accurate line numbers.
3.  Better support for jit interpreted Python installations like PyPy
by
removing the need to call locals() or sys._getframe which is
currently
required by Jinja2 in order to implement includes.
4.  Python generating compiled for GAE.  This one would not have the
same
error handling capabilities as the interpreter and the AST based
compiler, but has the advantage of running everywhere and would
allow
module based importing like the one currently in use in Jinja2.

Reasons why this is interesting for Django:

1.  It would greatly speed up execution of core tags.  The idea is to
use
code generation for the builtin tags and to generate a context as
necessary when droppping into old-style custom tags.
2.  Provide a new API that can be used for new tags that allow code
generation behind the scenes.
3.  Better debugging capabilties by providing actual Python tracebacks
for
errors in templates.

How much speedup will we see in Django?  Probably not as much as
people
are hoping as Django's current semantics just cannot be sped up to
Jinja2
levels by keeping the extension API unchanged.  However because there
are
already some abstractions in place we can probably achieve a
significant
speedup for certain common template types that it reduces the need to
switch to Jinja2.

Why am I proposing this as a GSOC Django project instead of hacking on
this under the umbrella of the PSF for Jinja2 alone?  First of all,
speeding up Jinja2 is a boring task right now as it is very close to
raw
Python performance already.  Secondly there was a proposal by Alex
last
year and there is already a proposal here about template engine
compilation backends and if I am going to do this for Jinja2 anyways,
I
can keep it generic enough to also support Django.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

#10216 - TEMPLATE_DEBUG / TemplateSyntaxError handling doesn't play nice with Jinja2

[Armin Ronacher]

Hello everybody,

I would love to propose a new solution for ticket #10216 which is
currently a wontfix. Basically the problem is that with TEMPLATE_DEBUG
turned on, Django expects a certain protocol from exceptions catched
by the debugging system:

Either an exception does not have a .source attribute, or it has one
and this is a tuple that can be unpacked where one of the items is an
template loader origin and the second is the position in the template,
as far as I can see that.

Now the direct problem for me is that people get no proper debug
output in Django if Jinja 2 raise an exception, because the .source
attribute of a Jinja 2 exception has a different protocol: it's a
string. Of course I can work around that problem in Jinja and tell
people to disable TEMPLATE_DEBUG, but I think the issue is not limited
to Jinja 2.

The purpose of a debugging middleware is to show what went wrong.
Currently the debugging middleware does the opposite if you have any
exception with a source attribute that is not a template origin and
position information.

A quick search on google code showed me that .source is not an
uncommon attribute on exceptions. For example PyXML uses it in the
xpath for SyntaxException and internal exceptions. If any of these
would be raised in a django application, the user would experience a
broken debug view. Not sure how common PyQwt is in web applications
but that would have the same problem ;)

I would like to propose a different solution: do an instance check
against a django TemplateSyntaxError (or any other template exception
that has these .source attributes). That should be easy to accomplish
and the debug system already has a dependency on the template engine
anyways for the rendering of the UI.


Regards,
Armin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: r9766-related issues

[Armin Ronacher]

Hi,

On 13 Mai, 00:42, Yuri Baburov  wrote:
> Also on OS X you can set if filename is case-sensitive on per-volume
> basis, when formatting, and usually it's case-insensitive. windows is
> always case insensitive, linux is usually case-sensitive.
I'm not talking about case sensitivity here but unicode normalization
which is controlled in the IO system. For example if you're creating a
file öäü.txt (\x94\x84\x81) OS X will store it as o\u0308a\u0308u
\u0308.txt on the filesystem. No matter if the filesystem is HSF, NFS,
a mounted samba share or anything else. On Linux the filesystem might
have an encoding but is generally non unicode-aware and no
normalization takes place at all. This could lead to weird effects
where you have \x94\x84\x81.txt in the database but o\u0308a\u0308u
\u0308.txt on the filesystem which obviously makes it impossible to
move the data from an OS X system to a linux one or access an NFS
share from two systems at the same time.

> This does matter when one filename has the same letters as another but
> some letters have different case. This also can cause troubles when
> moving existing DB and files from one filesystem to another.
I'm well aware of that, but that's a different story.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Armin Ronacher]

Hi,

On 11 Mai, 15:03, Marty Alchin  wrote:
> If you're using the filename to store it somewhere else, typically for
> denormalization, it'd be better to do that post-save, since then you
> know the record actually got saved in the database. Otherwise, you
> might be trying to access the content of the file, which would be
> better using the direct file access, which can often save yourself a
> round-trip to the storage backend (which can be a big win for paid
> storage solutions, like S3).
I was reading about #8593 today and I think this adds a new layer of
complexity to the problem.  Some operating systems perform filename
normalization on their own.  While the case covered in the bug only
affects Windows machines and is a weird behavior in the Python ntpath
module instead of the Kernel, it could cause some troubles.

On an OS X machine the operating system will normalize the filename to
some sort of unicode normalization form.  (The exact one depends on
the version of OS X you're running).  This translation happens in a
way that you will not notice it from inside your application. The
normalization however happens in the IO system and not the HSF file
system.  Now imagine you're writing to an NFS share that is also
accessed from linux.  If they two would use the same database as well,
the Linux version would not be able to locate the file on the
filesystem because the entry in the database would be only valid if
the OS does the normalization again when opening the file.

This sounds unrelated to the bug we're discussing here, but it raises
the problem that the value that ends up in the database is probably
not exactly the value the real name of the file. Maybe there should be
a setting in django to switch the filename generation for the uploaded
files to something pre-normalized inside django to make the filename
more reliable. (Maybe go as far as lowercase and ASCII only?).  Either
way it shows that the filename on the file system is unreliable for
pretty much everything except loading and replacing the file. On file
creation the code can already "rename" the file to avoid clashes.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Armin Ronacher]

Hi,

All bugs are fixed now except for #10788.  Now the problem with
closing that one is that this one requires a design descision.  I
updated the ticket accordingly for jacob or anyone else to decide on
it.  My personal opinion is that I consider it bad design for the
application to depend on the final filename of an uploaded file on the
filesystem.  This makes it nearly impossible to create "atomic"
uploads for unique filenames.

Regards,
Armin

Links:
  http://code.djangoproject.com/ticket/10788
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Armin Ronacher]

Heyho,

On May 7, 10:01 pm, Marty Alchin  wrote:
> It's not explicitly related to the MRO and method stuff you and Alex
> have been working on, but it's definitely related to the r9766
> discussion, since it's caused by the delayed saving. I have a clear
> understanding of the problem, but I don't have time to write it up at
> the moment, so I'll try to catch you guys on IRC if you're still up
> when I get home.
Florian Apolloner just wrote a patch for that and attached it to the
ticket.

> That only solves the problem if you're trying to access the file's
> contents, which I'll admit is the majority case. Other apps may need
> the filename for other reasons, though, and it was available in 1.0,
> so the current behavior is backwards-incompatible. Of course, all the
> uses I can think of would actually be better as post_save handlers
> anyway, but the fact is that compatibility was broken, so we should
> see if we can restore it.
There are not many situations where you would need the filename of the
thing in the pre-save command.  If we can collect some situations
where this would be useful it makes sense to think about it.  I
personally think it was a design mistake in the first place and should
be changed to the behavior it's showing currently.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Armin Ronacher]

Hi,

On May 7, 5:37 pm, Karen Tracey  wrote:
> #10249: can't create consistent MRO (method resolution order) when assigning
> a File to a FileField.
This is fixed.

> #10300: custom storage backend can't get length of content to save.
This *should* be fixed.  I can't test it, no access to S3.

> #10404: height_field and width_field for an ImageField don't always work.
This is not really related to that and I have a hard time
understanding that part of the ORM, but it should be fixable.

> #10788: actual file name not known until after model save.
That's the problem with the new implementation.  The final filename is
decided when the file is actually saved.  The cleaner way is to use
the assigned file and directly read from it when it's still in the
temporary location or memory.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Armin Ronacher]

Hi,

On May 7, 5:37 pm, Karen Tracey  wrote:
> So far as I know there 4 open ticks remaining related to r9766.  Three are
> regressions so I believe something really needs to be done about them before
> 1.1; one I think is just a bug in the new function.  Personally I'd rather
> not revert the new function at this point (it's been available in the
> alpha/beta releases, it would be nasty to pull it at this point).  However,
> reverting the part of r9766 that delayed saving the file to the backend
> until model save (in the case where a ModelForm is used to create a model)
> may be necessary to fix the last of the four problems, unless we want to
> document that one as a backwards-incompatible change.
I'm working with Alex on that right now here in Prague.  We have some
branches on github related to that.  Basically the idea is to start
with getting rid of some of the over engineering in the abstract base
classes and make sure the subclasses call the constructors properly.
Once that works we should be able to fix all the "method missing on
temporary file" tickets that are currently just worked around.  The
fixing of the problems caused by the changeset will be covered by that
branch as well.

> The four tickets I know of are:
>
> #10249:
> #10300:
> #10404:
> #10788:
We're working on that.  We'll report back some status when we get the
existing testcases passing again :)


Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Concrete django.template Suggestions

[Armin Ronacher]

Hi everybody,

After my quite controversal blog post about Jinja and Django I summed
up some suggestions for the Django template engine that don't require
a complete reimplementation / breaking backwards compatibility :)

First of all the thread in-safeties I discovered [and the explanation
why nobody noticed so far].  First of all the explanation, because
that's probably more interesting: Per default templates are parsed and
evaluated each request so you will never notice that.  However I've
seen enough applications by now that keep the template objects in
memory where these problems become visible.  Why would anyone want to
not parse the templates each request?  Because it's a lot faster.  How
much?  ~100 req/sec versus 60 req/sec for the file browser page on
bitbucket, and that's the overall request time including all of the
request handling, http header parsing and of course database.

Where are threading bugs?  Everywhere, where a tag render method
somehow modifies the node object.  Where does this happen?  From what
I've seen so far it's done in "block", "ifchanged", "cycle" and
"extends".  Now, of course the attribute assignments happen for a
reason: they are necessary, at least for cycle and ifchanged.  Block
and extends look like accidents, they could be solved a lot cleaner
(block would put a reference to a hypothetical SuperBlock into the
context, rather than a reference to itself) and extends doesn't have
to store the evaluated value on the node at all, a local variable is
just fine.  Ifchanged and cycle however have to somewhere store the
information, and I recommend using the context object for that.  The
context could get `get_state_var`, `set_state_var` functions that can
be used to store node related information from one render call to the
next::

def get_state_var(self, node, name, default=None):
storage = self._state_vars.get(id(node), None)
if storage is not None and name in storage:
return storage[name]
return default

def set_state_var(self, node, name, value):
self._state_vars.setdefault(id(node), {})[name] = value

The cycle node could then look like this::

class CycleNode(Node):
def __init__(self, cyclevars, variable_name=None):
self.cycle_vars = map(Variable, cyclevars)
self.variable_name = variable_name

def render(self, context):
pos = context.get_state_var(self, 'pos', 0)
value = self.cycle_iter[pos].resolve(context)
context.set_state_var(self, 'pos', (pos + 1) %
len(self.cycle_vars))
if self.variable_name:
context[self.variable_name] = value
return value

This used in all core tags makes the `Template` object thread safe,
but not the `Context` which is perfectly okay because it doesn't
happen (or should not happen) that a context object is shared between
threads.  At least I don't see a situation where this is useful.

What's harder to fix is how the i18n integration translates filter
arguments or gettext constants (those _("foo") thingies).  Currently
that happens very often at node/object creation rather than node/
object evaluation.  A good example for that problem is
FilterExpression.__init__.  The translation would have to be moved to
the resolve method in that case.  When would a language change between
template compilation and rendering?  If the language changes each
request which is a very common case on multilingual websites.

Changing the parser to a more advanced system or making the template
engine more consistent is not so important for the time being, but I
want to raise a few situations I encountered where the behaviour is
confusing:

  - cycle arguments are delimited by spaces and each item can be an
expression (however if there is a comma involved somewhere, it
seems like the tag is interpreted as comma separated list of
strings
which makes the 'cycle "foo", "bar", "baz"' yield unexpected
results.

  - On the other hand arguments for the url tag are comma delimited,
but whitespace after comma is not allowed.

  - The group-by part of the regroup tag is an expression, but
everything
else but a variable name returns unexpected results.  Furthermore
does this tag only work on dicts.  By the group-by part I'm
refering
to the expression after the "by" keyword:

  {% regroupy foo by bar as blah %}

bar is here treated as constant string, even though it's
represented
as variable in the regroup node and in the syntax.

  - Likewise ssi excepts one argument that looks like a variable, but
is
treated as constant string.

Tags to be excluded from the variable-as-string rule should be block
and load because they represent neither variables nor strings.  The
argument for block works like a label and the argument for load is a
library.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to 

Re: RFC: django.template refactoring (#7806)

[Armin Ronacher]

On Sep 18, 12:52 am, Malcolm Tredinnick <[EMAIL PROTECTED]>
wrote:
> On Wed, 2008-09-17 at 14:42 +0100, Ben Ford wrote:
> Parts of it are very well thought out and if it had been a post on "how
> Jinja works" it would have been excellent. Other parts are completely
> unconstrained by facts or acknowledgement that Django's standard
> templates and Jinja's ones have different goals (which is important,
> since this isn't an apples to apples comparison at all).
I'm starting to feel like Don Quixote here (the Windmills being a
community that learned to live with the implications of the
limitations of their beloved template engine).  After my blog post I
continued to dig in the Django sources and had to notice that it's
impossible to reused Template objects at all so you are pretty much
forced to reparse them over and over again.  (Not reparsing would for
example also break the FilterExpression embedded i18n support because
it translates on compilation, not on resolving)

With that in mind you can pretty much everything said about thread
safety in my blog post.

The rest of my reply is missing now.  I actually wrote a mail here
that was a lot longer than my blog post from yesterday but noticed
that I repeated myself with slightly different examples.  I don't want
to start the discussions again because I know the first reply will be
again that missing support for expressions is good, that the AST
evaluator is a blessing and in general the Django template engine is
the best ever happened to mankind.  I know I'm sort of exaggerating
here but in general that's what I read out of the comments I've seen
on the reddit discussion thread about that topic.

Please apologise my blog post, it wasn't meant to criticise the Django
template engine concept in any way, just to inspire a possible
improved version of Django templates.  I think Jinja does solve a few
things better than Django (and probably some worse) but I can't be a
mistake to look at how things work there to get an inspiration of how
to make things better.  The primary inspiration for myself for Jinja
was Django, and I'm not ashamed to say that.  When I started
developing Jinja it was a simple port of the concept into a standalone
library not depending on Django with unicode support.  What you can
see now in Jinja2 is the second iteration of a vastly changed template
engine, that however is still based on the same principles the Django
template engine is: no XML, template designer friendly, easy to get
started, extensible.

If I can help out in any way to get some of the Jinja concepts ported
to Django, or just explain why things work that way and not different
in detail I would love to do so.  I think Django can't lose, it just
takes someone to have a look at it.

Regards,
Armin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Escaping in templates...

[Armin Ronacher]

Hoi,

Another small notice. Pylons and other frameworks thought about
implementing "__html__" for objects that return an html representation
of the object. If there is none it's converted to unicode and escaped.
Adding something like "__html__ = lambda s: s" to the escaped string
base classes could improve support for other template engines like
genshi.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: #3527 - better debug traceback with code executing...

[Armin Ronacher]

Hoi,

A possible solution would be moving the debug view from the core into
a middleware. Then it's possible to replace the middleware and move
the debugger into a contrib package.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Escaping in templates...

[Armin Ronacher]

Hi,

On Apr 16, 1:08 pm, Michael Radziej <[EMAIL PROTECTED]> wrote:
> Armin, you're coming a bit late to this discussion. We had a few quite
> extensive threads months ago. If you seriously want to engage in this
> discussion, please read what's been written before. I understand that you
> cannot know when you joined later, but I wouldn't like to go through all the
> arguments again and again.
Dammit, haven't investigated further. Sorry for that.

> - there's a way to mark strings as "don't need further escaping"
>   (mark_safe())
>
> Whether it is an advantage or not is probably a matter of how you develop and
> how much you trust your web designer (or yourself). Different people will
> see different trade-offs here, that's just normal.
Yeah. After reading the current implementation it looks quite sane to
me. As long as there is a way to turn it off completely I can live
with that :-)
The problems I discovered was what happens if you pass an string
containing markup to a filter etc. In my test cases I often screwed up
things because a filter concatenated the markup object with another
string with the result of having a plain string with the markup in.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Escaping in templates...

[Armin Ronacher]

Hoi,

-sys.maxint for autoescaping. I added support for that into jinja
quite a while ago and it was pain in the ass. It makes things more
complicated (speaking for return values and arguments of filters) and
it blows up the implementation. Not worth the work.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Dictionary lookup with variable in template

[Armin Ronacher]

Hoi,

What about using the Jinja approach?

{{ foo.bar.baz }}
is the same as
{{ foo['bar']['baz'] }}

So it's possible to use variables using this (where bar is a
variable):

{{ foo[bar].baz }}

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: #3527 - better debug traceback with code executing...

[Armin Ronacher]

Hoi,

Alright. Patch here: http://pocoo.org/~mitsuhiko/django_debugger.patch
Screenshot here: http://pocoo.org/~mitsuhiko/djangodebug.png

(Note, I edited those files in the same checkout I did the patches for
the __loader__ hook. So the patch is a bit bigger. Sorry for that)

Enable by adding ENABLE_DEBUGGER = True into the settings.py file

Also note: I had to modify the CommonMiddleware. That's not the best
way to hook the traceback AJAX call in, maybe there are better ways.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Status of SQLAlchemy branch

[Armin Ronacher]

Hoi,

On Mar 29, 9:07 am, "Mir Nazim" <[EMAIL PROTECTED]> wrote:
> Now can anybody tell me about the status of Django's SQLAlchemy
> branch. I could not  find any place describing the status on
> code.djangoproject.com.

Looks like it's pretty dead. Very sad. SQLAlchemy is kicking-ass,
especially with more complex queries. I would love to see some love in
that branch.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

__loader__ support for debugging module (#3734)

[Armin Ronacher]

Hoi,

I filed this ticket (http://code.djangoproject.com/ticket/3734) some
time ago and it's still marked as "Design decision needed" :). Now i
added a patch and it works for me. I don't see any disadvantages in it
so i hope someone can apply that patch. If not I would love to read
comments about it.

Short summary: this ticket adds support for __loader__ which is
defined in the import hook pep and useful to retrieve sources from
packages that are imported using import hooks (eggs for example) and
also some template engines (currently afair only jinja) which use
__loader__ to send debugging sources into the traceback formatter.

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Unbundling the ORM

[Armin Ronacher]

The django ORM is the best out there. The only problem is that it's
only working from inside django.

There is a ticket for that: http://code.djangoproject.com/ticket/1321
and a discussion here:
http://groups.google.com/group/django-developers/browse_thread/thread/5b590b5487e500c1/c0f0d30cec89b107?q=orm=5#c0f0d30cec89b107

Currently there are only two usable ORMs out there. SQLObject which
isn't pep8 compatible and isn't so nice to work with as the django one
and SQLAlchemy which is very complicated.

For my current non django project i would like to use the django
magic-removed ORM because it's great to use. The only problem is that
using it require a complete django installation which isn't that good.

So... Are there any plans to completly unbudle the ORM from django?


Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Jinja Template Engine

[Armin Ronacher]

hugo wrote:
> >Jinja isn't compatible with django templates. It uses a parser for the
> >tag arguments
>
> A parser for tag arguments is available in Django, too - not all
> template tags use only regexps. :-)

You're right. Think i've overlooked it :-)

Regards,
Armin


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Jinja Template Engine

[Armin Ronacher]

Adrian Holovaty wrote:
> On 2/22/06, Ivan Fedorov <[EMAIL PROTECTED]> wrote:
> > What is Jinja?
> >
> > Jinja is a text-based template language similar to cheetah and Smarty.
> > The syntax and parts of the code where taken from the django template.
> >
> > http://wsgiarea.pocoo.org/jinja/
>
> Hey Ivan,
>
> What are the differences between Jinja and Django's template system?
> At a glance, I see little things such as different available default
> template filters and the fact that you have to use stuff like
> FileSystemLoader()?
Jinja isn't compatible with django templates. It uses a parser for the
tag arguments and features some other syntax elements like {% marker %}
/ {% range %}...

For my colubrid projects I wanted to have a django spin off with a
improved argument handling. For example it's possible to do this:


{% for variable in list %}

{{ variable.text | replace "demo", "other" | escapexml
}}
{% print variable.value | escapexml %}

{% endfor %}


Heiko Wundram worked hard on a easy to extend argument parser which is
explained here: http://wsgiarea.pocoo.org/jinja/docs/tag-dev.html

> Rather than creating a separate project, why not just integrate your
> changes into Django templates proper? We're going to make them
> downloadable as a standalone package for Django's next version.
Because it's a complete rewrite. I started it for improving my python
skills and because I was sick of kid and cheeath ^^ I don't think that
it can replace / should replace the current template sytem.

Regards,
Armin

appendix: because of my bad english i've copied and modified some parts
of the django template documentation. Sorry for that, I will rewrite
those parts in the next days.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: speeding Django up

[Armin Ronacher]

Does django provide a system for caching the nodelist? In Jinja I've
added a loader which cpickles the nodelist using protocol 2 which is
very fast because you don't have to parse it again.

Additonally you can store the templates in an global dict like in the
solution proposed in that thread:
http://groups.google.com/group/django-users/browse_thread/thread/40ecfffa8aa85a21/759da88e6d075ad7