The proposal and motivation are essentially the same as in the last
thread [1] and the ticket description [2]. I put it on the 1.2 feature
list.
I tried to split my patch into smaller, more readable commits here:
http://github.com/emulbreh/django/tree/master
I haven't converted defaulttags, loadertags and django.templatetags.*
yet to show that it's (mostly) backwards compatible (see below). It
would be useful to decide on some questions in the "API Design .."
section below first.
I'll summarize the core ideas again as well as problems and pending
design decisions.
Concepts: Expressions and TokenStream
=================================
Variable and FilterExpression will be deprecated as they require the
caller to know the length of the expression in advance. This makes
whitespace handling cumbersome and some syntactical constructs really
hard to parse (e.g., the {% url ... %} parser is broken by design).
Instead there would be a tokenizer (currently named TokenStream) that
lets you read expressions from a stream of tokens - this would replace
Token.split_contents().
>>> bits = TokenStream(p, 'url views.client_action
id=client.id,action="update",num=3').tokens; bits
[('name', 'url'), ('name', 'views.client_action'), ('name', 'id'),
('char', '='), ('name', 'client.id'), ('char', ','), ('name',
'action'), ('char', '='), ('string_literal', '"update"'), ('char',
','), ('name', 'num'), ('char', '='), ('numeric_literal', '3')]
Now, instead of explicitly constructing a FilterExpression object you
would simply call `bits.parse_expression()` and would get back an
object with a `.resolve(context)` method (a subclass of
django.template.expressions.Expression) that represents a Literal,
Variable (called Lookup), or FilterExpression. For compatibility
issues see "Variable vs FilterExpression" below.
A TokenStream can be read via the following api (the current stream
position is called `offset`).
Low level API
-------------------
``def pop(self)``
Returns a pair (tokentype, lexeme) and offset+=1; tokentype is one of
"string_literal", "numeric_literal", "char", "name".
``def pop_lexem(self, match)``
Returns True and offset+=1 if the next token's lexeme equals `match`.
Returns False otherwise.
``def pop_name(self)``
Returns the next token's lexeme and offset+=1 if its tokentype is
"name". Returns None otherwise.
``def pushback(self)``
offset-=1
High level API
--------------------
These methods raise TokenSyntaxError and leave offset untouched if
the expected result cannot be parsed.
Each accepts a boolean required=False kwarg which turns
TokenSyntaxErrors into TemplateSyntaxErrors if True.
``def parse_string(self, bare=False)``
Returns the value of the following string literal. If bare=True,
unquoted strings will be accepted.
``def parse_int(self)``
Returns the value of the following numeric literal, if it is an int.
``def parse_value(self)``
Returns an Expression that evaluates the following literal, variable
or translated value.
``def parse_expression(self)``
Returns an Expression that evaluates the following value or
filterexpression.
``def parse_expression_list(self, minimum=0, maximum=None, count=None)``
Returns a list `e` of expressions; minimum <= len(e) <= maximum.
count=n is a shortcut for minimum=maximum=n.
Variable vs FilterExpression
======================
I could only find documentation for Variable. If the internally used
Parser.compile_filter() is indeed undocumented there is no official
way to use FilterExpression in custom tags. If that means that
FilterExpression.resolve() behaviour doesn't have to be backwards
compatible, that would help a lot ..
1.) Currently `Variable("unresolvable")` and
`FilterExpression("unresolvable", p)` resolve differently: Variable
raises VariableDoesNotExist and FilterExpression returns
TEMPLATE_STRING_IF_INVALID. Except when `ignore_failures=True` is
given. But `FilterExpression("foo|cut:unresolvable", p)` will again
raise VariableDoesNotExist - regardless of `ignore_failures=True`.
My Expression implementation unifies these behaviours: If any part of
an expression is unresolvable a LookupError will be raised.
`ignore_failures` will be deprecated but there's a resolve_safe()
method on the Expression base class that reads:
def resolve_safe(self, context, default=None):
try:
return self.resolve(context)
except LookupError:
return default
This would be a small backwards incompatible change. I have one
failing test (exception02) because the ExtendsNode implementation
depends on the current FilterExpression behaviour.
2.) Currently `Variable("a._hidden")` works - but
`FilterExpression("a._hidden", p)` raises a TemplateSyntaxError. This
would be unified: lookup parts may not start with an underscore. If
it's not acceptable to break bc here leading underscores might simply
be allowed.
3.) Numeric literals ending with "." are currently treated as
Variables. The following test (ifequal-numeric07) succeds because
`ignore_failures=True` in IfEqualNode suppresses the inevitable
VariableDoesNotExist exception and then compares None to 2: ('{%
ifequal x 2. %}yes{% endifequal %}', {'x': 2}, ''). My current
implementation raises a TemplateSyntaxError as soon as it encounters a
numeric literal with a trailing dot. This could easily be made
backwards compatible if it's considered worth it.
API Design, Class Based Parsers, Combinable Mircorparsers
===============================================
To be backwards compatible my original idea was to wrap TokenStream
based parser functions with a decorator:
@uses_token_stream
def mytag(parser, bits):
exp = ts.parse_expression(required=True)
if not ts.pop_lexeme("as"):
raise TemplateSyntaxError, "'as' expected"
name = ts.pop_name()
if not name:
raise TemplateSyntaxError, "identifier expected"
return MyNode(exp, name)
But regarding Malcolm's request 'Helper methods that make it easy for
the common cases to say "I expect three paramters matching the 'x y as
z' pattern where x and z are strings and y must be an integer" would
be very handy.' [1] class bases parsers looked promising. They would
be resusable:
class ExpAsNameTagParser(TagParser):
def parse(self, parser, bits):
exp = ts.parse_expression(required=True)
if not ts.pop_lexeme("as"):
raise TemplateSyntaxError, "'as' expected"
name = ts.pop_name()
if not name:
raise TemplateSyntaxError, "identifier expected"
return (exp, name)
register.tag('sin', ExpAsNameTagParser(node_cls=SinNode))
register.tag('cos', ExpAsNameTagParser(node_cls=CosNode))
And with some kind of microparser infrastructure:
register.tag('sin', TagParser(microparser.expression + "as" +
mircoparser.name, node_cls=SinNode))
But then we should also have microparser.optional(), a | b, ... and
soon a clone of pyparsing. Luckily this can be implemented later on
top of the basic API - and would work as external code. If class based
parsers are the preferred approach TokenStream might be renamed to
TagParserState (or something similar) to make clear that it is not a
stream of Token instances.
Tags
=====
* {% url ... %}, {% ssi ... %} currently accept unquoted literal
strings. This will continue to work but the preferred syntax will use
properly quoted strings. This may one day allow for viewnames and ssi
paths that come from expressions.
* {% if not %} would fail with my current implementation. This is
fixable. But I'm not sure it's worth it: not/and/or should be keywords
in the context of boolean expressions.
tl;..
Johannes
[1]
http://groups.google.com/group/django-developers/browse_thread/thread/4f5b01719e497325
[2] http://code.djangoproject.com/ticket/7806
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---
