Server side includes and Python.
A few weeks back I created a JIRA entry relating to integrating server
side includes with Python. The entry is:
http://issues.apache.org/jira/browse/MODPYTHON-104
I finally got around to having a go at implementing it and have some
initial code now working. The point of this email is to get feedback on
the approach used and whether anyone has any alternate suggestions of
how it should in practice work, or any other ideas for that matter.
For those who might not know what server side includes are all about,
see:
http://httpd.apache.org/docs/2.0/mod/mod_include.html
Effectively, what it is is an Apache output filter that will look for
SGML (HTML/XML) comments in content returned by a handler or as may be
found in a static file. It will identify specially tagged comments and
process them to yield new content which will be inserted in place in the
output instead of the original comment. The filter even supports
conditionals and is able to trigger sub requests and CGI scripts.
To cater for other sources of input, the filter can be extended to
support additional tags. For example, mod_perl adds the tag "perl",
allowing one to use:
Now, there is no reason that support for Python can't also be added and
it is actually quite straightforward to do. The only question that needs
to be answered is how it would be used. What one would get out of it is
a simple template mechanism which uses only mod_python and existing
features of Apache itself, and which can also be used in conjunction
with other modules like mod_perl as well.
Now for how the syntax would work. What I have got going at the moment
are the following two scenarios:
This first will run "eval" with the result being converted into a string
and included direct into the output.
The second will run "exec" instead. In this case there is no result and
thus filter.write() has to be used to generate the output.
In both cases, the "filter" object is pushed into the local variable
set and would be accessible.
When using "exec", multiple lines of Python code can be provided. As
usual Python indentation is always fun, and as such, when providing
multiple lines of code, you can't have leading whitespace unless it is
truly required for nesting.
Having to include all the code in the HTML isn't fun and it is actually
better to separate it out anyway.
To do that, one simply has to perform an import of the required module
and execute some function contained in it or access data and write it
out.
If the target module is already the subject of automatic module
reloading
as implemented by mod_python, you might instead use:
For "eval", one could provide a short cut mechanism whereby one could
specify
the module in which the code should be evaluated.
Ie., this is essentially equivalent to having said:
from mod_python import apache
module = apache.import_module("sys")
filter.write(str(eval("version", module.__dict__, {"filter":filter})))
The "module" short cut wouldn't exist for "exec" as any module imports
then
performed would pollute the global name space of the specified module,
which
may not be desirable.
As you can see, various things are possible but what is the minimum that
should be provided?
In the interests of discouraging a lot of code inside of a HTML file and
thus protecting people from themselves, should the "exec" variant above
be discarded, leaving just the "eval" variant? This would have the
effect
of forcing users to put complex code into separate modules and simply
triggering calls of the separate code.
The only hard bit is how one should treat the response from the function
called. One could say that if the result of the eval is 'None' then no
output is written and if it was a function which was called, it is
assumed
that the called function did that by writing back to the filter object.
In other words, the result of the eval isn't simply converted to a
string.
This would only be done if result is not 'None', which 'None' truly
meaning
no output.
Anyway, that is all there is to it.
Given that the intent here is to try and get this functionality rolled
into
mod_python, any feedback would be most appreciated. No consensus
probably
means it would get rejected.
Thanks.
Graham
Re: Server side includes and Python.
I like the SSI feature. It would fill a nice gap between using plain HTML files and having to go to a more featured template or engine. Some things are simple enough that the SSI concept should be enough, and having Python would be nice. I do need to give your proposal some more thought before I can properly comment, but it looks interesting so far. One thing I think that *should* be very easy to do in an SSI setting is HTML-escaping. I shouldn't have to do something like 'from cgi import escpe'. Perhaps adding another parameter, like where esc is a built-in escaping filter: h=html, u=url, x=xml (difference between h and x is how it escapes quote chars). Another question. How are character sets handled? If the output is a Unicode string, how does it get encoded? Should it always asume say UTF-8, or can it determine the actual character encoding for this reponse somehow? Also, my vote is that None should result in no output. -- Deron Meranda
Re: Server side includes and Python.
On 23/01/2006, at 4:59 PM, Deron Meranda wrote: I like the SSI feature. It would fill a nice gap between using plain HTML files and having to go to a more featured template or engine. Some things are simple enough that the SSI concept should be enough, and having Python would be nice. I do need to give your proposal some more thought before I can properly comment, but it looks interesting so far. One thing I think that *should* be very easy to do in an SSI setting is HTML-escaping. I shouldn't have to do something like 'from cgi import escpe'. Perhaps adding another parameter, like where esc is a built-in escaping filter: h=html, u=url, x=xml (difference between h and x is how it escapes quote chars). Using "print" like that can't be done in an "eval", would need to use "exec". Even then, you have to explicitly direct the "print" to the filter as not possible with mod_python to change sys.stdout so that "print" by itself could be used. Thus, with making things more verbose: Unfortunately though, this would not work. This is because it is writing direct to the filter and there is no opportunity to escape the content as it is written. Such automatic escaping could only be done for "eval", and only if the content is the result. Something could still write direct to the filter object and bypass it. Thus, although it indeed sounds like it may be a useful thing to have, it would not be practical to implement it such that it was able to capture all generated content. Thus, requiring the user to explicitly escape where needed might still be necessary. In terms of avoiding having to do imports at each point of use, have got it going now so that all code executing in the page uses the same local variable space. Thus one could do a whole lot of imports and variable assignments in one exec at the start and then use that later on. Thus: Another question. How are character sets handled? If the output is a Unicode string, how does it get encoded? Should it always asume say UTF-8, or can it determine the actual character encoding for this reponse somehow? Changes made in the mod_python.publisher layer in mod_python 3.2 might be used as a guide here. Ie., it tries to be smart about encoding: elif isinstance(object,UnicodeType): # We've got an Unicode string to publish, so we have to encode # it to bytes. We try to detect the character encoding # from the Content-Type header if req._content_type_set: charset = re_charset.search(req.content_type) if charset: charset = charset.group(1) else: # If no character encoding was set, we use UTF8 charset = 'UTF8' req.content_type += '; charset=UTF8' else: # If no character encoding was set, we use UTF8 charset = 'UTF8' result = object.encode(charset) In mod_python.publisher you can see though that it only worries about it if the result is a Unicode string. Ie., if anything else is returned or the request object is written to direct, then it is up to the user to implement what they want. If this approach in mod_python.publisher as seen as reasonable and works, then for 'eval' one could take the same approach. If you are going to allow that though, then allowing "escape" for "eval" might also be seen as reasonable. One would just need to document the caveats. That said, can you explain more about the differences between HTML and XML escaping with the quoting. I don't really understand the differences. Are there particular Python routines that implement each variant, or is it some option to 'cgi.escape'? Also, which is the preferred routine for url encoding? Thanks for your interest. Graham
Re: Server side includes and Python.
On 24/01/2006, at 3:07 AM, Deron Meranda wrote: On 1/23/06, Graham Dumpleton <[EMAIL PROTECTED]> wrote: Using "print" like that can't be done in an "eval", would need to use "exec". Sorry, I probably didn't mean to use the print in my example. Of course though you can always wrap sys.stdout if you wanted to capture the output for post-escaping. You can't wrap sys.stdout or replace it for the purposes of a single request. This is because mod_python needs to be able to work in a multi thread MPM where multiple requests need to be handled at the same time. The cgihandler in mod_python does fiddle with sys.stdout, but to do that it has to lock on the cgihandler so only one at a time can run, thus affecting performance. Graham
