At 09:22 PM 9/22/2009 -0500, Ian Bicking wrote:
OK, I mentioned this in the last thread, but... I can't keep up with
all this discussion, and I bet you can't either.
So, here's a rough proposal for WSGI and unicode:
I propose we switch primarily to "native" strings: str on both Python 2 and 3.
+1, if you mean the strings have the same content,
character-for-character on Python 2.3. That is, a \x80 byte in a
Python 2 'str' is matched by an \x80 character in the Python 3
'str'. (I presume that's what we mean by "native", but I want to be sure.)
Specifically:
environ keys: native
environ CGI values: native
wsgi.* (that is text): native
response status: native
response headers: native
+1 all of the above, again per the previous caveat. IOW, if all this
stuff is still exactly as laid out in PEP 333 to start with. (Minor
issue: the CGI environment vars will need transcoding on Python 3, if
the system encoding is not latin 1.)
wsgi.input remains byte-oriented, as does the response app_iter.
+1, acceptable as errata of the original PEP wrt to Python 3, as long
as we also the app to yield "native" strings (Unicode chars 0-255) on Python 3.
I then propose that we eliminate SCRIPT_NAME and PATH_INFO. Instead we have:
wsgi.script_name
wsgi.path_info (I'm not entirely set on these names)
These both form the original path. It is not URL decoded, so it
should be ASCII. (I believe non-ASCII could be rejected by the
server, with Bad Request? A server could also choose to treat it
as UTF8 or Latin1 and encode unsafe characters to make it
ASCII)Â Thus to re-form the URL, you do:
environ['wsgi.url_scheme'] + '://' + environ['HTTP_HOST'] +
environ['wsgi.script_name'] + environ['wsgi.path_info'] + '?' +
environ['QUERY_STRING']
I'm not clear how all the above is going to work, since I had the
impression from the comments of server specialists here (e.g. Graham,
Robert and Alan) that you simply can't do this
correctly/consistently, especially under Java. I think a more
detailed proposal is needed here, along with a brief rationale for
why we need them to be specially named like this.
All incoming headers will be treated as Latin1. If an application
suspects another encoding, it is up to the application to transcode
the header into another encoding.Â
+1.
The transcoded value should not be put into the environ.Â
Make that a MUST NOT, and I'm good. ;-)
In most cases headers should be ASCII, and Latin1 is simply a
fallback that allows all bytes to be represented in both Python 2 and 3.
Hurray!
Similarly all outgoing headers will be Latin1. Thus if you
(against good sense) decide to put UTF8 into a cookie, you can do:
headers.append(('Set-Cookie', unicode_text.encode('UTF8').decode('latin1')))
The server will then decode the text as latin1, sending the UTF8
bytes. This is lame, but non-ASCII in headers is lame. It would
be preferable to do:
headers.append(('Set-Cookie', urllib.quote(unicode_text.encode('UTF8'))))
This sends different text, but is highly preferable. If you wanted
to parse a cookie that was set as UTF8, you'd do:
parse_cookie(environ['HTTP_COOKIE'].encode('latin1').decode('utf8'))
Again, it would be better to do;
parse_cookie(urllib.unquote(environ['HTTP_COOKIE']).decode('utf8'))
Looking good.
Other variables like environ['wsgi.url_scheme'],
environ['CONTENT_TYPE'], etc, will be native strings. A Python 3
hello work app will then look like:
def hello_world(environ):
   return ('200 OK', [('Content-type', 'text/html;
charset=utf8')], ['Hello World!'.encode('utf8')])
start_response and changes to wsgi.input are incidental to what I'm
proposing here (except that wsgi.input will be bytes); we can decide
about themseparately.
More +1 goodness from me.
Outstanding issues:
Well, the biggie: is it right to use native strings for the environ
values, and response status/headers? Specifically, tricks like the
latin1 transcoding won't work in Python 2, but will in Python
3. Is this weird? Or just something you have to think about when
using the two Python versions?
If your app is written to spec and you're *not* using unicode,
nothing changes. But if your app is written to spec and you *are*
using unicode, then you're currently calling .decode() on anything
you read from environ or input, in order to get to unicode. That
will break on 3.x, since str lacks a .decode() method. So, you will
at least fail every time with an error message, and fix it, unless
somebody writes a 2to3 fixer that can infer a "wsgi native" type and
do the right thing. I don't know if such a thing is possible.
However, it also seems to me that it's pretty unlikely anybody is
doing this decoding in multiple places in their code, so it seems
unlikely to be very painful for very long. Libraries tend to have
request objects, or functions to do this kind of thing, so only those
should have to change in 3.x.
What happens if you give unicode text in the response headers that
cannot be encoded as Latin1?
Then something's going to fail, and unfortunately it's going to take a while.
Should some things specifically be ASCII? E.g., status.
+1.
Should some things be unicode on Python 2?
Not that I can think of. To be honest, this whole thread has made me
appreciate the Python 2 str type more than ever. ;-)
Is there a common case here that would be inefficient?
Beats me.
_______________________________________________
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
