Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Ian Bicking
I can't read all this thread carefully, too much stuff. I will note however that people are STILL ignoring surrogateescape (http://www.python.org/dev/peps/pep-0383/). This is like the third or fourth time I've brought it up. It was added to Python 3.1 for some of the exact issues we are encounte

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Henry Precheur
On Sat, Sep 19, 2009 at 07:14:23PM +0100, René Dudfield wrote: > Yes, I think the callables will need a set method... rather than > letting the middleware replace callables. > > I think this could be used for middleware: > environ['SCRIPT_NAME'](set = "/bla/", urldecoding = False, encoding > ='

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 6:00 PM, Armin Ronacher wrote: > Hi, > > René Dudfield schrieb: >> What is proposed: > Where was that proposed? > >>     1. Default utf-8 to be used. > That's a possibility yes, but it has to be carefully be considered. > >>     2. A buffer to be used for raw data. > What i

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, René Dudfield schrieb: > What is proposed: Where was that proposed? > 1. Default utf-8 to be used. That's a possibility yes, but it has to be carefully be considered. > 2. A buffer to be used for raw data. What is raw data? If you mean we keep the unencoded data around, I would stro

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 5:07 PM, P.J. Eby wrote: > At 05:00 PM 9/19/2009 +0100, René Dudfield wrote: >> >> Issues with proposal?  Things this proposal did not consider? > > I'm wary of anything that makes correct middleware harder to write.  How > should an application that does path traversal wor

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread P.J. Eby
At 05:00 PM 9/19/2009 +0100, René Dudfield wrote: Issues with proposal? Things this proposal did not consider? I'm wary of anything that makes correct middleware harder to write. How should an application that does path traversal work? More precisely, take a look at the wsgiref.util.shift_

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Massimo Di Pierro
I agree I was forced to write two files. The problems where 1) b'' vs '' (could be solved using eval('b""') if python3 else eval("") but ugly) 2) try:...:except Exeption,e: vs try:...except Exception as e. (no way around it) 3) it would have required a lot of if s

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
Hello again, ok, getting back on topic... away from py3k porting methods... Using an API where the user can request the type wanted solves a lot of encoding issues. This is similar to Grahams suggestion, but instead allowing a user to request which encoding they want, and also get access to th

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Georg Brandl
René Dudfield schrieb: > On Sat, Sep 19, 2009 at 2:26 PM, Georg Brandl > wrote: >> René Dudfield schrieb: >> > Here is a snippet from the compat.py we used to port pygame to support > python2.3 through 3.1 How is that related? >>> >>> Rather than using a 2to3 tool - which then m

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 2:26 PM, Georg Brandl wrote: > René Dudfield schrieb: > Here is a snippet from the compat.py we used to port pygame to support python2.3 through 3.1 >>> How is that related? >>> >> >> Rather than using a 2to3 tool - which then makes you have two versions >> of you

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, René Dudfield schrieb: > Rather than using a 2to3 tool - which then makes you have two versions > of your code, making the code work in python 2.x and 3.x. 2to3 > outputs python2.x incompatible code - when it doesn't have to. 2to3 is intended to be run automatically for each release. You wou

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, Graham Dumpleton schrieb: > So, no strict need to make the WSGI adapter do it differently. You may > want to only do that if concerned about overhead of transcoding. > > Transcoding just these is most probably going to be less overhead than > the WSGI adapter having to set up both unicode and

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 1:54 PM, Graham Dumpleton wrote: > 2009/9/19 Armin Ronacher : >> Graham's suggestion for URL encodings means that the URL encoding would >> ahve to be passed to the WSGI server from outside (he proposed the >> apache config as an example).  This means that the application b

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Georg Brandl
René Dudfield schrieb: >>> Here is a snippet from the compat.py we used to port pygame to support >>> python2.3 through 3.1 >> How is that related? >> > > Rather than using a 2to3 tool - which then makes you have two versions > of your code, making the code work in python 2.x and 3.x. 2to3 > out

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 1:34 PM, Armin Ronacher wrote: > Hi, > > René Dudfield schrieb: >> I think that shows that they are being handled differently depending >> on type.  Which is against polymorphism... but some people prefer to >> have separate functions for different types(in and out).  I don

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Graham Dumpleton
2009/9/19 Armin Ronacher : > Graham's suggestion for URL encodings means that the URL encoding would > ahve to be passed to the WSGI server from outside (he proposed the > apache config as an example).  This means that the application behavior > will change based on the server configuration, causin

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, René Dudfield schrieb: > I think that shows that they are being handled differently depending > on type. Which is against polymorphism... but some people prefer to > have separate functions for different types(in and out). I don't > think other python functions do this though. So maybe this

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread René Dudfield
On Sat, Sep 19, 2009 at 11:59 AM, Armin Ronacher wrote: > Hi, > > Armin Ronacher schrieb: >> urllib.parse appears to be buggy with bytestrings: >> >> I'm pretty sure the latter is a bug and I will file one, however if >> there is broken behavior with bytestrings in Python 3.1 that's another >> thi

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, Armin Ronacher schrieb: > http://paste.pocoo.org/show/140501/ Corrected version without Werkzeug leftovers: http://paste.pocoo.org/show/140502/ Regards, Armin ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Un

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, Armin Ronacher schrieb: > I have to correct myself, there are separate functions for byte quoting. > (parse.unquote_to_bytes, parse.quote_from_bytes). However, urlencode and urldecode are string only. Regards, Armin ___ Web-SIG mailing list Web-SI

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, Another observation from the HTTP server that comes with the Python 3 Standard Library: it does not support non-ASCII headers: def send_header(self, keyword, value): """Send a MIME header.""" if self.request_version != 'HTTP/0.9': self.wfile.write(("%s: %s\r\n"

Re: [Web-SIG] Unicode in Python 3

2009-09-19 Thread Armin Ronacher
Hi, Armin Ronacher schrieb: > urllib.parse appears to be buggy with bytestrings: > > I'm pretty sure the latter is a bug and I will file one, however if > there is broken behavior with bytestrings in Python 3.1 that's another > thing we have to keep in mind. I have to correct myself, there are se