Re: [Web-SIG] Are you going to convert Pylons code into Python 3000?
Hey, On Wed, Mar 5, 2008 at 3:25 AM, Guido van Rossum [EMAIL PROTECTED] wrote: On Tue, Mar 4, 2008 at 6:13 PM, Martijn Faassen [EMAIL PROTECTED] wrote: Hey, On Wed, Mar 5, 2008 at 1:48 AM, Graham Dumpleton [EMAIL PROTECTED] wrote: [snip] In the case of code which directly talks to the interface defined by WSGI specification I very much doubt the py2to3 script will help. This is because for WSGI to work with Python 3.0 there needs to be a change from use of string type objects to byte string type objects. I would suspect that py2to3 is only get help in any sort of automated way with the fact that a string object becomes unicode aware, not where with WSGI the code would have to change to use and deal with a different type of object completely. The implications of this change to a byte string type object are going to be much more complicated. I have no idea what the capabilities of this script are. I would *imagine* it would convert classic strings into the bytes types, and unicode strings into the new string type. It does nothing of the kind. It leaves 'xxx' literals alone and translates u'xxx' to 'xxx'. That's because (in many apps) both are used primarily for text. BTW I suggest that you play with it at least a little bit (run it on its own example.py file) before diving into this discussion... I accurately described my lack of knowledge of the script, then. :) Sure, I need to play with the script. I guess the best route would be to introduce bytes in your code in Python 2.x and have the script leave that alone. If WSGI 2.0 then makes it into Python 2.x as well, then there's no problem with API breakage. Playing with the script will happen sometime, but I think it's quite clear the script will be of no help if important library APIs also break down because people take their chances during transition (and the script doesn't take care of it, which it can't for third party APIs). WSGI is probably not the best example given the string issue and its inclusion in the Python core, though: as Graham expressed, it's probably going to have problems no matter what. I also think any new version could be developed on Python 2.6 first, as this will support the bytes type as far as I understand. And yes, I need to try the Python 2.6 alpha interpreter first too. :) Regards, Martijn ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
[Web-SIG] ngx.poll extension (was Re: Are you going to convert Pylons code into Python 3000?)
Brian Smith ha scritto:
Manlio Perillo wrote:
Fine with me but there is a *big* problem.
WSGI 2.0 breaks support for asynchronous applications
(since you can no more send headers in the app iter).
WSGI 1.0 doesn't guarentee that all asynchronous applications will work
either, because it allows the WSGI gateway to wait for and buffer all
the input from the client before even calling the application callable.
And, it doesn't provide a way to read an indefinite stream of input from
the client, which is also problematic.
Anyway, please post a small example of a program that fails to work
because of these proposed changes for WSGI 2.0.
Thanks,
Brian
Attached there are two working examples (I have not committed it yet,
because I'm still testing - there are some problems that I need to solve).
The `curl_client` module is an high level interface to pycurl.
The `nginx-poll-proxy.py` script is an asyncronous WSGI application that
implements an HTTP proxy.
The `nginx-poll-sleep.py` script is a simple asynchronous WSGI
application that get the content of an HTTP resource using poll just to
sleep (suspend execution) for a fixed amount of time.
NOTE: I have also added a `ngx.sleep` extension, but I'm going to remove
it since the same behaviour can be obtained with ngx.poll.
An explanation of the interfaces
The ngx.poll extension is based on the Python stdlib select.poll interface.
There are two constants: `ngx.WSGI_POLLIN` and `ngx.WSGI_POLLOUT`.
These are defined in the WSGI environment, but their value is know
(`0x01` and `0x04`) and can be used for bit masking.
The `ngx.connection_wrapper(fd)` function takes as input a file
descriptor (as integer) and returns a Connection wrapper object, to be
used for later operations.
The Connection wrapper object has the following methods:
- fileno():
return the associated socket descriptor
- register(flags):
register the connection with the server reactor;
flags is a bit mask of ngx.WSGI_POLLIN and ngx.WSGI_POLLOUT
- deregister(flags=None):
deregister the connection from the server reactor
- close:
close the connection object, deregisterering it from the server
reactor if still active.
XXX it also can close the socket, but this should be done by the
client
The last function is `ngx.poll(timeout)`.
When called, the user *should* yield an empty string (yielding a non
empty string will result in an undefined behaviour).
The WSGI application iteration will be suspended until a connection is
ready for reading or writing, or the timeout expires.
The `ngx.poll` function returns a callable that, when called, returns a
tuple with the connection object ready (or None if timedout) and a
flag indicating if the connection is ready for reading or writing.
NOTE: due to the internal architecture of the Nginx event module (it
have to support several different event systems), mod_wsgi for
Nginx will only return ngx.WSGI_POLLIN or ngx.WSGI_POLLPUT,
*never* ngx.WSGI_POLLIN | ngx.WSGI_POLLPUT.
Also, no error status is reported.
That's all.
An asynchronous application is simply impossible to develope with the
current draft of WSGI 2.0, since I need to send the headers after some
steps in the application iterator.
So, please, don't ruin the WSGI specification just to make it more
easy to implement and to use.
For me asynchronous support is very important.
P.S: I have chosen to implement this interface, instead of
`wsgi.pause_output`, because IMHO it is very easy to implement for
normal servers.
Moreover it is also more simple to use, with a very natural
interface, and it avoids the use of callbacks and a more strict
interaction with the server reactor.
Regards Manlio Perillo
import time
from cStringIO import StringIO
from wsgiref.headers import Headers
import pycurl
# These constants are not defined in pycurl
CURL_POLL_NONE = 0
CURL_POLL_IN = 1
CURL_POLL_OUT = 2
CURL_POLL_INOUT = 3
CURL_POLL_REMOVE = 4
# These constants are defined in the WSGI environment but their value
# is know
WSGI_POLLIN = 0x01
WSGI_POLLOUT = 0x04
class CurlClient(object):
Asyncronous client using pycurl.
Support only one connection at a time.
def __init__(self):
# Create the multi instance
self.multi = pycurl.CurlMulti()
self.multi.setopt(pycurl.M_SOCKETFUNCTION, self._socket_callback)
self.multi.setopt(pycurl.M_TIMERFUNCTION, self._timeout_callback)
def request(self, environ, url):
self.connection = environ['ngx.connection_wrapper']
self.log = environ['wsgi.errors']
self.timeout = 3
self.can_send_headers = False
self.headers = Headers([])
self.buf = StringIO()
# map sockets to connection wrapper objects
self.connection_map = {}
# XXX keep the reference to the simple instance
c = self.add_simple(url)
Re: [Web-SIG] ngx.poll extension (was Re: Are you going to convert Pylons code into Python 3000?)
Let me get this right. You are complaining that the WSGI 2.0 would break your non standard extension which was never a part of the WSGI 1.0 specification to begin with. I also find it interesting that in the very early days you were pushing very very hard for WSGI 2.0 to be specified and you had no intention of even supporting WSGI 1.0 style interface. Now things seem to be the complete opposite. Anyway, your complaint seems to resolve around: An asynchronous application is simply impossible to develope with the current draft of WSGI 2.0, since I need to send the headers after some steps in the application iterator. You probably need to explain the second half of that sentence a bit better. From memory the WSGI 1.0 specification says that for an iterable, the headers should be sent upon the generation of the first non empty string being yielded. How does what you are doing relate to that, are you not doing that? Why would WSGI 2.0 necessarily be any different and cause a problem? Graham On 06/03/2008, Manlio Perillo [EMAIL PROTECTED] wrote: Brian Smith ha scritto: Manlio Perillo wrote: Fine with me but there is a *big* problem. WSGI 2.0 breaks support for asynchronous applications (since you can no more send headers in the app iter). WSGI 1.0 doesn't guarentee that all asynchronous applications will work either, because it allows the WSGI gateway to wait for and buffer all the input from the client before even calling the application callable. And, it doesn't provide a way to read an indefinite stream of input from the client, which is also problematic. Anyway, please post a small example of a program that fails to work because of these proposed changes for WSGI 2.0. Thanks, Brian Attached there are two working examples (I have not committed it yet, because I'm still testing - there are some problems that I need to solve). The `curl_client` module is an high level interface to pycurl. The `nginx-poll-proxy.py` script is an asyncronous WSGI application that implements an HTTP proxy. The `nginx-poll-sleep.py` script is a simple asynchronous WSGI application that get the content of an HTTP resource using poll just to sleep (suspend execution) for a fixed amount of time. NOTE: I have also added a `ngx.sleep` extension, but I'm going to remove it since the same behaviour can be obtained with ngx.poll. An explanation of the interfaces The ngx.poll extension is based on the Python stdlib select.poll interface. There are two constants: `ngx.WSGI_POLLIN` and `ngx.WSGI_POLLOUT`. These are defined in the WSGI environment, but their value is know (`0x01` and `0x04`) and can be used for bit masking. The `ngx.connection_wrapper(fd)` function takes as input a file descriptor (as integer) and returns a Connection wrapper object, to be used for later operations. The Connection wrapper object has the following methods: - fileno(): return the associated socket descriptor - register(flags): register the connection with the server reactor; flags is a bit mask of ngx.WSGI_POLLIN and ngx.WSGI_POLLOUT - deregister(flags=None): deregister the connection from the server reactor - close: close the connection object, deregisterering it from the server reactor if still active. XXX it also can close the socket, but this should be done by the client The last function is `ngx.poll(timeout)`. When called, the user *should* yield an empty string (yielding a non empty string will result in an undefined behaviour). The WSGI application iteration will be suspended until a connection is ready for reading or writing, or the timeout expires. The `ngx.poll` function returns a callable that, when called, returns a tuple with the connection object ready (or None if timedout) and a flag indicating if the connection is ready for reading or writing. NOTE: due to the internal architecture of the Nginx event module (it have to support several different event systems), mod_wsgi for Nginx will only return ngx.WSGI_POLLIN or ngx.WSGI_POLLPUT, *never* ngx.WSGI_POLLIN | ngx.WSGI_POLLPUT. Also, no error status is reported. That's all. An asynchronous application is simply impossible to develope with the current draft of WSGI 2.0, since I need to send the headers after some steps in the application iterator. So, please, don't ruin the WSGI specification just to make it more easy to implement and to use. For me asynchronous support is very important. P.S: I have chosen to implement this interface, instead of `wsgi.pause_output`, because IMHO it is very easy to implement for normal servers. Moreover it is also more simple to use, with a very natural interface, and it avoids the use of callbacks and a more strict interaction
Re: [Web-SIG] ngx.poll extension (was Re: Are you going to convert Pylons code into Python 3000?)
At 09:37 AM 3/6/2008 +1100, Graham Dumpleton wrote: You probably need to explain the second half of that sentence a bit better. From memory the WSGI 1.0 specification says that for an iterable, the headers should be sent upon the generation of the first non empty string being yielded. How does what you are doing relate to that, are you not doing that? Why would WSGI 2.0 necessarily be any different and cause a problem? Because (in concept anyway) WSGI 2.0 is synchronous with respect to headers -- you don't get to yield empty strings and *then* return the headers. Personally, I see truly-async web apps as a niche, because in order to write a useful async app, you need *other* async APIs besides your incoming HTTP one. Which means you're going to have to write to Twisted or some other library's API, or else roll your own. At which point, connecting your app to a web server is the least of your concerns. (Since it has to be a web server that's compatible with the API you're using, which means you might as well use its native API.) That having been said, I don't see a problem with having a Web Server Asynchronous Interface (WSAI?) for folks who want that sort of thing. Ideally, such a thing would be the CPS (continuation-passing style) mirror of WSGI 2.0. Where in WSGI 2.0 you return a 3-tuple, in WSAI you'd essentially use start_response() and write(). In essence, you might say that WSGI 1.0 is a broken-down version of a hideous crossbreeding of pure WSGI and pure WSAI. It would probably be better to split them and have bridges. A truly-async system like Twisted has to (effectively) do WSAI-WSGI bridging right now, but if we had a WSAI standard, then there could perhaps be third-party bridges. Even so, it's quite a niche: Twisted, nginx, and...? I know there are a handful of async frameworks, and how many of those have web servers included? ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
