Re: [Web-SIG] WSGI and asyncio (tulip)?
Hi, 2014-10-14 18:47 GMT+02:00 Guido van Rossum gu...@python.org: I'm wondering if a small extension to the WSGI protocol might be sufficient to support this: the special environ variable wsgi.async_input could optionally be tied to a standard asyncio stream reader ( https://docs.python.org/3/library/asyncio-stream.html#streamreader), from which bytes can be read using yield from stream.read([nbytes]) or yield from stream.readline(). If we support async backends by simply escaping WSGI, don't you feel it kind of make most of the whole discussion moot? To me, asyncio already provides a de-facto standard API for asynchronous backends and Tornado/Twisted provide a high level API on top of it. I have to say, I don't precisely grasp what WSGI actually wishes to bring to the table. As I said in a different thread, most frameworks seem eager to wrap the environ dictionary and hide away all of the WSGI internals (wasting CPU cycles in the process). Is there rationale for continuining down that road? -- - Sylvain http://www.defuze.org http://twitter.com/lawouach ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
To me, asyncio already provides a de-facto standard API for asynchronous backends and Tornado/Twisted provide a high level API on top of it. I have to say, I don't precisely grasp what WSGI actually wishes to bring to the table. I guess if we're really talking about this, the lowest common denominator is a set of cognizable abstractions for HttpRequest and HttpResponse, right? On Tue, Oct 14, 2014 at 11:54 AM, Sylvain Hellegouarch s...@defuze.org wrote: Hi, 2014-10-14 18:47 GMT+02:00 Guido van Rossum gu...@python.org: I'm wondering if a small extension to the WSGI protocol might be sufficient to support this: the special environ variable wsgi.async_input could optionally be tied to a standard asyncio stream reader (https://docs.python.org/3/library/asyncio-stream.html#streamreader), from which bytes can be read using yield from stream.read([nbytes]) or yield from stream.readline(). If we support async backends by simply escaping WSGI, don't you feel it kind of make most of the whole discussion moot? To me, asyncio already provides a de-facto standard API for asynchronous backends and Tornado/Twisted provide a high level API on top of it. I have to say, I don't precisely grasp what WSGI actually wishes to bring to the table. As I said in a different thread, most frameworks seem eager to wrap the environ dictionary and hide away all of the WSGI internals (wasting CPU cycles in the process). Is there rationale for continuining down that road? -- - Sylvain http://www.defuze.org http://twitter.com/lawouach ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/justin%40justinholmes.com -- jMyles Holmes Chief Chocobo Breeder, slashRoot slashRoot: Coffee House and Tech Dojo New Paltz, NY 12561 845.633.8330 ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
2014-10-14 20:57 GMT+02:00 Justin Holmes jus...@justinholmes.com: To me, asyncio already provides a de-facto standard API for asynchronous backends and Tornado/Twisted provide a high level API on top of it. I have to say, I don't precisely grasp what WSGI actually wishes to bring to the table. I guess if we're really talking about this, the lowest common denominator is a set of cognizable abstractions for HttpRequest and HttpResponse, right? That's my feeling as well. But, considering the sheer amount of content produced in other threads, I guess I'm missing the point somehow ;) -- - Sylvain http://www.defuze.org http://twitter.com/lawouach ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 08:40:05 +1300 Robert Collins robe...@robertcollins.net wrote: On 15 October 2014 07:30, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 14 Oct 2014 09:47:35 -0700 Guido van Rossum gu...@python.org wrote: I'm wondering if a small extension to the WSGI protocol might be sufficient to support this: the special environ variable wsgi.async_input could optionally be tied to a standard asyncio stream reader ( https://docs.python.org/3/library/asyncio-stream.html#streamreader), from which bytes can be read using yield from stream.read([nbytes]) or yield from stream.readline(). I think it would be frankly better to hook at the transport/protocol level, and let people wrap that inside an asyncio stream if that's their preference. For things like mod_wsgi and uwsgi, we're not actually implementing the transport or protocol inside of Python at all - its all happening in C and often in an entirely separate process. You may have misunderstood me. I am talking about the Transport and Protocol abstractions defined in PEP 3156. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On 15 October 2014 08:41, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 15 Oct 2014 08:40:05 +1300 Robert Collins robe...@robertcollins.net wrote: On 15 October 2014 07:30, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 14 Oct 2014 09:47:35 -0700 Guido van Rossum gu...@python.org wrote: I'm wondering if a small extension to the WSGI protocol might be sufficient to support this: the special environ variable wsgi.async_input could optionally be tied to a standard asyncio stream reader ( https://docs.python.org/3/library/asyncio-stream.html#streamreader), from which bytes can be read using yield from stream.read([nbytes]) or yield from stream.readline(). I think it would be frankly better to hook at the transport/protocol level, and let people wrap that inside an asyncio stream if that's their preference. For things like mod_wsgi and uwsgi, we're not actually implementing the transport or protocol inside of Python at all - its all happening in C and often in an entirely separate process. You may have misunderstood me. I am talking about the Transport and Protocol abstractions defined in PEP 3156. Lets assume I did. Given say nginx + uwsgi + asyncio, you're proposing that there be a uwsgi-asyncio module that listens to the uwsgi socket and demuxes packets from that into something that then exposes a ReadTransport + WriteTransport pair and a Protocol on top of that. That Protocol would have a 1:1 correspondence with a WSGI request, and would *not* be HTTP itself but rather the subset that is exposed via uwsgi? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 09:22:28 +1300
Robert Collins
robe...@robertcollins.net wrote:
You may have misunderstood me. I am talking about the Transport and
Protocol abstractions defined in PEP 3156.
Lets assume I did. Given say nginx + uwsgi + asyncio, you're proposing
that there be a uwsgi-asyncio module that listens to the uwsgi socket
and demuxes packets from that into something that then exposes a
ReadTransport + WriteTransport pair and a Protocol on top of that.
Let's call it uwsgi-pep3156. It shouldn't be asyncio-specific.
[Note Guido's original remark: However, the situation is different
for reading the request body. The app can read this in dribs and drabs
if it wants to, by reading limited amounts of data from
environ['wsgi.input'], but in an asyncio-driven world reading
operations should really be marked with yield from (or perhaps just
yield -- again, I'm sure an adaptor won't have a problem with this).
Ergo, this is about streaming the request and response bodies. Not
asynchronously receiving the headers, etc.]
The server would implement a Transport for the input body stream +
output body stream. It would accept a Protocol factory from the
application/middleware.
Then when a request comes:
protocol = protocol_factory()
transport = Transport(...)
protocol.connection_made(transport)
and when an input body chunk is available:
protocol.data_received(chunk)
and when the input body is finished:
protocol.eof_received()
The protocol would be able to call transport.extra_info(...) to get
HTTP-specific information, e.g. transport.extra_info('headers') perhaps.
(that's for the HTTP part; a websockets layer would probably implement
a separate transport and accept a separate protocol factory; actually,
it could be implemented as a protocol that would parse the websockets
protocol and provide its own transport on top of that... there may
already be such a thing on the Internet :-))
Regards
Antoine.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On 15 October 2014 10:04, Antoine Pitrou solip...@pitrou.net wrote: ... (that's for the HTTP part; a websockets layer would probably implement a separate transport and accept a separate protocol factory; actually, it could be implemented as a protocol that would parse the websockets protocol and provide its own transport on top of that... there may already be such a thing on the Internet :-)) So thats the bit that I'm having conceptual trouble with - servers may well implement the framing (and I rather think that they have to do so in some deployments), so we need to make sure that we don't do that in this context - we're behind an abstraction. With the HTTP example you gave, it looks fine, and I'm sure that equivalents can be made for websockets etc too. The question in mind is whether that is amenable to the same unification and layering that WSGI brought to the HTTP synchronous case. It looks like you'd make a protocol factory decorator. Will it be sufficiently flexible and interoperable to be something that becomes a lingua franca? I don't know pep-3156 well enough to judge that myself. Seems to me that there are two broad directions here: we can have a WSGI-thing where it looks just a little different to WSGI, or we can have a pep-3156 Protocol interface. We can share a bunch of logic either way - e.g. CONTENT_LENGTH etc, but the mechanics of writing middleware might be quite different. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 10:48:37 +1300 Robert Collins robe...@robertcollins.net wrote: On 15 October 2014 10:04, Antoine Pitrou solip...@pitrou.net wrote: ... (that's for the HTTP part; a websockets layer would probably implement a separate transport and accept a separate protocol factory; actually, it could be implemented as a protocol that would parse the websockets protocol and provide its own transport on top of that... there may already be such a thing on the Internet :-)) So thats the bit that I'm having conceptual trouble with - servers may well implement the framing (and I rather think that they have to do so in some deployments), so we need to make sure that we don't do that in this context - we're behind an abstraction. If the new WSGI specifies that the server implements websocket support, then indeed the application shouldn't redo it. The question in mind is whether that is amenable to the same unification and layering that WSGI brought to the HTTP synchronous case. It looks like you'd make a protocol factory decorator. I'm not sure why you would need a decorator. In the PEP 3156 mindset, a protocol factory can be any 0-argument Python callable: a class, a closure, a bound method, a global function, etc. Just pass that object it to the server when asking it to listen (or when it tells you it wants to listen, which is the WSGI approach AFAIU). Will it be sufficiently flexible and interoperable to be something that becomes a lingua franca? I don't know pep-3156 well enough to judge that myself. Yes, it should. That's the formalism already adopted for serving stuff on listening connections, e.g. (from PEP 3156): create_server(protocol_factory, host, port, options). Enters a serving loop that accepts connections. [...] Each time a connection is accepted, protocol_factory is called without arguments(**) to create a Protocol, a bidirectional stream Transport is created to represent the network side of the connection, and the two are tied together by calling protocol.connection_made(transport). Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On 15 October 2014 11:10, Antoine Pitrou solip...@pitrou.net wrote: Each time a connection is accepted, protocol_factory is called without arguments(**) to create a Protocol, a bidirectional stream Transport is created to represent the network side of the connection, and the two are tied together by calling protocol.connection_made(transport). So where would headers etc be supplied to the protocol for reads (and for outputs)? Since the transport isn't the raw socket, its the bodies only. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 11:28:42 +1300
Robert Collins
robe...@robertcollins.net wrote:
On 15 October 2014 11:10, Antoine Pitrou solip...@pitrou.net wrote:
Each time a connection is accepted, protocol_factory is called without
arguments(**) to create a Protocol, a bidirectional stream Transport is
created to represent the network side of the connection, and the two
are tied together by calling protocol.connection_made(transport).
So where would headers etc be supplied to the protocol for reads (and
for outputs)? Since the transport isn't the raw socket, its the bodies
only.
For reads, it could be provided on e.g.
transport.get_extra_info('headers'). Or if you want a flat mapping API,
transport.get_extra_info('http_content_type'), etc.
As for out-bound headers, it's a good question. But I think it's ok for
the transport to have HTTP-specific methods, so
transport.write_header(name, value) could be ok too.
Regards
Antoine.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
