Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 08:40:05 +1300 Robert Collins robe...@robertcollins.net wrote: On 15 October 2014 07:30, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 14 Oct 2014 09:47:35 -0700 Guido van Rossum gu...@python.org wrote: I'm wondering if a small extension to the WSGI protocol might be sufficient to support this: the special environ variable wsgi.async_input could optionally be tied to a standard asyncio stream reader ( https://docs.python.org/3/library/asyncio-stream.html#streamreader), from which bytes can be read using yield from stream.read([nbytes]) or yield from stream.readline(). I think it would be frankly better to hook at the transport/protocol level, and let people wrap that inside an asyncio stream if that's their preference. For things like mod_wsgi and uwsgi, we're not actually implementing the transport or protocol inside of Python at all - its all happening in C and often in an entirely separate process. You may have misunderstood me. I am talking about the Transport and Protocol abstractions defined in PEP 3156. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 09:22:28 +1300
Robert Collins
robe...@robertcollins.net wrote:
You may have misunderstood me. I am talking about the Transport and
Protocol abstractions defined in PEP 3156.
Lets assume I did. Given say nginx + uwsgi + asyncio, you're proposing
that there be a uwsgi-asyncio module that listens to the uwsgi socket
and demuxes packets from that into something that then exposes a
ReadTransport + WriteTransport pair and a Protocol on top of that.
Let's call it uwsgi-pep3156. It shouldn't be asyncio-specific.
[Note Guido's original remark: However, the situation is different
for reading the request body. The app can read this in dribs and drabs
if it wants to, by reading limited amounts of data from
environ['wsgi.input'], but in an asyncio-driven world reading
operations should really be marked with yield from (or perhaps just
yield -- again, I'm sure an adaptor won't have a problem with this).
Ergo, this is about streaming the request and response bodies. Not
asynchronously receiving the headers, etc.]
The server would implement a Transport for the input body stream +
output body stream. It would accept a Protocol factory from the
application/middleware.
Then when a request comes:
protocol = protocol_factory()
transport = Transport(...)
protocol.connection_made(transport)
and when an input body chunk is available:
protocol.data_received(chunk)
and when the input body is finished:
protocol.eof_received()
The protocol would be able to call transport.extra_info(...) to get
HTTP-specific information, e.g. transport.extra_info('headers') perhaps.
(that's for the HTTP part; a websockets layer would probably implement
a separate transport and accept a separate protocol factory; actually,
it could be implemented as a protocol that would parse the websockets
protocol and provide its own transport on top of that... there may
already be such a thing on the Internet :-))
Regards
Antoine.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 10:48:37 +1300 Robert Collins robe...@robertcollins.net wrote: On 15 October 2014 10:04, Antoine Pitrou solip...@pitrou.net wrote: ... (that's for the HTTP part; a websockets layer would probably implement a separate transport and accept a separate protocol factory; actually, it could be implemented as a protocol that would parse the websockets protocol and provide its own transport on top of that... there may already be such a thing on the Internet :-)) So thats the bit that I'm having conceptual trouble with - servers may well implement the framing (and I rather think that they have to do so in some deployments), so we need to make sure that we don't do that in this context - we're behind an abstraction. If the new WSGI specifies that the server implements websocket support, then indeed the application shouldn't redo it. The question in mind is whether that is amenable to the same unification and layering that WSGI brought to the HTTP synchronous case. It looks like you'd make a protocol factory decorator. I'm not sure why you would need a decorator. In the PEP 3156 mindset, a protocol factory can be any 0-argument Python callable: a class, a closure, a bound method, a global function, etc. Just pass that object it to the server when asking it to listen (or when it tells you it wants to listen, which is the WSGI approach AFAIU). Will it be sufficiently flexible and interoperable to be something that becomes a lingua franca? I don't know pep-3156 well enough to judge that myself. Yes, it should. That's the formalism already adopted for serving stuff on listening connections, e.g. (from PEP 3156): create_server(protocol_factory, host, port, options). Enters a serving loop that accepts connections. [...] Each time a connection is accepted, protocol_factory is called without arguments(**) to create a Protocol, a bidirectional stream Transport is created to represent the network side of the connection, and the two are tied together by calling protocol.connection_made(transport). Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and asyncio (tulip)?
On Wed, 15 Oct 2014 11:28:42 +1300
Robert Collins
robe...@robertcollins.net wrote:
On 15 October 2014 11:10, Antoine Pitrou solip...@pitrou.net wrote:
Each time a connection is accepted, protocol_factory is called without
arguments(**) to create a Protocol, a bidirectional stream Transport is
created to represent the network side of the connection, and the two
are tied together by calling protocol.connection_made(transport).
So where would headers etc be supplied to the protocol for reads (and
for outputs)? Since the transport isn't the raw socket, its the bodies
only.
For reads, it could be provided on e.g.
transport.get_extra_info('headers'). Or if you want a flat mapping API,
transport.get_extra_info('http_content_type'), etc.
As for out-bound headers, it's a good question. But I think it's ok for
the transport to have HTTP-specific methods, so
transport.write_header(name, value) could be ok too.
Regards
Antoine.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI: allowing short reads
On Sun, 28 Sep 2014 10:43:33 +1300 Robert Collins robe...@robertcollins.net wrote: It was in the context of improving streamed unpickling, which is a problem a bit similar - but less horrible - to JSON unserializing; since then, the problem was solved in a different way by adding a framing layer to pickle protocol 4 :-). BufferedReader solves this well though, doesn't it? Assuming it's the only one accessing the raw stream, yes. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] A 'shutdown' function in WSGI
Tarek Ziadé ziade.tarek@... writes: On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton graham.dumple...@gmail.com wrote: ... But I don't think you can guarantee that everything is still up in memory by the time atexit gets called, so you can't really call cleanup code there. The only thing which is done prior to atexit callbacks being called is waiting on threads which weren't marked as daemonised. which can lead to completely lock the shutdown if a lib or the program has a thread with a loop that waits for a condition.which it is not the case with signals, since you get a chance to properly stop everything beforehand. That's a buggy lib or program. This has nothing to do with WSGI really. The snippet Graham showed is run at any interpreter shutdown, even when you simply run python in your shell. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] A 'shutdown' function in WSGI
Tarek Ziadé ziade.tarek@... writes: here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505 There are two typos but the effect remains the same since you are locked before you reach those lines: - atexit call worker.stop() instead of worker.join() - in worker.stop(), it calls worker.join() instead of worker.join(self) I find your example convincing. I think that's worth fixing in Python (e.g. by offering an atexit() method on Thread objects). Perhaps you can open a bug? Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Declaring PEP 3333 accepted (CGI example)
Hello, P.J. Eby p...@... writes: if headers_sent: # Re-raise original exception if headers sent -raise exc_info[0], exc_info[1], exc_info[2] +raise exc_info[0](exc_info[1]).with_traceback(exc_info[2]) finally: exc_info = None # avoid dangling circular ref Can somebody weigh in on what the correct translation here is? The only real Python 3 coding I've done to date has been experiments to test changes to other aspects of WSGI. You don't need the with_traceback. Just raise exc_info[1]. The original traceback is already attached to the exception instance (major difference from Python 2). Oh and by the way: headers_set = [] headers_sent = [] This is really a recipe for disaster. Please give these two variables clearly distinct names. Your example is very confusing to read because of this. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 / WSGI 2 Async
Alice Bevan–McGregor al...@... writes: I don't understand why you want a yield at this level. IMHO, WSGI needn't involve generators. A higher-level wrapper (framework, middleware, whatever) can wrap fd-waiting in fancy generator stuff if so desired. Or, in some other environments, delegate it to a reactor with callbacks and deferreds. Or whatever else, such as futures. WSGI already involves generators: the response body. Wrong. The response body is an arbitrary iterable, which means it can be a sequence, a generator, or something else. WSGI doesn't mandate any specific feature of generators, such as coroutine-like semantics, and the server doesn't have to know about them. Everyone has their own idea of what a deferred is, and there is only one definition of a future, which (in a broad sense) is the same as the general idea of a deferred. A Twisted deferred is as well defined as a Python stdlib future; actually, deferreds have been in use by the Python community for much, much longer than futures. But that's besides the point, since I'm proposing that your spec doesn't rely on a high-level abstraction at all. Ratification of PEP 444 is a long way off itself. Right, that's why I was suggesting you drop your concern for Python 2 compatibility. Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 / WSGI 2 Async
Alice Bevan–McGregor al...@... writes: On 2011-01-07 09:04:07 -0800, Antoine Pitrou said: Alice Bevan–McGregor al...@... writes: I don't understand why you want a yield at this level. IMHO, WSGI needn't involve generators. A higher-level wrapper (framework, middleware, whatever) can wrap fd-waiting in fancy generator stuff if so desired. Or, in some other environments, delegate it to a reactor with callbacks and deferreds. Or whatever else, such as futures. WSGI already involves generators: the response body. Wrong. I'm aware that it can be any form of iterable, [snip] Ok, so, WSGI doesn't already involve generators. QED. Right, that's why I was suggesting you drop your concern for Python 2 compatibility. -1 There is practically no reason for doing so; Of course, there is one: a less complex PEP without any superfluous compatibility language sprinkled all over. And a second one: a simpler PEP is probably easier to get contructive comments about, and (perhaps some day) consensus on. esp. considering that I've managed to write a 2k/3k polygot server that is more performant out of the box than any other WSGI HTTP server I've come across and is far simpler in implementation than most of the ones I've come across with roughly equivelant feature sets. Just because you managed to write some piece of code for a *particular* use case doesn't mean that cross-compatibility is a solved problem. If you think it's easy, then I'm sure the authors of various 3rd-party libs would welcome your help achieving it. Python 2.x will be around for a long time. And so will PEP and even PEP 333. People who value legacy compatibility will favour these old PEPs over your new one anyway. People who don't will progressively jump to 3.x. Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 / WSGI 2 Async
Alice Bevan–McGregor al...@... writes: On 2011-01-07 13:21:36 -0800, Antoine Pitrou said: Ok, so, WSGI doesn't already involve generators. QED. This can go around in circles; by allowing all forms of iterable, it involves generators. Geneators are a type of iterable. QED right back. ;) Please read back in context. There isn't any compatibility language sprinkled within the PEP.[...] Using native strings where possible encourages compatibility, [snip] The whole native strings thing *is* compatibility cruft. A Python 3 PEP would only need two string types: bytes and unicode (str). Just because you managed to write some piece of code for a *particular* use case doesn't mean that cross-compatibility is a solved problem. The particular use case happens to be PEP 444 as implemented using an async and multi-process (some day multi-threaded) HTTP server, so I'm not quite sure what you're getting at, here. It's becoming to difficult to parse. You aren't sure yet what the async part of PEP 444 should look like but you have already implemented it? If you think it's easy, then I'm sure the authors of various 3rd-party libs would welcome your help achieving it. I helped proof a book about Python 3 compatibility and am giving a presentation in March that contains information on Python 3 compatibility from the viewpoint of implementing the Marrow suite. Well, I hope not too many people will waste time trying to write code cross-compatible code rather than solely target Python 3. The whole point of Python 3 is to make developers' life better, not worse. Python 2.x will be around for a long time. And so will PEP and even PEP 333. People who value legacy compatibility will favour these old PEPs over your new one anyway. People who don't will progressively jump to 3.x. Yup. Not sure how this is really an issue. PEP 444 is the /future/, 333[3] is /now/ [-ish]. Please read back in context (instead of stripping it), *again*. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 / WSGI 2 Async
Alice Bevan–McGregor al...@... writes: agronholm: what new features does pep 444 propose to add to pep ? \ async, filters, no buffering? GothAlice: Async, filters, no server-level buffering, native string usage, the definition of byte string as the format returned by socket read (which, on Java, is unicode!), and the allowance for returned data to be Latin1 Unicode. Regardless of the rest, I think the latter would be a large step backwards. Clear distinction between bytes and unicode is a *feature* of Python 3. Unicode-ignorant programmers should use frameworks which do the encoding work for them. (by the way, why you are targeting both Python 2 and 3?) agronholm: I'm not very comfortable with the idea of wsgi.input in async apps \ I'm just thinking what would happen when you do environ['wsgi.input'].read() GothAlice: One of two things: in a sync environment, it blocks until it can read, in an async environment [combined with yield] it pauses/shelves your application until the data is available. Er, for the record, in Python 3 non-blocking file objects return None when read() would block. For example: r, w = os.pipe() flags = fcntl.fcntl(r, fcntl.F_GETFL, 0); fcntl.fcntl(r, fcntl.F_SETFL, flags | os.O_NONBLOCK) 0 os.read(r, 1) Traceback (most recent call last): File stdin, line 1, in module OSError: [Errno 11] Resource temporarily unavailable f = open(r, rb) f.read(1) is None True agronholm: the requirements of async apps are a big problem agronholm: returning magic values from the app sounds like a bad idea agronholm: the best solution I can come up with is to have wsgi.async_input or something, which returns an async token for any given read operation The idiomatic abstraction for non-blockingness under POSIX is file descriptors. So, at the low level (the WSGI level), exchanging fds between server and app could be enough to allow both to wake up each other (perhaps two fds: one the server can wait on, one the app can wait on). Similarly to what signalfd() does. Then higher-level tools can wrap inside Futures or whatever else. However, this also means Windows compatibility becomes more complicated, unless the fds are sockets. Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
[Web-SIG] CGI in PEP 444
Alice Bevan–McGregor al...@... writes: [1] http://bit.ly/e7rtI6 So, while we are at it, could we get rid of the CGI server example in this new SWGI spec? This is 2011, and we should promote modern idioms, not encourage people to do 1995 Web programming. 10 years ago, CGI was already frown upon. (and even the idea that WSGI should provide some kind of CGI compatibility sounds a bit ridiculous to me) Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
[Web-SIG] decoding environ
Ian Bicking i...@... writes: So... there's been some discussion of WSGI on Python 3 lately. I'm not feeling as pessimistic as some people, I feel like we were close but just didn't *quite* get there. Here's my thoughts: * Everyone agrees keys in the environ should be native strings I don't know how that related to WSGI but it should be noted that Python 3.2 comes with two synchronized views of the environment: os.environ (str - str mapping) and os.environb (bytes - bytes mapping). See http://docs.python.org/dev/library/os.html#os.environ Also, the way os.environ is decoded from bytes values involves the surrogateescape error handler, which ensures that non-decodeable bytes get their own unicode escape sequences, and can get re-encoded losslessly: http://www.python.org/dev/peps/pep-0383/ Regards Antoine. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
