Re: [Web-SIG] Collating follow-up on the future of WSGI
I am not speaking about websockets. You could use it for SSE, or some apps could use the Upgrade header to upgrade from http to their own protocol etc... The only discussion i saw about websockets are about the addition of an async api or an external api. I am not describing that. I am speaking about providing a low level abstraction like wsgi.input but adding to it the support of output. (I was referring to wsgi.multithread...). This low level interface would allow anyone to provide its own implementation(server) or usage (application) still acting as a *gateway* . Also who are "we"? I am starting to think the discussion is already done and only obscure details like the content_length or headers encoding should be discussed. The RAW_SOCKET have been added on demand of the gunicorn users. Such thing also exist in things like cherrypy if I remember. A lot of code around have been created over it. So before deciding it's unworkable or whatever I strongly invite you to consider it as an addition to the environ. And since some servers need to pass the data differently I then suggest a Resource object on which you can read and write and eventually poll. This is not a websocket but more a proxy ressource to the client connexion. I will come back asap with a small spec. I also propose a second addition to the protocol that formalize the addition of extensions to the protocol by the servers if they want to. Having for example something like "`wsg.extensions` . Such addition would help anyone to experiment changes over the wsgi before making such changes in the specification by itself possibly. I think we have a good opportunity to extend the WSGI specification to allow the users to take over the new challenges on the web without forcing them to use a concurrency mode or skip completely the WSGI spec. The interest I see in WSGI is its simplicity and low level interface allowing users to build whatever they want over it. The different workers and their support of different concurrency models and framework in gunicorn let me think it is possible. Are the participants of this thread ready to discuss it? - benoît On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > again. any server can do such implementation if we create a new Resource > abstraction. This abstraction would expose a common api to read and write. > The implementation would be specific to the server. > > > If you mean not exposing the raw socket and having a separate high level > API for implementing something like WebSocket this was already talked > about. The suggestion was that it should not be a part of WSGI. Develop > that API independently with no link to WSGI. The idea of upgrading from > WSGI to a different API isn’t practical for various WSGI servers as it > isn’t possible to unwind the state of the connection path created to get to > point of handling the WSGI application. The better scenario is that the > switch to an alternate WebSocket API is handled completely within the web > server however it needs to handle it, when it needs to handle it, and not > be reliant on going into a WSGI application which then says, oh, I actually > need that to be WebSocket. > > Now like we have wsgi.thread I would instead suggest to add a system of > capability or extension like in smtp, imap, ... so the servers that > implement a specific extension can legally published it. Would it work for > you? > > > Since there is nothing in WSGI environ called wsgi.thread now I have no > idea what you are really suggesting here. > > Graham > > benoit > > On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com> > wrote: > >> >> On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: >> >> >> >> On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net> >> wrote: >> >>> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> >>> wrote: >>> >>> > >>> > not at all. But I made the assumption that the wsgi server maintained a >>> > thread directly or not where the python application is running . >>> > >>> > In any case there is some sort of wrapping done in the same >>> thread/process >>> > where the python application is running. And then nothing stop to give >>> the >>> > socket away to the application and tell to the server to stop to >>> communicate >>> > with it. >>> >>> What socket? >>> >>> Data could be being passed by shm, for instance. >>> >>> -Rob >>> >>> >> Whil
[Web-SIG] Fwd: Collating follow-up on the future of WSGI
-- Forwarded message - From: Benoit Chesneau <bchesn...@gmail.com> Date: Thu, 21 Jan 2016 at 06:12 Subject: Re: [Web-SIG] Collating follow-up on the future of WSGI To: Graham Dumpleton <graham.dumple...@gmail.com> I am not speaking about websockets. You could use it for SSE, or some apps could use the Upgrade header to upgrade from http to their own protocol etc... The only discussion i saw about websockets are about the addition of an async api or an external api. I am not describing that. I am speaking about providing a low level abstraction like wsgi.input but adding to it the support of output. (I was referring to wsgi.multithread...). This low level interface would allow anyone to provide its own implementation(server) or usage (application) still acting as a *gateway* . Also who are "we"? I am starting to think the discussion is already done and only obscure details like the content_length or headers encoding should be discussed. The RAW_SOCKET have been added on demand of the gunicorn users. Such thing also exist in things like cherrypy if I remember. A lot of code around have been created over it. So before deciding it's unworkable or whatever I strongly invite you to consider it as an addition to the environ. And since some servers need to pass the data differently I then suggest a Resource object on which you can read and write and eventually poll. This is not a websocket but more a proxy ressource to the client connexion. I will come back asap with a small spec. I also propose a second addition to the protocol that formalize the addition of extensions to the protocol by the servers if they want to. Having for example something like "`wsg.extensions` . Such addition would help anyone to experiment changes over the wsgi before making such changes in the specification by itself possibly. I think we have a good opportunity to extend the WSGI specification to allow the users to take over the new challenges on the web without forcing them to use a concurrency mode or skip completely the WSGI spec. The interest I see in WSGI is its simplicity and low level interface allowing users to build whatever they want over it. The different workers and their support of different concurrency models and framework in gunicorn let me think it is possible. Are the participants of this thread ready to discuss it? - benoît On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > again. any server can do such implementation if we create a new Resource > abstraction. This abstraction would expose a common api to read and write. > The implementation would be specific to the server. > > > If you mean not exposing the raw socket and having a separate high level > API for implementing something like WebSocket this was already talked > about. The suggestion was that it should not be a part of WSGI. Develop > that API independently with no link to WSGI. The idea of upgrading from > WSGI to a different API isn’t practical for various WSGI servers as it > isn’t possible to unwind the state of the connection path created to get to > point of handling the WSGI application. The better scenario is that the > switch to an alternate WebSocket API is handled completely within the web > server however it needs to handle it, when it needs to handle it, and not > be reliant on going into a WSGI application which then says, oh, I actually > need that to be WebSocket. > > Now like we have wsgi.thread I would instead suggest to add a system of > capability or extension like in smtp, imap, ... so the servers that > implement a specific extension can legally published it. Would it work for > you? > > > Since there is nothing in WSGI environ called wsgi.thread now I have no > idea what you are really suggesting here. > > Graham > > benoit > > On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com> > wrote: > >> >> On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: >> >> >> >> On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net> >> wrote: >> >>> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> >>> wrote: >>> >>> > >>> > not at all. But I made the assumption that the wsgi server maintained a >>> > thread directly or not where the python application is running . >>> > >>> > In any case there is some sort of wrapping done in the same >>> thread/process >>> > where the python application is running. And then nothing stop to give >>> the >>> > socket away to the application and tell to the serve
Re: [Web-SIG] Collating follow-up on the future of WSGI
because i am not speaking about making a specification, but a way to expose in the API (environ) custom extensions that a server want to experiment. there are actually no easy way except checking "wsgi." indeed but that doesn't make it as clear as a separate namespace where to put all server extensions could be. Like capability field is in imap world. Also I am not trying to force anything, I want to discuss about a possible update of the wsgi spec which I thought was this thread about. What I just want to discuss is the *current* usage of some extensions that have been rapidly dismissed as unworkable. Like I said I will come with a more formal specification about them but I wanted to discuss first about and collect counter arguments which are good too. - benoît On Thu, 21 Jan 2016 at 07:24, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > I am still confused as to why you keep talking as if you seemingly are > trying to force extensions to the existing WSGI specification into the core > WSGI specification when an alternative has already been cited. > > Extensions and the process for describing them and getting them accepted, > plus the appropriate WSGI environ prefix, as has been mentioned before, is > what is covered by: > > http://wsgi.readthedocs.org/en/latest/specifications.html > > Being an extension means it is entirely optional for a WSGI server to try > and implement it, thus allowing WSGI servers/adapters that cannot implement > something to skip them. They stand as separate documents and would never > become part of the core WGSI PEP. > > Is there an issue with doing extensions per the process, and in the WSGI > environ namespace, that was outlined in that URL? You seem to be suggesting > a completely new way of handling extensions and ignoring what was laid down > before. > > So no one is saying you can’t have extensions, and that separate process > gives you all the scope you need to do it. > > In drafting your specification just fit it reference to what is described > in that URL, using ‘x-wsgiorg.’ prefix keys. > > Graham > > > On 21 Jan 2016, at 4:13 PM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > I am not speaking about websockets. You could use it for SSE, or some apps > could use the Upgrade header to upgrade from http to their own protocol > etc... The only discussion i saw about websockets are about the addition of > an async api or an external api. I am not describing that. I am speaking > about providing a low level abstraction like wsgi.input but adding to it > the support of output. (I was referring to wsgi.multithread...). This low > level interface would allow anyone to provide its own > implementation(server) or usage (application) still acting as a *gateway* . > > Also who are "we"? I am starting to think the discussion is already done > and only obscure details like the content_length or headers encoding should > be discussed. The RAW_SOCKET have been added on demand of the gunicorn > users. Such thing also exist in things like cherrypy if I remember. A lot > of code around have been created over it. So before deciding it's > unworkable or whatever I strongly invite you to consider it as an addition > to the environ. And since some servers need to pass the data differently I > then suggest a Resource object on which you can read and write and > eventually poll. This is not a websocket but more a proxy ressource to the > client connexion. I will come back asap with a small spec. > > I also propose a second addition to the protocol that formalize the > addition of extensions to the protocol by the servers if they want to. > Having for example something like "`wsg.extensions` . Such addition would > help anyone to experiment changes over the wsgi before making such changes > in the specification by itself possibly. > > I think we have a good opportunity to extend the WSGI specification to > allow the users to take over the new challenges on the web without forcing > them to use a concurrency mode or skip completely the WSGI spec. The > interest I see in WSGI is its simplicity and low level interface allowing > users to build whatever they want over it. The different workers and their > support of different concurrency models and framework in gunicorn let me > think it is possible. Are the participants of this thread ready to discuss > it? > > - benoît > > On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com> > wrote: > >> On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: >> >> again. any server can do such implementation if we create a new Resource >> abstraction. This abstraction would expose a common api to read and write
Re: [Web-SIG] Collating follow-up on the future of WSGI
On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net> wrote: > On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> wrote: > > > > > not at all. But I made the assumption that the wsgi server maintained a > > thread directly or not where the python application is running . > > > > In any case there is some sort of wrapping done in the same > thread/process > > where the python application is running. And then nothing stop to give > the > > socket away to the application and tell to the server to stop to > communicate > > with it. > > What socket? > > Data could be being passed by shm, for instance. > > -Rob > > While shared memory would be quite a bad idea, then why not. I still don't see why having a way to upgrade the connection can't be done. Call it I/O resource or Socket, the issue is the same. At the end nothing stop the server to pass the control to the app. If we forget the socket (which is btw the simplest design) then the server could stop to control the I/O resource when the application ask it to do it. At some point either a garbage collection or a basic resource return/claim flow could be used to definitely free the resource. The thing behind that is that it would allow the WSGI spec to only focus on providing a strict gateway workflow without forcing the application to adopt a concurrency model aync or not. - benoit. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Collating follow-up on the future of WSGI
again. any server can do such implementation if we create a new Resource abstraction. This abstraction would expose a common api to read and write. The implementation would be specific to the server. Now like we have wsgi.thread I would instead suggest to add a system of capability or extension like in smtp, imap, ... so the servers that implement a specific extension can legally published it. Would it work for you? benoit On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > > On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > > > On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net> > wrote: > >> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> wrote: >> >> > >> > not at all. But I made the assumption that the wsgi server maintained a >> > thread directly or not where the python application is running . >> > >> > In any case there is some sort of wrapping done in the same >> thread/process >> > where the python application is running. And then nothing stop to give >> the >> > socket away to the application and tell to the server to stop to >> communicate >> > with it. >> >> What socket? >> >> Data could be being passed by shm, for instance. >> >> -Rob >> >> > While shared memory would be quite a bad idea, then why not. I still don't > see why having a way to upgrade the connection can't be done. > > Call it I/O resource or Socket, the issue is the same. At the end nothing > stop the server to pass the control to the app. If we forget the socket > (which is btw the simplest design) then the server could stop to control > the I/O resource when the application ask it to do it. At some point either > a garbage collection or a basic resource return/claim flow could be used to > definitely free the resource. > > The thing behind that is that it would allow the WSGI spec to only focus > on providing a strict gateway workflow without forcing the application to > adopt a concurrency model aync or not. > > > No one has said you cannot do it. because though it is only able to be > implemented in a subset of WSGI servers/adapters, then it doesn’t seem > appropriate that it be a part of the core WSGI specification. > > This is the role of a WSGI extension as found at: > > http://wsgi.readthedocs.org/en/latest/specifications.html > > So go talk to the authors of uWSGI, and the other couple of packages > available for trying to plug these into some of the pure Python based WSGI > servers and come to an agreement between yourselves as to a standard way of > doing it and the extension specification can be added to the wsgi.org > site. > > Graham > > ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Collating follow-up on the future of WSGI
On Tue, Jan 19, 2016 at 11:58 PM Robert Collinswrote: > On 20 January 2016 at 05:55, Cory Benfield wrote: > > All, > > > > Thanks so much for your feedback to my original request for comments on > the future of WSGI. You provided a ton of really useful feedback: when > printed out on my printer it ended up at about 50 pages of information that > was really engaging reading. I also want to thank you all for keeping the > tone of the discussion so positive. On an topic like this one it can get > tricky and emotionally charged very easily, and you did a great job of > avoiding that problem. > > > > I spent a few hours this morning going over your feedback and trying to > extract some common threads. Altogether I believe that most participants > were mostly in agreement over the direction we should take, with a few > outliers in each case. I’d like to summarise what I believe were the big > take-aways from that discussion to confirm that I’ve understood everyone. > > > > I’d also like the members of this SIG to take this opportunity to > discuss these proposals more concretely. Rather than expressing our > sentiments about WSGI and its future more generally, I want people to > critique and offer opinions on *these specific proposals*. The goal here is > to get an understanding of whether these are worth doing, how we need to > prioritise the work, and whether there’s anything we’re missing. > > > > The below is formatted in restructured text for clarity, because it’s > quite a lot of information. > > > > > > Concrete Proposals > > == > > > > WSGI > > > > > > Overall, there was strong and fairly unanimous sentiment that WSGI > itself should more or less be left alone. Minor adjustments would be > valuable, and we should pursue them, but the contributors do not believe > that making a substantial revision to WSGI would be adviseable. > > > > I therefore propose we revise the WSGI specification to WSGI 1.1 and > then consider it "final". The following points were raised for revising > WSGI. > > Sure. > > > Asynchronous WSGI > > ~ > > > > This was generally regarded as too substantial a change to shoehorn into > WSGI. > > > > Benoit proposed that we could achieve this change by adding a correlator > to WSGI. This would allow servers to associate a given call to ``write()`` > with a specific request/response (most useful in HTTP/2 where this could > correspond to a stream ID). This would then allow WSGI to transition to a > purely callback-based model that could in principle cohabitate with an > async protocol in Python. > > Uh, WSGI doesn't need to be involved in this. Servers can already pass > arbitrary objects as write, permitting any correlation they want. I'd > want to see something very specific to consider the impact in terms of > WSGI, or WSGI-NG or whatever we do in future. > > > This proposal is worth highlighting not becuase I think we should pursue > it with our revision of WSGI, but because it's worth considering for any > future specification we come up with. Note also that Graham pointed out > that this would require some careful rethinking of reads from > ``wsgi.input``. > > > > > Server Push > > ~~~ > > > > We can support HTTP/2 server push using Link headers. This could > optionally be supplemented by defining a WSGI extension that provides a > callable for doing server push, which would be a mild improvement over the > Link header approach. A simple proposal for how to do this can easily be > implemented without revising the WSGI specification, and may not even need > to be enshrined in a PEP. However, if we revise the WSGI PEP we may want to > provide a small section that indicates how to add these headers. > > Since the point of WSGI is interop, we should document the expected > interop fashion for this, whatever it is. > > > Socket Escape Hatch > > ~~~ > > > > Aside from Benoit, server operators were unanimously dismissive of the > idea of a socket 'escape hatch'. In general it seems like servers would not > be capable of achieving this. I think, therefore, this idea is unworkable. > > I thin Benoit is making assumptions about what a 'server' is that are > not implied by WSGI - and this is leading to some confusion. Servers > are not necessarily Python processes :). > > not at all. But I made the assumption that the *wsgi* server maintained a thread directly or not where the python application is running . In any case there is some sort of wrapping done in the same thread/process where the python application is running. And then nothing stop to give the socket away to the application and tell to the server to stop to communicate with it. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Collating follow-up on the future of WSGI
On Tue, Jan 19, 2016 at 11:34 PM Graham Dumpleton < graham.dumple...@gmail.com> wrote: > > On 20 Jan 2016, at 8:29 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > > > On Tue, Jan 19, 2016 at 10:49 PM Graham Dumpleton < > graham.dumple...@gmail.com> wrote: > >> >> On 20 Jan 2016, at 7:43 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: >> >> I will make a more complete answer soon. But about: >> >> >>> >>> Socket Escape Hatch >>> ~~~ >>> >>> Aside from Benoit, server operators were unanimously dismissive of the >>> idea of a socket 'escape hatch'. In general it seems like servers would not >>> be capable of achieving this. I think, therefore, this idea is unworkable. >>> >>> >> Well it does work. This is how websockets works in gunicorn. Escape is >> not the right term anyway. Think it as a socket *upgrade*. And then you >> would wonder why it would be unworkable. After all this is how SSL sockets >> works, so is the protocol negotiation in http2 ... >> >> There is nothing magic there until you try to over engineer the stuff. >> Upgrading a sockets means that you tell to the server to forget it. This is >> how most concurrent servers work today. >> >> >> The problem was that it would only work in a WSGI server where the >> original request was accepted on a socket in the same process as the WSGI >> application is running. It cannot work where where the WSGI application is >> behind a bridging protocol. >> > >> So it can’t work for CGI, SCGI, FASTCGI, mod_wsgi daemon mode and >> possibly other implementations. >> > > uh? But we don't care about bridging protocols. In WSGI (the gateway), the > server accept the socket and anyway pass it to the application via actually > a wrapper. Then expect a response from the application. > > Upgrading a socket would simply mean that the server will forget it (and > then consider its job done) once it got an appropriate response from the > application. How this is unworkable? > > > Bridging protocols such as FASTCGI do not provide an ability to upgrade > the connection end to end. > > That is, yes you could pass the raw socket to the WSGI application when > behind FASTCGI, but you are passing it a socket from same process where > data being received (and expected to be sent), is using FASTGCI message > frames. It is not a raw HTTP socket connection. > > There is no way to send a message back to the front end side of the > bridged connection where the raw HTTP socket is, to tell the client side of > the FASTCGI implementation to stop treating it as a FASTCGI connection to > backend process and then suddenly start acting as a raw socket pass through. > > But how it is related to the WSGI protocol? It is expected that the wsgi server accept on a socket (directly or behind a proxy) and provide enough to the application so they can read and write. Upgrading or escaping in WSGI/Server sense would mean that it would skip the dialog with the application once the application told it to do. For the rest, the application would do what it want on the socket, even giving it to another process/thread. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Collating follow-up on the future of WSGI
On Tue, Jan 19, 2016 at 10:49 PM Graham Dumpleton < graham.dumple...@gmail.com> wrote: > > On 20 Jan 2016, at 7:43 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > I will make a more complete answer soon. But about: > > >> >> Socket Escape Hatch >> ~~~ >> >> Aside from Benoit, server operators were unanimously dismissive of the >> idea of a socket 'escape hatch'. In general it seems like servers would not >> be capable of achieving this. I think, therefore, this idea is unworkable. >> >> > Well it does work. This is how websockets works in gunicorn. Escape is > not the right term anyway. Think it as a socket *upgrade*. And then you > would wonder why it would be unworkable. After all this is how SSL sockets > works, so is the protocol negotiation in http2 ... > > There is nothing magic there until you try to over engineer the stuff. > Upgrading a sockets means that you tell to the server to forget it. This is > how most concurrent servers work today. > > > The problem was that it would only work in a WSGI server where the > original request was accepted on a socket in the same process as the WSGI > application is running. It cannot work where where the WSGI application is > behind a bridging protocol. > > So it can’t work for CGI, SCGI, FASTCGI, mod_wsgi daemon mode and possibly > other implementations. > uh? But we don't care about bridging protocols. In WSGI (the gateway), the server accept the socket and anyway pass it to the application via actually a wrapper. Then expect a response from the application. Upgrading a socket would simply mean that the server will forget it (and then consider its job done) once it got an appropriate response from the application. How this is unworkable? - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Collating follow-up on the future of WSGI
I will make a more complete answer soon. But about: > > Socket Escape Hatch > ~~~ > > Aside from Benoit, server operators were unanimously dismissive of the > idea of a socket 'escape hatch'. In general it seems like servers would not > be capable of achieving this. I think, therefore, this idea is unworkable. > > Well it does work. This is how websockets works in gunicorn. Escape is not the right term anyway. Think it as a socket *upgrade*. And then you would wonder why it would be unworkable. After all this is how SSL sockets works, so is the protocol negotiation in http2 ... There is nothing magic there until you try to over engineer the stuff. Upgrading a sockets means that you tell to the server to forget it. This is how most concurrent servers work today. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest
On Wed, Jan 6, 2016 at 10:19 AM Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > Hello Benoît, > > Thanks for clarifying that you also had the reverse problem in mind, > headers sent by applications. This side is less problematic in the sense > that application authors can adapt to stronger requirements. > > In general this is a bit of a mess due to differences between what the RFC > 2616 says and what browsers do in practice. That’s why I believe the > pragmatic solution is to exchange bytes. (This isn’t a major issue in the > grand scheme of things anyway.) > > Best regards, > > -- > Aymeric. > > Since sockets are only accepting bytes in python 3.x, having everything as bytes would also allow the server to process the response more easily... - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest
On Wed, Jan 6, 2016 at 10:29 AM Graham Dumpleton <graham.dumple...@gmail.com> wrote: > > On 6 Jan 2016, at 12:13 AM, Benoit Chesneau <bchesn...@gmail.com> wrote: > > So for me what should be WSGI 2? WSGI 2 should add against WSGI 1 the > following: > > - tell to the application it is actually an HTTP2 request (maybe > populating a wsgi.http2 true env) > > > In CGI implementations you would for HTTP/1.1 already get: > > SERVER_PROTOCOL: 'HTTP/1.1’ > > Under HTTP/2 when I tested some time back, I recollect it came through as > one would assume would be expected: > > SERVER_PROTOCOL: ‘HTTP/2’ > > Is there any reason that this existing CGI variable wouldn’t be sufficient > for this purpose? > > Graham > True. I don't think there is a need for another env. The current way also work for the future :) - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Draft 2: WSGI Response Upgrade Bridging
On Mon, Oct 13, 2014 at 11:52 AM, Robert Collins robe...@robertcollins.net wrote: On 13 October 2014 17:12, Benoit Chesneau bchesn...@gmail.com wrote: ... OK, So I should probably know you, but I can't recollect right now what you do or write. Its not clear to me who you were replying to. I answered at the bottom of the thread so to PJE. Anyway I find it really disturbing the way you're actually acting and try to push your ideas based on private feedback coming from unknown or choosing who should be a reference. That certainly not the right way to have all actors on the table. Because if we go for a new WSGI spec, you certainly want it. And I am speaking as one of these actors. As I said when folk talked about going private in the first thread on this, I'm willing to discuss anything publically or privately - I can't tell folk where they will be comfortable discussing things. But I'm going to do *my* work on this in public, because I think that is essential to get broad consensus. In my opinion, if we want to go further we should first define what are the problem we want to solve, and then get the feedback from all the actors around: I think I've been fairly clear about the problem *I* want to solve. We want to create a clean common API for applications and middleware written in a post HTTP/2 world - where single servers may accept up to all three of HTTP/1.x, HTTP/2 and Websocket connections, and applications and middleware want to be able to take advantage of HTTP/2 and websockets when available, but also degrade gracefully. We also want to ensure that there is a graceful incremental path to adoption of the new API, including Python 2.7 support, and shims to enable existing WSGI apps/middleware/servers to respectively be contained, contain-or-be-contained and contain, things written to this new API. We want a clean, fast and approachable API, and we want to ensure that its no less friendly to work with than WSGI, for all that it will expose much more functionality. By which problem we need to solve, I mean we need to identify clearly what are the problem not solved by the current spec. And see why, and how it is actually solved in the python world. we need to clearly identify these issues and make sure we have a comprehensive view of them. - framweorks authors I reached out to a number of such authors directly. I encourage you to do the same. I could do that eventually if we are all agree on the process. - libraries author Ditto and - server authors Ditto :). If you don't have all actors around and majors are missing, there is probably no point to continue. I do think the idea of having a repository to collect it with people arbitrating the discussions on them on the mailing is a good way to go further. Now I think we are still missing of a clear definition of the problem. This is from what we should start instead of starting by giving our philosophy on how the problem should be solved. Here's my definition of some of the problems: A - there is no common spec equivalent to WSGI that permits writing server side code that takes advantage of HTTP/2. There's *a* http/2 server out there which one can write code for, but that code is either specific to that servers plumbing, or plain WSGI and misses the HTTP/2 goodness. B - WSGI has some oddness and overheads due in large part to the situation it was aiming to fix (which it broadly did) that perhaps we can now come together to fix. C - Support for chunked uploads, comet, bosh and websockets is effectively impossible within WSGI - one ends up writing server specific code, and being tied to a single server - even though multiple servers support (some of) those things. This defeats the point of WSGI IMNSHO: its not that WSGI is broken or anything, its just that we're once again writing all our generic middleware in server-specific fashions. Because the world has moved on and we haven't. Chunkedn upload is possible and already handled with Gunicorn. But there is no standard for that. For C I would separate it from the rest. This a different discussion and imo not everything can be achieved at the same time. Maybe we should start first by fixing them, then go for the next step anyway. So the transition could be incremental in servers and frameworks and actually fix the current spec. For A (And C), i think we should keep the new specification enough agnostic. Especially since HTTP 2 is not yet completely out. - benoit I ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] REMOTE_ADDR and proxys
On Sun, Oct 12, 2014 at 11:38 PM, Robert Collins robe...@robertcollins.net wrote: On 30 September 2014 11:47, Alan Kennedy a...@xhaus.com wrote: [Robert] So it sounds like it should be the responsibility of a middleware to renormalize the environment? In order for that to be the case, you have strictly define what normalization means. For a given deployment its well defined. I agree that in general its not. I believe that it is not possible to fully specify normalization, and that any attempt to do so is futile. If you want to attempt it for the specific scenarios that your particular application has to deal with, then by all means code your version of normalization into your application. Or write some middleware to do it. But trying to make normalization a part of a WSGI-style specification is impossible. I don't recall proposing that it should be in a WSGI-style spec. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com All this issue looks like the problem raised (and not yet solved) recently in Gunicorn when the REMOTE_ADDR has been handled more strictly and we removed all the X-Forward-* headers handling: https://github.com/benoitc/gunicorn/issues/797 There is another case to take in consideration, when your server is answering on unix sockets, so you don't have any TCP address to present. For now we answer with an empty field. Also some application frameworks recently removed the middleware handling X-Forward-* headers. I wonder why. There is an RFC for forward headers: http://tools.ietf.org/html/rfc7239 . For me instead of trying to change the strict behaviour of REMOTE_ADDR I wonder if we shouldn't rather add a new field to the environ. Thoughts? - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Draft 2: WSGI Response Upgrade Bridging
On Fri, Oct 10, 2014 at 9:10 PM, PJ Eby p...@telecommunity.com wrote: On Fri, Oct 10, 2014 at 8:56 AM, Graham Dumpleton graham.dumple...@gmail.com wrote: So PJE, please step back and do not go rushing out to create a PEP. That is the worst thing you could do at this point and will only serve to deter people from the community contributing and so stifle proper discussion about this whole topic. Huh? Have you *read* the PEP? The entire point of it is to provide a basis for *experimenting* with new standards, not to stifle discussion of them. It's not even an *API*, for heavens' sake. It's just a description of how to upgrade to new standards from within existing WSGI frameworks, without needing to tunnel responses and without breaking subrequest middleware. IOW, it's a WSGI *1.0* server extension protocol, and a fairly *minor* one at that. (Indeed, it's little more than an enhanced variation of wsgi.file_wrapper!) It's not any sort of competitor or alternative to what Robert's working on; it's a *stepping stone* for what Robert's working on. In early discussion with Robert -- both here and on github -- it became apparent to me that restricting post-WSGI specifications to what can be achieved in WSGI 1 tunneling is a bad idea. So I've created a *bridging* specification, that allows post-WSGI APIs to be accessed from within WSGI-based apps and frameworks. That's *all*. All of the things you've mentioned as being in scope for discussion, are *still* in scope for discussion. All *this* proposal does is show how those things could be *accessed*, *today*, from inside existing web apps and frameworks, once those new APIs exist. You have no more experience or mandate to be specifying a standard for this than anyone else. If by this you're referring to HTTP/2 or some other new post-WSGI API, then I agree with you. But that's not what the PEP is about. By creating a PEP though that gets perceived by many as meaning the discussion is over. This is exactly what you did for PEP and which caused previous discussion about improving WSGI to get shutdown. That's an interesting perspective, but I don't see how it can be reconciled with the facts. First off, I didn't write a new PEP; I wrote up some of *your* proposed clarifications for Python 3 as WSGI 1.0.1, which was intended to add new clarifying text to PEP 333, *not* to create a new PEP. It was *Guido* who said it must be a new PEP, as you will see here: https://mail.python.org/pipermail/web-sig/2010-September/004691.html and here (where he even says, Don't see this as a new spec. See it as a procedural issue.): https://mail.python.org/pipermail/web-sig/2010-September/004694.html Second, I didn't make anybody stop discussing alternatives for moving things forward. Nobody *ever* said to stop working on a version 2 or even 1.1, certainly not me. See for example, this message, where I agreed with Ian's POV that there was room for both PEP 333 fixes *and* continued work on PEP 444: https://mail.python.org/pipermail/web-sig/2010-September/004662.html Third, and finally, as far as I can tell from the record of the discussion back then, it was you -- and *only* you -- who suggested that the acceptance of PEP meant the discussion was *over*. Indeed, on your blog you actually pushed back at Alice for bringing up more PEP 444 discussion! Nonetheless, discussion of PEP 444 and async APIs and such proceeded well past the introduction of PEP , even without its original authors' participation. And, ironically enough, your posts show up in that discussion, bemoaning that Alice (the new PEP 444 champion) was creating confusion by calling that proposal WSGI 2.0! The result was that the only thing that really got addressed in PEP was Python 3 compatibility and a lot of the other bits of the WSGI specification which are poorly defined, contradictory or restrictive and which cause WSGI server and application developers pain never got addressed. If that prior discussion hadn't been shutdown in that way, we could have been using a better defined and improved WSGI years ago already. Those things didn't get addressed because *you* didn't take up the lead -- a lead which I more than once mentioned you should take up. For example, as I said in https://mail.python.org/pipermail/web-sig/2010-September/004693.html : The full list of things Graham and others have asked for or recommended would indeed require a 1.1 version at minimum, and thus a new PEP. But I really don't want to start down that road right now, and therefore hope that I can talk Graham or some other poor soul into shepherding a 1.1 PEP instead. ;-) You didn't, and haven't, taken up that slack. What you've consistently done is mutter and grumble on the sidelines about how it's never going to happen and disclaim any responsibility for actually writing a proposal
Re: [Web-SIG] WSGI2: write callable?
On Fri, Sep 26, 2014 at 9:58 PM, PJ Eby p...@telecommunity.com wrote: On Thu, Sep 25, 2014 at 11:32 PM, Robert Collins robe...@robertcollins.net wrote: So I propose we drop the write callable, and include a queue based implementation in the adapter for PEP- code. If you're dropping write(), then you might as well drop start_response() altogether, and replace it with returning a (status, headers, body-iterator) tuple, as in wsgi_lite ( https://github.com/pjeby/wsgi_lite ) or as found in other languages' versions of WSGI. (start_response+write was only ever needed in order to support legacy apps, so other languages never bothered.) wsgi_lite has a couple of other protocol extensions, namely the 'wsgi_lite.closing' environment key, flagging callables' supported WSGI version (for transparent interop), and the argument binding protocol, but for the most part these are orthogonal to the calling schema. I would suggest, however, that the calling protocol be flagged in some way to allow easier interop. I quite like the idea of always returning an iterator for the body it would simplify the code a lot... About returning the status and other thing, I quite agree, but imo we also need to return an extra parameter where the application or the middleware could maintain a state or something like it. Thoughts? - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI for HTTP/2.0 ?
Hi, I would prefer to have this work being done transparently. If we do it rationally it could work imo. Anyway before thinking to change the protocol or criticizing it maybe we could first collect the requirements in HTTP 2 (stream and such) so we can think about possible implementations. And see what it misses in WSGI. I am thinking we could adopt the same path used to decided to go for HTTP 1.x or HTTP 2 on the client part. Ie keeping WSGI and PEP for HTTP 1.1 applications and go for a new interface in HTTP2. But such decision should be done once we have a clear view of what requires HTTP 2 and how it can be handled on the python side. Thoughts? - benoit On Sat, Sep 20, 2014 at 8:31 AM, Graham Dumpleton graham.dumple...@gmail.com wrote: On 20/09/2014, at 3:49 PM, Roberto De Ioris robe...@unbit.it wrote: I can help a bit (i am the uWSGI lead developer and a nginx and Cherokee contributor, and i have already implemented a spdy3 server last year) I honestly think that WSGI by itself needs a complete rewrite/rethink to be adapted to modern (ok someone could say 'fashioned') patterns (that are somewhat more 'urgent' than HTTP/2), but i agree that starting thinking about HTTP/2 could be a good thing. I agree. Overhauling WSGI has more relevance because an underlying web server updating itself to support HTTP 2.0 will in the main have little relevance at the application layer as the web server is more than likely to have an adapter layer which makes things look the same to existing modules/protocol adapters. In other words, Apache adding support for HTTP 2.0 isn't going to result in some sort of wholesale change of the Apache module interface, it would stay the same say whether HTTP 2.0 is used, especially just as an alternate way of doing the same thing as HTTP 1.1. In that respect, since no HTTP 2.0 specific functionality is going to be made visible through exist interfaces, then Apache modules or adapters for FASTCGI/SCGI etc or even mod_wsgi are simply not going to change. So, overhaul WSGI as the primary aim, but within that factor in things to allow for HTTP 2.0 functionality. The problem with trying to overhaul WSGI is that if it is done in an open forum like the Web-SIG it will die of a thousand cuts, as past efforts to update it in even minor ways have suffered. The only way that WSGI itself will ever see an overhaul is through the strong willed determination of a few people off list, out of sight, to allow it it to be fully fleshed out, with input coming from direct consultation with and review by other related parties who have a vested interested or significant experience in the area. I may be up for such an off list effort, but be warned I may want to run roughshod over it and exert quite a lot of influence over the process and outcome. :-) Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI for HTTP/2.0 ?
On Sat, Sep 20, 2014 at 3:22 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Sat, Sep 20, 2014 at 9:23 AM, Robert Collins robe...@robertcollins.net wrote: Well, thats certainly a challenge :). Whats the governance model here? Is a PEP appropriate, and if so - that gives us a BFDL or BFDL PEP-delegate to decide between bikeshed issues; and if its not a bikeshed issue then resolving it is actually necessary. Yes, I think a good way forward would be to have a small cabal write a PEP and then announce it here for further feedback and then pronouncement by a BDFL or -delegate. If you want to be lead/editor, that sounds great. It also seems like you should definitely involve Graham and give credence to his thoughts. I'd be excited about this and happy to give feedback a little later once you've got some initial draft, as someone who likes to implement his applications directly on top of WSGI for now (but I've also implemented a couple of WebSocket servers). Cheers, Dirkjan Last time was more about everyone wanted to discuss about the changes with its own requirement list. Which is already conflicting. Discussing if it should be done outside the media designed for it is already out of topic imo. So I won't discuss about it further. Instead I wonder what is the appropriate medium to collect requirements and others stuffs about it. Wiki ? Anything else? For a start I see these different topics 1) HTTP 1.1 vs HTTP 2: - HTTP 1.1 and HTTP2 have quite the same high level syntax (methods, uri, headers, ...) but the way the data is transported differs. (data are sent by frames in HTTP 2). - in HTTP 2, data can be encrypted and compressed. - in HTTP2 data can pushed from the server to the clients. More data can be sent to the client - in HTTP2 streams are multiplexed We have the concept of data channels and these are more like message passings. Multiplexing existed in HTTP 1.1 with pipelines but is barely supported right now by WSGI servers. The concept of data channels and the PUSH features will requires more concurrency at the server level. At the application, things doesn't change that much. Everything can appear like before. The only change is the PUSH feature. 2) Websockets, SSE and other similar protocoles are completely asynchronous. All this part is not really handled by WSGI. The way it is generally implemented right now is awkward. The server generally extend the WSGI protocol so the application get the socket. Then a specific library handle the rest. I actually wonder if websockets or other asynchronous protocols should be handled by the new WSGI SPEC. Shouldn't we just standardize the way the socket is given to another library? Anyway I think we should collect all requirements at application and server level and then start to confront the current WSGI spec to them. And iterate. Thoughts? Any other topic? - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI for HTTP/2.0 ?
got an idea. What about having a page collecting feedback from anyone in the python community about this topic. So we can have true data from different perspectives: developer, library/framework author, server author. I'm OK to collect the data from it and make a summary of it once it's done. The form it could take should be discussed first but imo that a good way to engage the community. What do you think? On Sat, Sep 20, 2014 at 4:17 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Sat, Sep 20, 2014 at 3:22 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Sat, Sep 20, 2014 at 9:23 AM, Robert Collins robe...@robertcollins.net wrote: Well, thats certainly a challenge :). Whats the governance model here? Is a PEP appropriate, and if so - that gives us a BFDL or BFDL PEP-delegate to decide between bikeshed issues; and if its not a bikeshed issue then resolving it is actually necessary. Yes, I think a good way forward would be to have a small cabal write a PEP and then announce it here for further feedback and then pronouncement by a BDFL or -delegate. If you want to be lead/editor, that sounds great. It also seems like you should definitely involve Graham and give credence to his thoughts. I'd be excited about this and happy to give feedback a little later once you've got some initial draft, as someone who likes to implement his applications directly on top of WSGI for now (but I've also implemented a couple of WebSocket servers). Cheers, Dirkjan Last time was more about everyone wanted to discuss about the changes with its own requirement list. Which is already conflicting. Discussing if it should be done outside the media designed for it is already out of topic imo. So I won't discuss about it further. Instead I wonder what is the appropriate medium to collect requirements and others stuffs about it. Wiki ? Anything else? For a start I see these different topics 1) HTTP 1.1 vs HTTP 2: - HTTP 1.1 and HTTP2 have quite the same high level syntax (methods, uri, headers, ...) but the way the data is transported differs. (data are sent by frames in HTTP 2). - in HTTP 2, data can be encrypted and compressed. - in HTTP2 data can pushed from the server to the clients. More data can be sent to the client - in HTTP2 streams are multiplexed We have the concept of data channels and these are more like message passings. Multiplexing existed in HTTP 1.1 with pipelines but is barely supported right now by WSGI servers. The concept of data channels and the PUSH features will requires more concurrency at the server level. At the application, things doesn't change that much. Everything can appear like before. The only change is the PUSH feature. 2) Websockets, SSE and other similar protocoles are completely asynchronous. All this part is not really handled by WSGI. The way it is generally implemented right now is awkward. The server generally extend the WSGI protocol so the application get the socket. Then a specific library handle the rest. I actually wonder if websockets or other asynchronous protocols should be handled by the new WSGI SPEC. Shouldn't we just standardize the way the socket is given to another library? Anyway I think we should collect all requirements at application and server level and then start to confront the current WSGI spec to them. And iterate. Thoughts? Any other topic? - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] question about connection pool, task queue in WSGI
On Sun, Jul 15, 2012 at 5:14 PM, Simon Sapin simon.sa...@exyr.org wrote: Le 14/07/2012 06:07, Graham Dumpleton a écrit : 2. Is the socket FD the same mechanism like nginx? If you upgrade nginx binary, restart nginx, the existing http connection won't break. I would be very surprised if you could upgrade nginx, perform a restart and preserve the HTTP listener socket. If you are talking about some other socket I don't know what you are talking about. As you can with Apache, you can likely enact a configuration file change and perform a restart or trigger rereading of the configuration and it would maintain the HTTP listener socket across the configuration restart, but an upgrade implies changing the binary and I know no way that you could easily persist a HTTP listener socket across to an invocation of a new web server instance using a new executable. In Apache you certainly cannot do it, and unless nginx has some magic where the existing nginx execs the new nginx version and somehow communicates through open socket connections to the new process, I very much doubt it would as it would be rather messy to do so. I think that est refers to this: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly Apparently yes, there is specific code in nginx to start the new binary and give it the existing socket. And I think that yes, Tarek’s new Circus is similar to the nginx magic upgrade in that an open socket is passed around processes. Maybe nginx even does this in normal operation with multiple worker processes, but I don’t know. Regards, -- Simon Sapin Gunicorn does upgrade itself using the USR2 signal just like nginx and share the socket like using an fd between OS processes. However the case of a db is a little different since you handle a connection to a db and not listening on a port. You will need either a multiprocess queue passing messages to one process or launching a connection per processes. You can do that using the hook system of gunicorn. - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Starting Web Servers using socket FDs
On Tue, Jun 5, 2012 at 12:26 PM, Tarek Ziadé ta...@ziade.org wrote: On 6/5/12 11:46 AM, Roberto De Ioris wrote: ... Gunicorn can already bind (or better, accept) from file descriptors specifying an environment variable. I don't think you can start gunicorn using a file descriptor, or I failed to do it. The best I was able to do was to create a small wsgi server using Gunicorn as a lib. Gunicorn uses an environment variable when it respawns workers but it does not offer it as a public option as far as I understand how it works export GUNICORN_FD=your fd and then gunicorn will use this file descriptor when it starts. But it should be possible to pass it directly using a config option if it's needed. - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] A 'shutdown' function in WSGI
On Mon, Feb 20, 2012 at 5:03 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: Hello I need to be able to call a function when the web application shuts down (SIGTERM/SIGINT) -- the use case is to stop a background thread. I am currently using signals because it seems to be the most clean way to do this. atexit is much trickier since you don't know when it's going to get called and you might try to call objects that were garbage collected unless you hack something to keep references alive. But signals are also tricky beasts since you may compete with other code that are listening to them. For instance mod_wsgi don't like apps that have signal handlers. Anyways, the bottom line is that the cleanest way to do this -- as per Chris McDonough idea, would be to introduce in the WSGI protocol a shutdown function the servers would be obligated to call before exiting. I am not sure yet about its arguments, maybe a signum + frame or simply an exit code... But how do you like the idea ? That would solve for me the problem of having to deal differently here depending on if I am called with mod_wsgi or gunicorn or xxx Hi Tarek, In gunicorn we have the concept of hook which exactly does what you want (on_exit hook). You can do this via the configuration file. And you can do this for the other steps (reload, pre/post fork, pre/post request ...). Anyway, I think there are 2 issues to solve with this shutdown thing (and more) in a generic way: 1. how to make your application aware of the server capabilities (can it or not handle shutdown) 2. how to pass such functions to the server. There are also some others details like determining what is the context needed by the application at this step. Today the server is acting as a gateway so it just pass a request to an application and return the response from the application. The application by itself only lives in this context. Maybe, like some said, if we want to give more info about that, we could imagine the application object or function as a bus reacting on different contexts: def application(environ, start_application): pass where the environ can be different depending on the context an action, we could have: http request{wsgi.context: http, ...} shutdown{wsgi.context; server, action: shutdown, ...} smth like it. But how to know when the server can't pass the action shutdown? Also what would be the environ at shutdown? One other method, like the one described by Sylvain Hellegouarch, would be adding some properties/attributes to the application object. So the server would eventually check if they exists and eventually call them. But again how to know when the server can't pass the action shutdown? Though, I don't like to handle more than the application callable today, I can see some problems when you have to handle server reload and such things. It seems important for me that the application knows about the server capabilities if it want to handle such things like the `shutdown`. And this step is solved by gunicorn others by specifically handling a configuration. But maybe we could just considers that a server always handle the reload, shutdown, also pre/post request evenst and have optionnals message passed to the application? (how do we handled process context changes - if we re in a thread for ex - ?). We could also imagine a web application package that provides hooks handling depending on the server target... I quite like this idea. Not sure what is the right way. What do you think? - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] A 'shutdown' function in WSGI
On Tue, Feb 21, 2012 at 10:35 AM, Sylvain Hellegouarch s...@defuze.org wrote: On Tue, Feb 21, 2012 at 10:31 AM, Graham Dumpleton graham.dumple...@gmail.com wrote: On 21 February 2012 20:26, Simon Sapin simon.sa...@exyr.org wrote: Le 21/02/2012 09:23, Tarek Ziadé a écrit : Instead of having to provide two or three objects separately to a server, how about making the callbacks attributes of the application callable? can you show us an example ? Proposal: Function-based: def startup(): return open_resource(something) def shutdown(resource): resource.close() def application(environ, start_response): # ... return response_body application.startup = startup application.shutdown = shutdown Class-based: class App(object): def startup(self): return open_resource(something) def shutdown(self, resource): resource.close() def __call__(self, environ, start_response): # ... return response_body application = App() The return value of startup() can be any python object and is opaque to the server. It is passed as-is to shutdown() startup() could take more parameters. Maybe the application (though can we already have it as self for class-based or in a closure for function-based) You do realise you are just reinventing context managers? With this 'application' do requests. But then it was sort of suggested that was a bit too radical idea when I have mentioned viewing it that way before. :-( One might wonder if having access to process management should be part of WSGI in the first place. that's the thing. This is no more a gateway. Like I said in my previous post, maybe having another spec describing a web app package would do the trick? - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
On Tue, Jan 4, 2011 at 12:13 AM, Jacob Kaplan-Moss ja...@jacobian.org wrote: On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum gu...@python.org wrote: Although [PEP ] is still marked as draft, I personally think of it as accepted; [...] What does it take to get PEP formally marked as accepted? Is there anything I can do to push that process forward? The lack of a WSGI answer on Py3 is the main thing that's keeping me, personally, from feeling excited about the platform. Once that's done I can feel comfortable coding to it -- and browbeating those who don't support it. I understand that PEP 444/Web3/WSGI 2/whatever might be a better answer, but it's clearly got some way to go. In the meantime, what's next to get PEP officially endorsed and accepted? Jacob I we pep is considered accepted, I will start the port of gunicorn based on it too. It shouldn't be hard since we are using our own http parser. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Most WSGI servers close connections to early.
On Wed, Sep 22, 2010 at 2:46 PM, Marcel Hellkamp m...@gsites.de wrote: I just discovered a problem that affects most WSGI server implementations and most current web-browsers (tested with wsgiref, paste, firefox, chrome, wget and curl): If the server closes the connection while the client is still uploading data via POST or PUT, the browser displays an error message ('Connection closed') and does not display the response sent by the server. The error occurs if an application chooses to not process a form submissions before returning to the WSGI server. This is quite rare in real world scenarios, but hard to debug because the server logs the request as successfully sent to the client. To reproduce the problem, run the following script, visit http://localhost:8080/ and upload a big file:: from wsgiref.simple_server import make_server def application(environ, start_response): start_response('200 OK', [('Content-Type', 'text/html')]) return [ form method='post' enctype='multipart/form-data' Upload bog file: input type='file' name='file' / input type='submit' / /form ] server = make_server('localhost', 8080, application) server.serve_forever() I would like to add a warning to the WSGI/web3 specification to address this issue: An application should read all available data from `environ['wsgi.input']` on POST or PUT requests, even if it does not process that data. Otherwise, the client might fail to complete the request and not display the response. -- Mit freundlichen Grüßen Marcel Hellkamp Your application and client should be aware of Expect: 100-Continue header : http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html - benoît (resent, because web-sig doesn't set well the default reply-to) ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] Most WSGI servers close connections to early.
On Wed, Sep 22, 2010 at 5:34 PM, Robert Brewer fuman...@aminus.org wrote: However, the caveat requires a caveat: servers must still be able to protect themselves from malicious clients. In practice, that means allowing servers to close the connection without reading the entire request body if a certain number of bytes is exceeded. I don't see how it could be the responsability of the server. Can you develop a little ? The server shouldn't interfere in the HTTP request imo. - benpît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 (aka Web3)
On Fri, Sep 17, 2010 at 10:36 AM, Georg Brandl g.bra...@gmx.net wrote: Am 16.09.2010 23:07, schrieb James Mills: - the web3 name If there is any value in this PEP and we find something to decide on, there is no reason this couldn't be WSGI 2. But until it's just something a small part of the web-sig community worked on directly a separate name is a good thing I think, because it does not reserve the name WSGI 2 for something that might actually become WSGI 2 in case this PEP gets rejected. I personally still don't see any real benefit to changing the key names from wsgi to web3 (or whatever). I would prefer it remain the same. If you're going to use Python3, you know you're using Python3 (you don't need web3 key names to know that). (subjective) That statement shows another weakness of the web3 name: this spec is not in the least exclusive to Python 3. (Which would be a bit useless, having two incompatible WSGI/web specs on two incompatible Python versions.) The goal would be to first migrate to WSGI2/web3, and *then* have an easy transition going to Python 3. Georg also WSGI acronym is defining better the purpose by itself than web3 which mean nothing. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] PEP 444 (aka Web3)
On Thu, Sep 16, 2010 at 6:41 PM, Armin Ronacher armin.ronac...@active-4.com wrote: 4. The web3 spec says, In case a content length header is absent the stream must not return anything on read. It must never request more data than specified from the client. but later it says, Web3 servers must handle any supported inbound hop-by-hop headers on their own, such as by decoding any inbound Transfer-Encoding, including chunked encoding if applicable.. I would be sad if web3 did not support streaming uploads via Transfer-Encoding. One way to implement that would be to make the origin server handle read() transparently by returning '' on EOF, regardless of whether a Content-Length or a Transfer-Encoding header was provided. I was toying with the idea to have a websocket extension for web3 which would have solved my usecase for requests without a content-length header. The problem with the content length of incoming data is quite complex and that seemed to be the solution that was easiest for everybody involved. uh ? Since with Transfer-Encoding: chunked we know when the stream end, I would be in favor of returning an EOF too at the end. Also most of servers know when a stream end even if there is no content-length. Maybe we could have a capability setting in environ that say if the server support streaming or not. And in all cases returning EOF at the end? - benoît ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
[Web-SIG] process ..
Hi all, As external I'm a bit surprised and also disappointed about actual process about choosing/discussing next features or changes of SGI spec. I thought and I think process should be more formal. Like I see it since some months, there are people trying to put their views and technical considerations but that's all. I don't see any vote about adding/editing current version of the spec. Some people try to summarize, but there is also not formal way. Also working on different repositories for one spec, with changes without votes make me think something is wrong. I don't see any step in discussion nor roadmap and a standard shouldn't be handled like this in my opinion. Maybe I'm wrong, maybe votes are hidden, there is something outside I ignore and I will be happy to be wrong on this. Anyway I really think in this case process should be more transparent. At least having a roadmap with formal steps like you have on other standards would help. The web is changing, and as a developer I would like to know what and when will be adapted WSGI to new needs. - benoit ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com