Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

I am not speaking about websockets. You could use it for SSE, or some apps
could use the Upgrade header to upgrade from http to their own protocol
etc... The only discussion i saw about websockets are about the addition of
an async api or an external api. I am not describing that. I am speaking
about providing a low level abstraction like wsgi.input but adding to it
the support of output. (I was referring to wsgi.multithread...). This low
level interface would allow anyone to provide its own
implementation(server) or usage (application) still acting as a *gateway* .

Also who are "we"? I am starting to think the discussion is already done
and only obscure details like the content_length or headers encoding should
be discussed. The RAW_SOCKET have been added on demand of the gunicorn
users. Such thing also exist in things like cherrypy if I remember. A lot
of code around have been created over it. So before deciding it's
unworkable or whatever I strongly invite you to consider it as an addition
to the environ. And since some servers need to pass the data differently I
then suggest a Resource object on which you can read and write and
eventually poll. This is not a websocket but more a proxy ressource to the
client connexion. I will come back asap with a small spec.

I also propose a second addition to the protocol that formalize the
addition of extensions to the protocol by the servers if they want to.
Having for example something like "`wsg.extensions` . Such addition would
help anyone to experiment changes over the wsgi before making such changes
in the specification by itself possibly.

I think we have a good opportunity to extend the WSGI specification to
allow the users to take over the new challenges on the web without forcing
them to use a concurrency mode or skip completely the WSGI spec. The
interest I see in WSGI is its simplicity and low level interface allowing
users to build whatever they want over it. The different workers and their
support of different concurrency models and framework in gunicorn let me
think it is possible. Are the participants of this thread ready to discuss
it?

- benoît

On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> again. any server can do such implementation if we create a new Resource
> abstraction. This abstraction would expose a common api to read and write.
> The implementation would be specific to the server.
>
>
> If you mean not exposing the raw socket and having a separate high level
> API for implementing something like WebSocket this was already talked
> about. The suggestion was that it should not be a part of WSGI. Develop
> that API independently with no link to WSGI. The idea of upgrading from
> WSGI to a different API isn’t practical for various WSGI servers as it
> isn’t possible to unwind the state of the connection path created to get to
> point of handling the WSGI application. The better scenario is that the
> switch to an alternate WebSocket API is handled completely within the web
> server however it needs to handle it, when it needs to handle it, and not
> be reliant on going into a WSGI application which then says, oh, I actually
> need that to be WebSocket.
>
> Now like we have wsgi.thread I would instead suggest to add a system of
> capability or extension like in smtp, imap, ... so the servers that
> implement a specific extension can legally published it. Would it work for
> you?
>
>
> Since there is nothing in WSGI environ called wsgi.thread now I have no
> idea what you are really suggesting here.
>
> Graham
>
> benoit
>
> On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:
>
>>
>> On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>>
>>
>>
>> On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net>
>> wrote:
>>
>>> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com>
>>> wrote:
>>>
>>> >
>>> > not at all. But I made the assumption that the wsgi server maintained a
>>> > thread directly or not where the python application is running .
>>> >
>>> > In any case there is some sort of wrapping done in the same
>>> thread/process
>>> > where the python application is running. And then nothing stop to give
>>> the
>>> > socket away to the application and tell to the server to stop to
>>> communicate
>>> > with it.
>>>
>>> What socket?
>>>
>>> Data could be being passed by shm, for instance.
>>>
>>> -Rob
>>>
>>>
>> Whil

[Web-SIG] Fwd: Collating follow-up on the future of WSGI

[Benoit Chesneau]

-- Forwarded message -
From: Benoit Chesneau <bchesn...@gmail.com>
Date: Thu, 21 Jan 2016 at 06:12
Subject: Re: [Web-SIG] Collating follow-up on the future of WSGI
To: Graham Dumpleton <graham.dumple...@gmail.com>


I am not speaking about websockets. You could use it for SSE, or some apps
could use the Upgrade header to upgrade from http to their own protocol
etc... The only discussion i saw about websockets are about the addition of
an async api or an external api. I am not describing that. I am speaking
about providing a low level abstraction like wsgi.input but adding to it
the support of output. (I was referring to wsgi.multithread...). This low
level interface would allow anyone to provide its own
implementation(server) or usage (application) still acting as a *gateway* .

Also who are "we"? I am starting to think the discussion is already done
and only obscure details like the content_length or headers encoding should
be discussed. The RAW_SOCKET have been added on demand of the gunicorn
users. Such thing also exist in things like cherrypy if I remember. A lot
of code around have been created over it. So before deciding it's
unworkable or whatever I strongly invite you to consider it as an addition
to the environ. And since some servers need to pass the data differently I
then suggest a Resource object on which you can read and write and
eventually poll. This is not a websocket but more a proxy ressource to the
client connexion. I will come back asap with a small spec.

I also propose a second addition to the protocol that formalize the
addition of extensions to the protocol by the servers if they want to.
Having for example something like "`wsg.extensions` . Such addition would
help anyone to experiment changes over the wsgi before making such changes
in the specification by itself possibly.

I think we have a good opportunity to extend the WSGI specification to
allow the users to take over the new challenges on the web without forcing
them to use a concurrency mode or skip completely the WSGI spec. The
interest I see in WSGI is its simplicity and low level interface allowing
users to build whatever they want over it. The different workers and their
support of different concurrency models and framework in gunicorn let me
think it is possible. Are the participants of this thread ready to discuss
it?

- benoît


On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> again. any server can do such implementation if we create a new Resource
> abstraction. This abstraction would expose a common api to read and write.
> The implementation would be specific to the server.
>
>
> If you mean not exposing the raw socket and having a separate high level
> API for implementing something like WebSocket this was already talked
> about. The suggestion was that it should not be a part of WSGI. Develop
> that API independently with no link to WSGI. The idea of upgrading from
> WSGI to a different API isn’t practical for various WSGI servers as it
> isn’t possible to unwind the state of the connection path created to get to
> point of handling the WSGI application. The better scenario is that the
> switch to an alternate WebSocket API is handled completely within the web
> server however it needs to handle it, when it needs to handle it, and not
> be reliant on going into a WSGI application which then says, oh, I actually
> need that to be WebSocket.
>
> Now like we have wsgi.thread I would instead suggest to add a system of
> capability or extension like in smtp, imap, ... so the servers that
> implement a specific extension can legally published it. Would it work for
> you?
>
>
> Since there is nothing in WSGI environ called wsgi.thread now I have no
> idea what you are really suggesting here.
>
> Graham
>
> benoit
>
> On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:
>
>>
>> On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>>
>>
>>
>> On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net>
>> wrote:
>>
>>> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com>
>>> wrote:
>>>
>>> >
>>> > not at all. But I made the assumption that the wsgi server maintained a
>>> > thread directly or not where the python application is running .
>>> >
>>> > In any case there is some sort of wrapping done in the same
>>> thread/process
>>> > where the python application is running. And then nothing stop to give
>>> the
>>> > socket away to the application and tell to the serve

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

because i am not speaking about making a specification, but a way to expose
in the API (environ) custom extensions that a server want to experiment.
there are actually no easy way except checking "wsgi." indeed but that
doesn't make it as clear as a separate namespace where to put all server
extensions could be. Like capability field is in imap world.

Also I am not trying to force anything, I want to discuss about a possible
update of the wsgi spec which I thought was this thread about. What I just
want to discuss is the *current* usage of some extensions that have been
rapidly dismissed as unworkable. Like I said I will come with a more formal
specification about them but I wanted to discuss first about and collect
counter arguments which are good too.

- benoît


On Thu, 21 Jan 2016 at 07:24, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> I am still confused as to why you keep talking as if you seemingly are
> trying to force extensions to the existing WSGI specification into the core
> WSGI specification when an alternative has already been cited.
>
> Extensions and the process for describing them and getting them accepted,
> plus the appropriate WSGI environ prefix, as has been mentioned before, is
> what is covered by:
>
> http://wsgi.readthedocs.org/en/latest/specifications.html
>
> Being an extension means it is entirely optional for a WSGI server to try
> and implement it, thus allowing WSGI servers/adapters that cannot implement
> something to skip them. They stand as separate documents and would never
> become part of the core WGSI PEP.
>
> Is there an issue with doing extensions per the process, and in the WSGI
> environ namespace, that was outlined in that URL? You seem to be suggesting
> a completely new way of handling extensions and ignoring what was laid down
> before.
>
> So no one is saying you can’t have extensions, and that separate process
> gives you all the scope you need to do it.
>
> In drafting your specification just fit it reference to what is described
> in that URL, using ‘x-wsgiorg.’ prefix keys.
>
> Graham
>
>
> On 21 Jan 2016, at 4:13 PM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> I am not speaking about websockets. You could use it for SSE, or some apps
> could use the Upgrade header to upgrade from http to their own protocol
> etc... The only discussion i saw about websockets are about the addition of
> an async api or an external api. I am not describing that. I am speaking
> about providing a low level abstraction like wsgi.input but adding to it
> the support of output. (I was referring to wsgi.multithread...). This low
> level interface would allow anyone to provide its own
> implementation(server) or usage (application) still acting as a *gateway* .
>
> Also who are "we"? I am starting to think the discussion is already done
> and only obscure details like the content_length or headers encoding should
> be discussed. The RAW_SOCKET have been added on demand of the gunicorn
> users. Such thing also exist in things like cherrypy if I remember. A lot
> of code around have been created over it. So before deciding it's
> unworkable or whatever I strongly invite you to consider it as an addition
> to the environ. And since some servers need to pass the data differently I
> then suggest a Resource object on which you can read and write and
> eventually poll. This is not a websocket but more a proxy ressource to the
> client connexion. I will come back asap with a small spec.
>
> I also propose a second addition to the protocol that formalize the
> addition of extensions to the protocol by the servers if they want to.
> Having for example something like "`wsg.extensions` . Such addition would
> help anyone to experiment changes over the wsgi before making such changes
> in the specification by itself possibly.
>
> I think we have a good opportunity to extend the WSGI specification to
> allow the users to take over the new challenges on the web without forcing
> them to use a concurrency mode or skip completely the WSGI spec. The
> interest I see in WSGI is its simplicity and low level interface allowing
> users to build whatever they want over it. The different workers and their
> support of different concurrency models and framework in gunicorn let me
> think it is possible. Are the participants of this thread ready to discuss
> it?
>
> - benoît
>
> On Wed, 20 Jan 2016 at 23:37, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:
>
>> On 21 Jan 2016, at 9:27 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>>
>> again. any server can do such implementation if we create a new Resource
>> abstraction. This abstraction would expose a common api to read and write

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net>
wrote:

> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> >
> > not at all. But I made the assumption that the wsgi server maintained a
> > thread directly or not where the python application is running .
> >
> > In any case there is some sort of wrapping done in the same
> thread/process
> > where the python application is running. And then nothing stop to give
> the
> > socket away to the application and tell to the server to stop to
> communicate
> > with it.
>
> What socket?
>
> Data could be being passed by shm, for instance.
>
> -Rob
>
>
While shared memory would be quite a bad idea, then why not. I still don't
see why having a way to upgrade the connection can't be done.

Call it I/O resource or Socket, the issue is the same. At the end nothing
stop the server to pass the control to the app. If we forget the socket
(which is btw the simplest design) then the server could stop to control
the I/O resource when the application ask it to do it. At some point either
a garbage collection or a basic resource return/claim flow could be used to
definitely free the resource.

The thing behind that is that it would allow the WSGI spec to only focus on
providing a strict gateway workflow without forcing the application to
adopt a concurrency model aync or not.

- benoit.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

again. any server can do such implementation if we create a new Resource
abstraction. This abstraction would expose a common api to read and write.
The implementation would be specific to the server.

Now like we have wsgi.thread I would instead suggest to add a system of
capability or extension like in smtp, imap, ... so the servers that
implement a specific extension can legally published it. Would it work for
you?

benoit

On Wed, 20 Jan 2016 at 21:28, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

>
> On 21 Jan 2016, at 2:48 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
>
>
> On Wed, Jan 20, 2016 at 1:57 AM Robert Collins <robe...@robertcollins.net>
> wrote:
>
>> On 20 January 2016 at 12:04, Benoit Chesneau <bchesn...@gmail.com> wrote:
>>
>> >
>> > not at all. But I made the assumption that the wsgi server maintained a
>> > thread directly or not where the python application is running .
>> >
>> > In any case there is some sort of wrapping done in the same
>> thread/process
>> > where the python application is running. And then nothing stop to give
>> the
>> > socket away to the application and tell to the server to stop to
>> communicate
>> > with it.
>>
>> What socket?
>>
>> Data could be being passed by shm, for instance.
>>
>> -Rob
>>
>>
> While shared memory would be quite a bad idea, then why not. I still don't
> see why having a way to upgrade the connection can't be done.
>
> Call it I/O resource or Socket, the issue is the same. At the end nothing
> stop the server to pass the control to the app. If we forget the socket
> (which is btw the simplest design) then the server could stop to control
> the I/O resource when the application ask it to do it. At some point either
> a garbage collection or a basic resource return/claim flow could be used to
> definitely free the resource.
>
> The thing behind that is that it would allow the WSGI spec to only focus
> on providing a strict gateway workflow without forcing the application to
> adopt a concurrency model aync or not.
>
>
> No one has said you cannot do it. because though it is only able to be
> implemented in a subset of WSGI servers/adapters, then it doesn’t seem
> appropriate that it be a part of the core WSGI specification.
>
> This is the role of a WSGI extension as found at:
>
> http://wsgi.readthedocs.org/en/latest/specifications.html
>
> So go talk to the authors of uWSGI, and the other couple of packages
> available for trying to plug these into some of the pure Python based WSGI
> servers and come to an agreement between yourselves as to a standard way of
> doing it and the extension specification can be added to the wsgi.org
>  site.
>
> Graham
>
>
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

On Tue, Jan 19, 2016 at 11:58 PM Robert Collins 
wrote:

> On 20 January 2016 at 05:55, Cory Benfield  wrote:
> > All,
> >
> > Thanks so much for your feedback to my original request for comments on
> the future of WSGI. You provided a ton of really useful feedback: when
> printed out on my printer it ended up at about 50 pages of information that
> was really engaging reading. I also want to thank you all for keeping the
> tone of the discussion so positive. On an topic like this one it can get
> tricky and emotionally charged very easily, and you did a great job of
> avoiding that problem.
> >
> > I spent a few hours this morning going over your feedback and trying to
> extract some common threads. Altogether I believe that most participants
> were mostly in agreement over the direction we should take, with a few
> outliers in each case. I’d like to summarise what I believe were the big
> take-aways from that discussion to confirm that I’ve understood everyone.
> >
> > I’d also like the members of this SIG to take this opportunity to
> discuss these proposals more concretely. Rather than expressing our
> sentiments about WSGI and its future more generally, I want people to
> critique and offer opinions on *these specific proposals*. The goal here is
> to get an understanding of whether these are worth doing, how we need to
> prioritise the work, and whether there’s anything we’re missing.
> >
> > The below is formatted in restructured text for clarity, because it’s
> quite a lot of information.
> >
> >
> > Concrete Proposals
> > ==
> >
> > WSGI
> > 
> >
> > Overall, there was strong and fairly unanimous sentiment that WSGI
> itself should more or less be left alone. Minor adjustments would be
> valuable, and we should pursue them, but the contributors do not believe
> that making a substantial revision to WSGI would be adviseable.
> >
> > I therefore propose we revise the WSGI specification to WSGI 1.1 and
> then consider it "final". The following points were raised for revising
> WSGI.
>
> Sure.
>
> > Asynchronous WSGI
> > ~
> >
> > This was generally regarded as too substantial a change to shoehorn into
> WSGI.
> >
> > Benoit proposed that we could achieve this change by adding a correlator
> to WSGI. This would allow servers to associate a given call to ``write()``
> with a specific request/response (most useful in HTTP/2 where this could
> correspond to a stream ID). This would then allow WSGI to transition to a
> purely callback-based model that could in principle cohabitate with an
> async protocol in Python.
>
> Uh, WSGI doesn't need to be involved in this. Servers can already pass
> arbitrary objects as write, permitting any correlation they want.  I'd
> want to see something very specific to consider the impact in terms of
> WSGI, or WSGI-NG or whatever we do in future.
>
> > This proposal is worth highlighting not becuase I think we should pursue
> it with our revision of WSGI, but because it's worth considering for any
> future specification we come up with. Note also that Graham pointed out
> that this would require some careful rethinking of reads from
> ``wsgi.input``.
>
>
>
> > Server Push
> > ~~~
> >
> > We can support HTTP/2 server push using Link headers. This could
> optionally be supplemented by defining a WSGI extension that provides a
> callable for doing server push, which would be a mild improvement over the
> Link header approach. A simple proposal for how to do this can easily be
> implemented without revising the WSGI specification, and may not even need
> to be enshrined in a PEP. However, if we revise the WSGI PEP we may want to
> provide a small section that indicates how to add these headers.
>
> Since the point of WSGI is interop, we should document the expected
> interop fashion for this, whatever it is.
>
> > Socket Escape Hatch
> > ~~~
> >
> > Aside from Benoit, server operators were unanimously dismissive of the
> idea of a socket 'escape hatch'. In general it seems like servers would not
> be capable of achieving this. I think, therefore, this idea is unworkable.
>
> I thin Benoit is making assumptions about what a 'server' is that are
> not implied by WSGI - and this is leading to some confusion. Servers
> are not necessarily Python processes :).
>
>
not at all. But I made the assumption that the *wsgi* server maintained a
thread directly or not where the python application is running .

In any case there is some sort of wrapping done in the same thread/process
where the python application is running. And then nothing stop to give the
socket away to the application and tell to the server to stop to
communicate with it.


- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

On Tue, Jan 19, 2016 at 11:34 PM Graham Dumpleton <
graham.dumple...@gmail.com> wrote:

>
> On 20 Jan 2016, at 8:29 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
>
>
> On Tue, Jan 19, 2016 at 10:49 PM Graham Dumpleton <
> graham.dumple...@gmail.com> wrote:
>
>>
>> On 20 Jan 2016, at 7:43 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>>
>> I will make a more complete answer soon. But about:
>>
>>
>>>
>>> Socket Escape Hatch
>>> ~~~
>>>
>>> Aside from Benoit, server operators were unanimously dismissive of the
>>> idea of a socket 'escape hatch'. In general it seems like servers would not
>>> be capable of achieving this. I think, therefore, this idea is unworkable.
>>>
>>>
>> Well it does work. This is how websockets works in gunicorn.  Escape is
>> not the right term anyway. Think it as a socket *upgrade*. And then you
>> would wonder why it would be unworkable. After all this is how SSL sockets
>> works, so is the protocol negotiation in http2 ...
>>
>> There is nothing magic there until you try to over engineer the stuff.
>> Upgrading a sockets means that you tell to the server to forget it. This is
>> how most concurrent servers work today.
>>
>>
>> The problem was that it would only work in a WSGI server where the
>> original request was accepted on a socket in the same process as the WSGI
>> application is running. It cannot work where where the WSGI application is
>> behind a bridging protocol.
>>
>
>> So it can’t work for CGI, SCGI, FASTCGI, mod_wsgi daemon mode and
>> possibly other implementations.
>>
>
> uh? But we don't care about bridging protocols. In WSGI (the gateway), the
> server accept the socket and anyway pass it to the application via actually
> a wrapper. Then expect a response from the application.
>
> Upgrading a socket would simply mean  that the server will forget it (and
> then consider its job done) once it got an appropriate response from the
> application. How this is unworkable?
>
>
> Bridging protocols such as FASTCGI do not provide an ability to upgrade
> the connection end to end.
>
> That is, yes you could pass the raw socket to the WSGI application when
> behind FASTCGI, but you are passing it a socket from same process where
> data being received (and expected to be sent), is using FASTGCI message
> frames. It is not a raw HTTP socket connection.
>
> There is no way to send a message back to the front end side of the
> bridged connection where the raw HTTP socket is, to tell the client side of
> the FASTCGI implementation to stop treating it as a FASTCGI connection to
> backend process and then suddenly start acting as a raw socket pass through.
>
>
But how it is related to the WSGI protocol?

It is expected that the wsgi server accept on a socket (directly or behind
a proxy) and provide enough to the application so they can read and write.

Upgrading or escaping in WSGI/Server sense would mean that it would skip
the dialog with the application once the application told it to do.

For the rest, the application would do what it want on the socket, even
giving it to another process/thread.

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

On Tue, Jan 19, 2016 at 10:49 PM Graham Dumpleton <
graham.dumple...@gmail.com> wrote:

>
> On 20 Jan 2016, at 7:43 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> I will make a more complete answer soon. But about:
>
>
>>
>> Socket Escape Hatch
>> ~~~
>>
>> Aside from Benoit, server operators were unanimously dismissive of the
>> idea of a socket 'escape hatch'. In general it seems like servers would not
>> be capable of achieving this. I think, therefore, this idea is unworkable.
>>
>>
> Well it does work. This is how websockets works in gunicorn.  Escape is
> not the right term anyway. Think it as a socket *upgrade*. And then you
> would wonder why it would be unworkable. After all this is how SSL sockets
> works, so is the protocol negotiation in http2 ...
>
> There is nothing magic there until you try to over engineer the stuff.
> Upgrading a sockets means that you tell to the server to forget it. This is
> how most concurrent servers work today.
>
>
> The problem was that it would only work in a WSGI server where the
> original request was accepted on a socket in the same process as the WSGI
> application is running. It cannot work where where the WSGI application is
> behind a bridging protocol.
>

> So it can’t work for CGI, SCGI, FASTCGI, mod_wsgi daemon mode and possibly
> other implementations.
>

uh? But we don't care about bridging protocols. In WSGI (the gateway), the
server accept the socket and anyway pass it to the application via actually
a wrapper. Then expect a response from the application.

Upgrading a socket would simply mean  that the server will forget it (and
then consider its job done) once it got an appropriate response from the
application. How this is unworkable?

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Collating follow-up on the future of WSGI

[Benoit Chesneau]

I will make a more complete answer soon. But about:


>
> Socket Escape Hatch
> ~~~
>
> Aside from Benoit, server operators were unanimously dismissive of the
> idea of a socket 'escape hatch'. In general it seems like servers would not
> be capable of achieving this. I think, therefore, this idea is unworkable.
>
>
Well it does work. This is how websockets works in gunicorn.  Escape is not
the right term anyway. Think it as a socket *upgrade*. And then you would
wonder why it would be unworkable. After all this is how SSL sockets works,
so is the protocol negotiation in http2 ...

There is nothing magic there until you try to over engineer the stuff.
Upgrading a sockets means that you tell to the server to forget it. This is
how most concurrent servers work today.

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Benoit Chesneau]

On Wed, Jan 6, 2016 at 10:19 AM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello Benoît,
>
> Thanks for clarifying that you also had the reverse problem in mind,
> headers sent by applications. This side is less problematic in the sense
> that application authors can adapt to stronger requirements.
>
> In general this is a bit of a mess due to differences between what the RFC
> 2616 says and what browsers do in practice. That’s why I believe the
> pragmatic solution is to exchange bytes. (This isn’t a major issue in the
> grand scheme of things anyway.)
>
> Best regards,
>
> --
> Aymeric.
>
>
Since sockets are only accepting bytes in python 3.x,  having everything as
bytes would also allow the server to process the response more easily...


- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Benoit Chesneau]

On Wed, Jan 6, 2016 at 10:29 AM Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

>
> On 6 Jan 2016, at 12:13 AM, Benoit Chesneau <bchesn...@gmail.com> wrote:
>
> So for me what should be WSGI 2? WSGI 2 should add against WSGI 1 the
> following:
>
> - tell to the application it is actually an HTTP2 request (maybe
> populating a wsgi.http2 true env)
>
>
> In CGI implementations you would for HTTP/1.1 already get:
>
> SERVER_PROTOCOL: 'HTTP/1.1’
>
> Under HTTP/2 when I tested some time back, I recollect it came through as
> one would assume would be expected:
>
> SERVER_PROTOCOL: ‘HTTP/2’
>
> Is there any reason that this existing CGI variable wouldn’t be sufficient
> for this purpose?
>
> Graham
>


True. I don't think there is a need for another env. The current way also
work for the future :)

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Draft 2: WSGI Response Upgrade Bridging

[Benoit Chesneau]

On Mon, Oct 13, 2014 at 11:52 AM, Robert Collins robe...@robertcollins.net
wrote:

 On 13 October 2014 17:12, Benoit Chesneau bchesn...@gmail.com wrote:
 
 ...
 
 
  OK,
 
  So I should probably know you, but I can't recollect right now what you
 do
  or write.

 Its not clear to me who you were replying to.


I answered at the bottom of the thread so to PJE.



  Anyway I find it really disturbing the way you're actually acting
  and try to push your ideas based on private feedback coming from unknown
 or
  choosing who should be a reference. That certainly not the right way to
 have
  all actors on the table. Because if we go for a new WSGI spec, you
 certainly
  want it. And I am speaking as one of these actors.

 As I said when folk talked about going private in the first thread on
 this, I'm willing to discuss anything publically or privately - I
 can't tell folk where they will be comfortable discussing things. But
 I'm going to do *my* work on this in public, because I think that is
 essential to get broad consensus.

  In my opinion, if we want to go further we should first define what are
 the
  problem we want to solve, and then get the feedback from all the actors
  around:

 I think I've been fairly clear about the problem *I* want to solve.

 
 We want to create a clean common API for applications and middleware
 written in a post HTTP/2 world - where single servers may accept up to
 all three of HTTP/1.x, HTTP/2 and Websocket connections, and
 applications and middleware want to be able to take advantage of
 HTTP/2 and websockets when available, but also degrade gracefully. We
 also want to ensure that there is a graceful incremental path to
 adoption of the new API, including Python 2.7 support, and shims to
 enable existing WSGI apps/middleware/servers to respectively be
 contained, contain-or-be-contained and contain, things written to this
 new API. We want a clean, fast and approachable API, and we want to
 ensure that its no less friendly to work with than WSGI, for all that
 it will expose much more functionality.
 



By which problem we need to solve, I mean we need to identify clearly what
are the problem not solved by the current spec. And see why, and how it is
actually solved in the python world. we need to clearly identify these
issues and make sure we have a comprehensive view of them.



  - framweorks authors

 I reached out to a number of such authors directly. I encourage you to
 do the same.



I could do that eventually if we are all agree on the process.



  - libraries author

 Ditto and

  - server authors

 Ditto :).

  If you don't have all actors around and majors are missing, there is
  probably no point to continue. I do think the idea of having a
 repository to
  collect it with people arbitrating the discussions on them on the
 mailing is
  a good way to go further. Now I think we are still missing of a clear
  definition of the problem. This is from what we should start instead of
  starting by giving our philosophy on how the problem should be solved.

 Here's my definition of some of the problems:
 A - there is no common spec equivalent to WSGI that permits writing
 server side code that takes advantage of HTTP/2. There's *a* http/2
 server out there which one can write code for, but that code is either
 specific to that servers plumbing, or plain WSGI and misses the HTTP/2
 goodness.
 B - WSGI has some oddness and overheads due in large part to the
 situation it was aiming to fix (which it broadly did) that perhaps we
 can now come together to fix.
 C - Support for chunked uploads, comet, bosh and websockets is
 effectively impossible within WSGI - one ends up writing server
 specific code, and being tied to a single server - even though
 multiple servers support (some of) those things. This defeats the
 point of WSGI IMNSHO: its not that WSGI is broken or anything, its
 just that we're once again writing all our generic middleware in
 server-specific fashions. Because the world has moved on and we
 haven't.


Chunkedn upload is possible and already handled with Gunicorn. But there is
no standard for that.

For C I would separate it from the rest. This a different discussion and
imo not everything can be achieved at the same time. Maybe we should start
first by fixing them, then go for the next step anyway. So the transition
could be incremental in servers and frameworks and actually fix the current
spec.


For A (And C), i think we should keep the new specification enough
agnostic. Especially since HTTP 2 is not yet completely out.


- benoit

I
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] REMOTE_ADDR and proxys

[Benoit Chesneau]

On Sun, Oct 12, 2014 at 11:38 PM, Robert Collins robe...@robertcollins.net
wrote:

 On 30 September 2014 11:47, Alan Kennedy a...@xhaus.com wrote:

  [Robert]
  So it sounds like it should be the responsibility of a middleware to
  renormalize the environment?
 
  In order for that to be the case, you have strictly define what
  normalization means.

 For a given deployment its well defined. I agree that in general its not.

  I believe that it is not possible to fully specify normalization, and
 that
  any attempt to do so is futile.
 
  If you want to attempt it for the specific scenarios that your particular
  application has to deal with, then by all means code your version of
  normalization into your application. Or write some middleware to do it.
 
  But trying to make normalization a part of a WSGI-style specification
 is
  impossible.

 I don't recall proposing that it should be in a WSGI-style spec.

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud
 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com



All this issue looks like the problem raised (and not yet solved) recently
in Gunicorn when the REMOTE_ADDR has been handled more strictly and we
removed all the X-Forward-* headers handling:

https://github.com/benoitc/gunicorn/issues/797

There is another case to take in consideration, when your server is
answering on unix sockets, so you don't have any TCP address to present.
For now we answer with an empty field.

Also some application frameworks recently removed the middleware handling
X-Forward-* headers. I wonder why.


There is an RFC for forward headers: http://tools.ietf.org/html/rfc7239 .
For me instead of trying to change the strict behaviour of REMOTE_ADDR I
wonder if we shouldn't rather add a new field to the environ. Thoughts?

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Draft 2: WSGI Response Upgrade Bridging

[Benoit Chesneau]

On Fri, Oct 10, 2014 at 9:10 PM, PJ Eby p...@telecommunity.com wrote:

 On Fri, Oct 10, 2014 at 8:56 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:
  So PJE, please step back and do not go rushing out to create a PEP. That
 is
  the worst thing you could do at this point and will only serve to deter
  people from the community contributing and so stifle proper discussion
 about
  this whole topic.

 Huh?  Have you *read* the PEP?  The entire point of it is to provide a
 basis for *experimenting* with new standards, not to stifle
 discussion of them.  It's not even an *API*, for heavens' sake.  It's
 just a description of how to upgrade to new standards from within
 existing WSGI frameworks, without needing to tunnel responses and
 without breaking subrequest middleware.

 IOW, it's a WSGI *1.0* server extension protocol, and a fairly *minor*
 one at that.  (Indeed, it's little more than an enhanced variation of
 wsgi.file_wrapper!)  It's not any sort of competitor or alternative to
 what Robert's working on; it's a *stepping stone* for what Robert's
 working on.

 In early discussion with Robert -- both here and on github -- it
 became apparent to me that restricting post-WSGI specifications to
 what can be achieved in WSGI 1 tunneling is a bad idea.  So I've
 created a *bridging* specification, that allows post-WSGI APIs to be
 accessed from within WSGI-based apps and frameworks.

 That's *all*.

 All of the things you've mentioned as being in scope for discussion,
 are *still* in scope for discussion.  All *this* proposal does is show
 how those things could be *accessed*, *today*, from inside existing
 web apps and frameworks, once those new APIs exist.


  You have no more experience or mandate to be specifying a
  standard for this than anyone else.

 If by this you're referring to HTTP/2 or some other new post-WSGI
 API, then I agree with you.  But that's not what the PEP is about.


   By creating a PEP though that gets
  perceived by many as meaning the discussion is over. This is exactly what
  you did for PEP  and which caused previous discussion about improving
  WSGI to get shutdown.

 That's an interesting perspective, but I don't see how it can be
 reconciled with the facts.

 First off, I didn't write a new PEP; I wrote up some of *your*
 proposed clarifications for Python 3 as WSGI 1.0.1, which was intended
 to add new clarifying text to PEP 333, *not* to create a new PEP.  It
 was *Guido* who said it must be a new PEP, as you will see here:

   https://mail.python.org/pipermail/web-sig/2010-September/004691.html

 and here (where he even says, Don't see this as a new spec. See it as
 a procedural issue.):

   https://mail.python.org/pipermail/web-sig/2010-September/004694.html

 Second, I didn't make anybody stop discussing alternatives for moving
 things forward.  Nobody *ever* said to stop working on a version 2 or
 even 1.1, certainly not me.  See for example, this message, where I
 agreed with Ian's POV that there was room for both PEP 333 fixes *and*
 continued work on PEP 444:

   https://mail.python.org/pipermail/web-sig/2010-September/004662.html

 Third, and finally, as far as I can tell from the record of the
 discussion back then, it was you -- and *only* you -- who suggested
 that the acceptance of PEP  meant the discussion was *over*.
 Indeed, on your blog you actually pushed back at Alice for bringing up
 more PEP 444 discussion!

 Nonetheless, discussion of PEP 444 and async APIs and such proceeded
 well past the introduction of PEP , even without its original
 authors' participation.  And, ironically enough, your posts show up in
 that discussion, bemoaning that Alice (the new PEP 444 champion) was
 creating confusion by calling that proposal WSGI 2.0!


  The result was that the only thing that really got
  addressed in PEP  was Python 3 compatibility and a lot of the other
 bits
  of the WSGI specification which are poorly defined, contradictory or
  restrictive and which cause WSGI server and application developers pain
  never got addressed. If that prior discussion hadn't been shutdown in
 that
  way, we could have been using a better defined and improved WSGI years
 ago
  already.

 Those things didn't get addressed because *you* didn't take up the
 lead -- a lead which I more than once mentioned you should take up.
 For example, as I said in
 https://mail.python.org/pipermail/web-sig/2010-September/004693.html :

  The full list of things Graham and others have asked for or
  recommended would indeed require a 1.1 version at minimum, and thus a
  new PEP.  But I really don't want to start down that road right now,
  and therefore hope that I can talk Graham or some other poor soul
  into shepherding a 1.1 PEP instead.  ;-)

 You didn't, and haven't, taken up that slack.  What you've
 consistently done is mutter and grumble on the sidelines about how
 it's never going to happen and disclaim any responsibility for
 actually writing a proposal 

Re: [Web-SIG] WSGI2: write callable?

[Benoit Chesneau]

On Fri, Sep 26, 2014 at 9:58 PM, PJ Eby p...@telecommunity.com wrote:

 On Thu, Sep 25, 2014 at 11:32 PM, Robert Collins
 robe...@robertcollins.net wrote:
  So I propose we drop the write callable, and include a queue based
  implementation in the adapter for PEP- code.

 If you're dropping write(), then you might as well drop
 start_response() altogether, and replace it with returning a (status,
 headers, body-iterator) tuple, as in wsgi_lite (
 https://github.com/pjeby/wsgi_lite ) or as found in other languages'
 versions of WSGI.  (start_response+write was only ever needed in order
 to support legacy apps, so other languages never bothered.)

 wsgi_lite has a couple of other protocol extensions, namely the
 'wsgi_lite.closing' environment key, flagging callables' supported
 WSGI version (for transparent interop), and the argument binding
 protocol, but for the most part these are orthogonal to the calling
 schema.  I would suggest, however, that the calling protocol be
 flagged in some way to allow easier interop.


I quite like the idea of always returning an iterator for the body it would
simplify the code a lot...

About returning the status and other thing, I quite agree, but imo we also
need to return an extra parameter where the application or the middleware
could maintain a state or something like it. Thoughts?

- benoit


 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI for HTTP/2.0 ?

[Benoit Chesneau]

Hi,

I would prefer to have this work being done transparently. If we do it
rationally  it could work imo.

Anyway before thinking to change the protocol or criticizing it maybe we
could first collect the requirements in HTTP 2 (stream and such) so we can
think about possible implementations. And see what it misses in WSGI.

I am thinking we could adopt the same path used to decided to go for HTTP
1.x or HTTP 2  on the client part. Ie keeping WSGI and PEP  for HTTP
1.1 applications  and go for a new interface in HTTP2. But such decision
should be done once we have a clear view of what requires HTTP 2 and how it
can be handled on the python side.

Thoughts?

- benoit



On Sat, Sep 20, 2014 at 8:31 AM, Graham Dumpleton 
graham.dumple...@gmail.com wrote:


 On 20/09/2014, at 3:49 PM, Roberto De Ioris robe...@unbit.it wrote:

  I can help a bit (i am the uWSGI lead developer and a nginx and Cherokee
  contributor, and i have already implemented a spdy3 server last year)
 
  I honestly think that WSGI by itself needs a complete rewrite/rethink to
  be adapted to modern (ok someone could say 'fashioned') patterns (that
 are
  somewhat more 'urgent' than HTTP/2), but i agree that starting thinking
  about HTTP/2 could be a good thing.

 I agree.

 Overhauling WSGI has more relevance because an underlying web server
 updating itself to support HTTP 2.0 will in the main have little relevance
 at the application layer as the web server is more than likely to have an
 adapter layer which makes things look the same to existing modules/protocol
 adapters.

 In other words, Apache adding support for HTTP 2.0 isn't going to result
 in some sort of wholesale change of the Apache module interface, it would
 stay the same say whether HTTP 2.0 is used, especially just as an alternate
 way of doing the same thing as HTTP 1.1. In that respect, since no HTTP 2.0
 specific functionality is going to be made visible through exist
 interfaces, then Apache modules or adapters for FASTCGI/SCGI etc or even
 mod_wsgi are simply not going to change.

 So, overhaul WSGI as the primary aim, but within that factor in things to
 allow for HTTP 2.0 functionality.

 The problem with trying to overhaul WSGI is that if it is done in an open
 forum like the Web-SIG it will die of a thousand cuts, as past efforts to
 update it in even minor ways have suffered.

 The only way that WSGI itself will ever see an overhaul is through the
 strong willed determination of a few people off list, out of sight, to
 allow it it to be fully fleshed out, with input coming from direct
 consultation with and review by other related parties who have a vested
 interested or significant experience in the area.

 I may be up for such an off list effort, but be warned I may want to run
 roughshod over it and exert quite a lot of influence over the process and
 outcome. :-)

 Graham
 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 https://mail.python.org/mailman/options/web-sig/bchesneau%40gmail.com

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI for HTTP/2.0 ?

[Benoit Chesneau]

On Sat, Sep 20, 2014 at 3:22 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote:

 On Sat, Sep 20, 2014 at 9:23 AM, Robert Collins
 robe...@robertcollins.net wrote:
  Well, thats certainly a challenge :). Whats the governance model here?
  Is a PEP appropriate, and if so - that gives us a BFDL or BFDL
  PEP-delegate to decide between bikeshed issues; and if its not a
  bikeshed issue then resolving it is actually necessary.

 Yes, I think a good way forward would be to have a small cabal write a
 PEP and then announce it here for further feedback and then
 pronouncement by a BDFL or -delegate. If you want to be lead/editor,
 that sounds great. It also seems like you should definitely involve
 Graham and give credence to his thoughts.

 I'd be excited about this and happy to give feedback a little later
 once you've got some initial draft, as someone who likes to implement
 his applications directly on top of WSGI for now (but I've also
 implemented a couple of WebSocket servers).

 Cheers,

 Dirkjan


Last time was more about everyone wanted to discuss about the changes with
its own requirement list.  Which is already conflicting. Discussing if it
should be done outside the media designed for it is already out of topic
imo. So I won't discuss about it further.

Instead I wonder what is the appropriate medium to collect requirements and
others stuffs about it. Wiki ? Anything else?

For  a start I see these different topics


1) HTTP 1.1 vs HTTP 2:

-  HTTP 1.1 and HTTP2 have quite the same high level syntax (methods, uri,
headers, ...) but the way the data is transported differs. (data are sent
by frames in HTTP 2).
- in HTTP 2, data can be encrypted and compressed.
- in HTTP2 data can pushed from the server to the clients. More data can be
sent to the client
- in HTTP2 streams are multiplexed We have the concept of data channels and
these are more like message passings. Multiplexing existed in HTTP 1.1 with
pipelines but is barely supported right now by WSGI servers.


The concept of data channels and the PUSH features will requires more
concurrency at the server level.

At the application, things doesn't change that much. Everything can appear
like before. The only change is the PUSH feature.


2) Websockets, SSE and other similar protocoles are completely
asynchronous.  All this part is not really handled by WSGI. The way it is
generally implemented right now is awkward. The server generally extend the
WSGI protocol so the application get the socket. Then a specific library
handle the rest.


I actually wonder if websockets or other asynchronous protocols should be
handled  by the new WSGI SPEC. Shouldn't we just standardize the way the
socket is given to another library?


Anyway I think we should collect all requirements at application and server
level and then start to confront the current WSGI spec to them. And
iterate. Thoughts? Any other topic?

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI for HTTP/2.0 ?

[Benoit Chesneau]

got an idea. What about having a page collecting feedback from anyone in
the python community about this topic. So we can have true data from
different perspectives: developer, library/framework author, server author.
I'm OK to collect the data from it and make a summary of it once it's done.

The form it could take should be discussed first but imo that a good way to
engage the community. What do you think?



On Sat, Sep 20, 2014 at 4:17 PM, Benoit Chesneau bchesn...@gmail.com
wrote:



 On Sat, Sep 20, 2014 at 3:22 PM, Dirkjan Ochtman dirk...@ochtman.nl
 wrote:

 On Sat, Sep 20, 2014 at 9:23 AM, Robert Collins
 robe...@robertcollins.net wrote:
  Well, thats certainly a challenge :). Whats the governance model here?
  Is a PEP appropriate, and if so - that gives us a BFDL or BFDL
  PEP-delegate to decide between bikeshed issues; and if its not a
  bikeshed issue then resolving it is actually necessary.

 Yes, I think a good way forward would be to have a small cabal write a
 PEP and then announce it here for further feedback and then
 pronouncement by a BDFL or -delegate. If you want to be lead/editor,
 that sounds great. It also seems like you should definitely involve
 Graham and give credence to his thoughts.

 I'd be excited about this and happy to give feedback a little later
 once you've got some initial draft, as someone who likes to implement
 his applications directly on top of WSGI for now (but I've also
 implemented a couple of WebSocket servers).

 Cheers,

 Dirkjan


 Last time was more about everyone wanted to discuss about the changes with
 its own requirement list.  Which is already conflicting. Discussing if it
 should be done outside the media designed for it is already out of topic
 imo. So I won't discuss about it further.

 Instead I wonder what is the appropriate medium to collect requirements
 and others stuffs about it. Wiki ? Anything else?

 For  a start I see these different topics


 1) HTTP 1.1 vs HTTP 2:

 -  HTTP 1.1 and HTTP2 have quite the same high level syntax (methods, uri,
 headers, ...) but the way the data is transported differs. (data are sent
 by frames in HTTP 2).
 - in HTTP 2, data can be encrypted and compressed.
 - in HTTP2 data can pushed from the server to the clients. More data can
 be sent to the client
 - in HTTP2 streams are multiplexed We have the concept of data channels
 and these are more like message passings. Multiplexing existed in HTTP 1.1
 with pipelines but is barely supported right now by WSGI servers.


 The concept of data channels and the PUSH features will requires more
 concurrency at the server level.

 At the application, things doesn't change that much. Everything can appear
 like before. The only change is the PUSH feature.


 2) Websockets, SSE and other similar protocoles are completely
 asynchronous.  All this part is not really handled by WSGI. The way it is
 generally implemented right now is awkward. The server generally extend the
 WSGI protocol so the application get the socket. Then a specific library
 handle the rest.


 I actually wonder if websockets or other asynchronous protocols should be
 handled  by the new WSGI SPEC. Shouldn't we just standardize the way the
 socket is given to another library?


 Anyway I think we should collect all requirements at application and
 server level and then start to confront the current WSGI spec to them. And
 iterate. Thoughts? Any other topic?

 - benoit









___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] question about connection pool, task queue in WSGI

[Benoit Chesneau]

On Sun, Jul 15, 2012 at 5:14 PM, Simon Sapin simon.sa...@exyr.org wrote:
 Le 14/07/2012 06:07, Graham Dumpleton a écrit :

 2. Is the socket FD the same mechanism like nginx? If you upgrade
  nginx
 binary, restart nginx, the existing http connection won't break.

 I would be very surprised if you could upgrade nginx, perform a
 restart and preserve the HTTP listener socket. If you are talking
 about some other socket I don't know what you are talking about.

 As you can with Apache, you can likely enact a configuration file
 change and perform a restart or trigger rereading of the configuration
 and it would maintain the HTTP listener socket across the
 configuration restart, but an upgrade implies changing the binary and
 I know no way that you could easily persist a HTTP listener socket
 across to an invocation of a new web server instance using a new
 executable. In Apache you certainly cannot do it, and unless nginx has
 some magic where the existing nginx execs the new nginx version and
 somehow communicates through open socket connections to the new
 process, I very much doubt it would as it would be rather messy to do
 so.


 I think that est refers to this:
 http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly

 Apparently yes, there is specific code in nginx to start the new binary and
 give it the existing socket.

 And I think that yes, Tarek’s new Circus is similar to the nginx magic
 upgrade in that an open socket is passed around processes. Maybe nginx even
 does this in normal operation with multiple worker processes, but I don’t
 know.

 Regards,
 --
 Simon Sapin
Gunicorn does upgrade itself using the USR2 signal just like nginx and
share the socket like using an fd between OS processes.

However the case of a db is a little different since you handle a
connection to a db and not listening on a port. You will need either a
multiprocess queue passing messages to one process or launching a
connection per processes. You can do that using the hook system of
gunicorn.


- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Starting Web Servers using socket FDs

[Benoit Chesneau]

On Tue, Jun 5, 2012 at 12:26 PM, Tarek Ziadé ta...@ziade.org wrote:
 On 6/5/12 11:46 AM, Roberto De Ioris wrote:

 ...

 Gunicorn can already bind (or better, accept) from file descriptors
 specifying an environment variable.

 I don't think you can start gunicorn using a file descriptor, or I failed to
 do it. The best I was able to do was to create a small wsgi server
 using Gunicorn as a lib.

 Gunicorn uses an environment variable when it respawns workers but it does
 not offer it as a public option as far as I understand how it works


export GUNICORN_FD=your fd

and then gunicorn will use this file descriptor when it starts.

But it should be possible to pass it directly using a config option if
it's needed.

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] A 'shutdown' function in WSGI

[Benoit Chesneau]

On Mon, Feb 20, 2012 at 5:03 PM, Tarek Ziadé ziade.ta...@gmail.com wrote:
 Hello

 I need to be able to call a function when the web application shuts down
 (SIGTERM/SIGINT) -- the use case is to stop a background thread.

 I am currently using signals because it seems to be the most clean way to do
 this. atexit is much trickier since you don't know when it's going to get
 called and you might try to call objects that were garbage collected unless
 you hack something to keep references alive.

 But signals are also tricky beasts since you may compete with other code
 that are listening to them. For instance mod_wsgi don't like apps that have
 signal handlers.

 Anyways, the bottom line is that the cleanest way to do this -- as per Chris
 McDonough idea, would be to introduce in the WSGI protocol a shutdown
 function the servers would be obligated to call before exiting.

 I am not sure yet about its arguments, maybe a signum + frame or simply an
 exit code...

 But how do you like the idea ?  That would solve for me the problem of
 having to deal differently here depending on if I am called with mod_wsgi or
 gunicorn or xxx

Hi Tarek,

In gunicorn we have the concept of hook which exactly does what you want
(on_exit hook). You can do this via the configuration file. And you can
do this for the other steps (reload, pre/post fork, pre/post request
...).

Anyway, I think there are 2 issues to solve with this shutdown thing
(and more) in a generic way:

1. how to make your application aware of the server capabilities (can it
or not handle shutdown)
2. how to pass such functions to the server.

There are also some others details like determining what is the context
needed by the application at this step.

Today the server is acting as a gateway so it just pass a request to an
application and return the response from the application. The
application by itself only lives in this context. Maybe, like some said,
if we want to give more info about that, we could imagine the application
object or function as a bus reacting on different contexts:

def application(environ, start_application):
pass

where the environ can be different depending on the context an action, we
could have:

http request{wsgi.context: http, ...}
shutdown{wsgi.context; server, action: shutdown, ...}

smth like it. But how to know when the server can't pass the action
shutdown? Also what would be the environ at shutdown?

One other method, like the one described by Sylvain Hellegouarch, would
be adding some properties/attributes to the application object. So the
server would eventually check if they exists and eventually call them.
But again how to know when the server can't pass the action shutdown?
Though, I don't like to handle more than the application callable today,
I can see some problems when you have to handle server reload and such
things.

It seems important for me that the application knows about the server
capabilities if it want to handle such things like the `shutdown`. And
this step is solved by gunicorn  others by specifically handling a
configuration. But maybe we could just considers that a server always
handle the reload, shutdown, also pre/post request evenst and have
optionnals message passed to the application? (how do we handled process
context changes - if we re in a thread for ex - ?).

We could also imagine a web application package that provides hooks
handling depending on the server target... I quite like this idea.

Not sure what is the right way. What do you think?

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] A 'shutdown' function in WSGI

[Benoit Chesneau]

On Tue, Feb 21, 2012 at 10:35 AM, Sylvain Hellegouarch s...@defuze.org wrote:


 On Tue, Feb 21, 2012 at 10:31 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:

 On 21 February 2012 20:26, Simon Sapin simon.sa...@exyr.org wrote:
  Le 21/02/2012 09:23, Tarek Ziadé a écrit :
 
     Instead of having to provide two or three objects separately to a
     server, how about making the callbacks attributes of the application
     callable?
 
 
  can you show us an example ?
 
 
  Proposal:
 
  Function-based:
 
     def startup():
         return open_resource(something)
 
     def shutdown(resource):
         resource.close()
 
     def application(environ, start_response):
         # ...
         return response_body
 
     application.startup = startup
     application.shutdown = shutdown
 
  Class-based:
 
     class App(object):
         def startup(self):
             return open_resource(something)
 
         def shutdown(self, resource):
             resource.close()
 
         def __call__(self, environ, start_response):
             # ...
             return response_body
 
     application = App()
 
  The return value of startup() can be any python object and is opaque to
  the
  server. It is passed as-is to shutdown()
 
  startup() could take more parameters. Maybe the application (though can
  we
  already have it as self for class-based or in a closure for
  function-based)

 You do realise you are just reinventing context managers?

 With this 'application' do requests.

 But then it was sort of suggested that was a bit too radical idea when
 I have mentioned viewing it that way before. :-(


 One might wonder if having access to process management should be part of
 WSGI in the first place.


that's the thing. This is no more a gateway. Like I said in my
previous post, maybe having another spec describing a web app package
would do the trick?

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)

[Benoit Chesneau]

On Tue, Jan 4, 2011 at 12:13 AM, Jacob Kaplan-Moss ja...@jacobian.org wrote:
 On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum gu...@python.org wrote:
 Although [PEP ] is still marked as draft, I personally think of it
 as accepted; [...]

 What does it take to get PEP  formally marked as accepted? Is
 there anything I can do to push that process forward?

 The lack of a WSGI answer on Py3 is the main thing that's keeping me,
 personally, from feeling excited about the platform. Once that's done
 I can feel comfortable coding to it -- and browbeating those who don't
 support it.

 I understand that PEP 444/Web3/WSGI 2/whatever might be a better
 answer, but it's clearly got some way to go. In the meantime, what's
 next to get PEP  officially endorsed and accepted?

 Jacob

I we pep is considered accepted, I will start the port of gunicorn
based on it too. It shouldn't be hard since we are using our own http
parser.

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Most WSGI servers close connections to early.

[Benoit Chesneau]

On Wed, Sep 22, 2010 at 2:46 PM, Marcel Hellkamp m...@gsites.de wrote:
 I just discovered a problem that affects most WSGI server
 implementations and most current web-browsers (tested with wsgiref,
 paste, firefox, chrome, wget and curl):

 If the server closes the connection while the client is still uploading
 data via POST or PUT, the browser displays an error message ('Connection
 closed') and does not display the response sent by the server.

 The error occurs if an application chooses to not process a form
 submissions before returning to the WSGI server. This is quite rare in
 real world scenarios, but hard to debug because the server logs the
 request as successfully sent to the client.

 To reproduce the problem, run the following script, visit
 http://localhost:8080/ and upload a big file::



 from wsgiref.simple_server import make_server

 def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [
    form method='post' enctype='multipart/form-data'
      Upload bog file:
      input type='file' name='file' /
      input type='submit' /
    /form
    ]

 server = make_server('localhost', 8080, application)
 server.serve_forever()




 I would like to add a warning to the WSGI/web3 specification to address
 this issue:

 An application should read all available data from
 `environ['wsgi.input']` on POST or PUT requests, even if it does not
 process that data. Otherwise, the client might fail to complete the
 request and not display the response.

 --
 Mit freundlichen Grüßen
 Marcel Hellkamp

Your application and client should be aware of Expect: 100-Continue header :

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

- benoît

(resent, because web-sig doesn't set well the default reply-to)
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Most WSGI servers close connections to early.

[Benoit Chesneau]

On Wed, Sep 22, 2010 at 5:34 PM, Robert Brewer fuman...@aminus.org wrote:

 However, the caveat requires a caveat: servers must still be able to protect 
 themselves from malicious clients. In practice, that means allowing servers 
 to close the connection without reading the entire request body if a certain 
 number of bytes is exceeded.

I don't see how it could be the responsability of the server. Can you
develop a little ? The server shouldn't interfere in the HTTP request
imo.

- benpît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] PEP 444 (aka Web3)

[Benoit Chesneau]

On Fri, Sep 17, 2010 at 10:36 AM, Georg Brandl g.bra...@gmx.net wrote:
 Am 16.09.2010 23:07, schrieb James Mills:
 - the web3 name

  If there is any value in this PEP and we find something to decide on,
  there is no reason this couldn't be WSGI 2.  But until it's just
  something a small part of the web-sig community worked on directly
  a separate name is a good thing I think, because it does not reserve
  the name WSGI 2 for something that might actually become WSGI 2
  in case this PEP gets rejected.

 I personally still don't see any real benefit to changing the key names
 from wsgi to web3 (or whatever). I would prefer it remain the
 same. If you're going to use Python3, you know you're using Python3
 (you don't need web3 key names to know that). (subjective)

 That statement shows another weakness of the web3 name: this spec is not
 in the least exclusive to Python 3.  (Which would be a bit useless, having
 two incompatible WSGI/web specs on two incompatible Python versions.)

 The goal would be to first migrate to WSGI2/web3, and *then* have an easy
 transition going to Python 3.

 Georg

also WSGI acronym is defining better the purpose by itself than web3
which mean nothing.

- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] PEP 444 (aka Web3)

[Benoit Chesneau]

On Thu, Sep 16, 2010 at 6:41 PM, Armin Ronacher
armin.ronac...@active-4.com wrote:

  4. The web3 spec says, In case a content length header is absent the
     stream must not return anything on read. It must never request more
     data than specified from the client. but later it says, Web3
     servers must handle any supported inbound hop-by-hop headers on
     their own, such as by decoding any inbound Transfer-Encoding,
     including chunked encoding if applicable.. I would be sad if web3
     did not support streaming uploads via Transfer-Encoding. One way to
     implement that would be to make the origin server handle read()
     transparently by returning '' on EOF, regardless of whether a
     Content-Length or a Transfer-Encoding header was provided.

 I was toying with the idea to have a websocket extension for web3 which
 would have solved my usecase for requests without a content-length header.
  The problem with the content length of incoming data is quite complex and
 that seemed to be the solution that was easiest for everybody involved.

uh ? Since with Transfer-Encoding: chunked we know when the stream
end, I would be in favor of returning an EOF too at the end. Also most
of servers know when a stream end even if there is no content-length.
Maybe we could have a capability setting in environ that say if the
server support streaming or not. And in all cases returning EOF at the
end?

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

[Web-SIG] process ..

[Benoit Chesneau]

Hi all,

As external I'm a bit surprised and also disappointed about actual  
process about choosing/discussing next features or changes of SGI spec.


I thought and I think process should be more formal. Like I see it  
since some months, there are people trying to put their views and  
technical considerations but that's all. I don't see any vote about  
adding/editing current version of the spec. Some people try to  
summarize, but there is also not formal way. Also working on different  
repositories for one spec, with changes without votes make me think  
something is wrong. I don't see any step in discussion nor roadmap and  
a  standard shouldn't be handled like this in my opinion.


Maybe I'm wrong, maybe votes are hidden, there is something outside I  
ignore and I will be happy to be wrong on this. Anyway I really think  
in this case process should be more transparent. At least having a  
roadmap with formal steps like you have on other standards would help.  
The web is changing, and as a developer I would like to know what and  
when will be adapted WSGI to new needs.


- benoit
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com