Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 6 Jan 2016, at 20:06, Graham Dumpleton  wrote:
> 
> 
>> On 6 Jan 2016, at 10:19 PM, Cory Benfield  wrote:
>> 
>> 
>>> On 6 Jan 2016, at 09:48, Graham Dumpleton  
>>> wrote:
>>> 
>>> If this does solve the push issue, what is there in HTTP/2 then that one 
>>> couldn’t do via the existing WSGI interface?
>> 
>> Well, plenty, but none that we’d *want* to expose via WSGI with the possible 
>> exception of long-running bi-directional communications channels like 
>> Websockets, which you’ve already expressed a desire to expose in a different 
>> API. =)
> 
> Can you elaborate more on the ‘plenty’ part.
> 
> This was the issue in the past. People appeared to want access to everything. 
> Thus why the belief you may as well allow them a separate API purpose built 
> for it. Maybe people are becoming more realistic in expectations now as to 
> what they really need for a typical web application as HTTP/2 is better 
> understood.

Sure. =)

HTTP/2 has the following extra things that a user may want control over:

- Preventing headers being added to the compression context. Good servers will 
automatically do this for things like Set-Cookie, but an application may have 
special knowledge about a header being security-relevant. There is no way to 
signal this in WSGI.
- Forcing certain headers to be added to the compression context. The flip side 
of the first part is that an application may know that a header is likely to be 
repeatedly re-used, despite being something that would ordinarily vary, and it 
may want to pass this knowledge to the server.
- Server push before the complete set of response headers are ready (i.e. 
before the original response body is rendered).
- Long-running bi-directional communications channels, like Websockets.
- Signaling information about client priority to the application so it can 
allocate resources effectively.
- Signaling information about flow control to the application so it can 
allocate resources effectively. (Both of these two could *maybe* be done using 
generators with WSGI, but it would be imperfect.)
- Controlling flow control windows for large responses (e.g. deliberately 
shrinking or widening them).
- Controlling flow control widows for connections as a whole.
- Graceful connection shutdown (emitting GOAWAY but continuing to process the 
outstanding requests/responses).
- Efficient stream cancellation (e.g. sending RST_STREAM for unwanted/invalid 
requests without tearing down the connection, likely related to the previous 
point)

HTTP/2 is a very featureful protocol which complex applications could really 
take advantage of. Worth remembering.


>> Pushing via Link headers is not ideal because it delays the push until after 
>> the headers are ready to go, and there’s a tricky ordering concern here (RFC 
>> 7540 points out that any PUSH_PROMISE frames should be sent before the 
>> response headers and body are sent, which means that we temporarily block 
>> the response that is ready to go from WSGI. This is a minor concern, but 
>> worth noting.)
>> 
>> However, I’m happy to say that Pushing via Link headers is the way to go if 
>> we’d rather not specify a WSGI-specific API for it.
> 
> 
> It appears that Link could at least be a fallback.
> 
> The idea of a separate callable to push resources in WSGI environ could still 
> be dealt with as an extension specification and so not need changes to the 
> WSGI specification itself. Worst case is all that callable does is add Link 
> headers to the response. This would likely have to be the case in Apache with 
> mod_h2 as wouldn’t expect handlers to have direct access to an API to do it 
> any other way, plus in mod_wsgi daemon mode is in the wrong process to access 
> any API.

Yup, this is definitely an option.



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Benoit Chesneau]

On Wed, Jan 6, 2016 at 10:19 AM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello Benoît,
>
> Thanks for clarifying that you also had the reverse problem in mind,
> headers sent by applications. This side is less problematic in the sense
> that application authors can adapt to stronger requirements.
>
> In general this is a bit of a mess due to differences between what the RFC
> 2616 says and what browsers do in practice. That’s why I believe the
> pragmatic solution is to exchange bytes. (This isn’t a major issue in the
> grand scheme of things anyway.)
>
> Best regards,
>
> --
> Aymeric.
>
>
Since sockets are only accepting bytes in python 3.x,  having everything as
bytes would also allow the server to process the response more easily...


- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 6 Jan 2016, at 12:13 AM, Benoit Chesneau  wrote:
> 
> So for me what should be WSGI 2? WSGI 2 should add against WSGI 1 the 
> following:
> 
> - tell to the application it is actually an HTTP2 request (maybe populating a 
> wsgi.http2 true env)

In CGI implementations you would for HTTP/1.1 already get:

SERVER_PROTOCOL: 'HTTP/1.1’

Under HTTP/2 when I tested some time back, I recollect it came through as one 
would assume would be expected:

SERVER_PROTOCOL: ‘HTTP/2’

Is there any reason that this existing CGI variable wouldn’t be sufficient for 
this purpose?

Graham___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Benoit Chesneau]

On Wed, Jan 6, 2016 at 10:29 AM Graham Dumpleton 
wrote:

>
> On 6 Jan 2016, at 12:13 AM, Benoit Chesneau  wrote:
>
> So for me what should be WSGI 2? WSGI 2 should add against WSGI 1 the
> following:
>
> - tell to the application it is actually an HTTP2 request (maybe
> populating a wsgi.http2 true env)
>
>
> In CGI implementations you would for HTTP/1.1 already get:
>
> SERVER_PROTOCOL: 'HTTP/1.1’
>
> Under HTTP/2 when I tested some time back, I recollect it came through as
> one would assume would be expected:
>
> SERVER_PROTOCOL: ‘HTTP/2’
>
> Is there any reason that this existing CGI variable wouldn’t be sufficient
> for this purpose?
>
> Graham
>


True. I don't think there is a need for another env. The current way also
work for the future :)

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 6 Jan 2016, at 09:19, Aymeric Augustin 
>  wrote:
> 
> Hello Benoît,
> 
> Thanks for clarifying that you also had the reverse problem in mind, headers 
> sent by applications. This side is less problematic in the sense that 
> application authors can adapt to stronger requirements.
> 
> In general this is a bit of a mess due to differences between what the RFC 
> 2616 says and what browsers do in practice. That’s why I believe the 
> pragmatic solution is to exchange bytes. (This isn’t a major issue in the 
> grand scheme of things anyway.)
> 
> Best regards,
> 

Folks, just a reminder: RFC 2616 is dead. RFC 7230 says that *newly defined* 
header fields should limit their field values to US-ASCII, but older header 
fields are a crapshoot (though it notes that “in practice, most” header field 
values use US-ASCII).

Regardless, it seems to me that the correct method of communicating field 
values would have been byte strings.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 5 Jan 2016, at 10:31 PM, Graham Dumpleton  
> wrote:
> 
>>> For example, mod_wsgi already supports HTTP/2 by virtue of the fact that 
>>> the mod_h2 module in Apache exists. The existing internal APIs of Apache 
>>> and how mod_wsgi uses those means that HTTP/2 bridges into the WSGI world 
>>> with no code changes to mod_wsgi.
>> 
>> Agreed. If all we want is to keep the request/response cycle intact, then 
>> WSGI supports H2 already. One possibility that has already been suggested 
>> here would be to define a HTTP/2 extension to WSGI, advertised in the 
>> environ dict, that allows the application to signal pushes to the server. 
>> This would be a fairly simple extension to write and implement.
> 
> Sorry to be cynical. Many people have said that changes in the past related 
> to WSGI will 'be a fairly simple extension to write and implement’. Dig 
> deeper and it never turns out to be the case. :-)
> 
> Such an extension presumes you actually have a tightly integrated HTTP/2 
> server which itself which can maintain a map of resources to push when 
> getting certain requests and also maintain what may have already been sent 
> for a session. Even getting to that point is going to be non trivial, even if 
> an extension may be simple for somehow notifying what the additional 
> resources to supply should be.
> 
> Right now I would say that with mod_h2 in Apache in would be plain impossible 
> as it doesn’t I believe even support the idea of pushing resources at this 
> point. Even then it would most likely be a huge undertaking to get it to work 
> for mod_wsgi daemon mode as the web application runs in a separate process to 
> where HTTP/2 is handled.
> 
> If you believe though it is as simple as an extra item in the environ 
> dictionary, then it can be handled as a separate extension specification per 
> the URL above.

A side discussion on Twitter has noted that this exists:

https://w3c.github.io/preload/ 

This is already implemented by mod_h2, nghttp2 and H20 at least.

This is not my area so I don’t know for sure whether this fits the bill as to 
what is being described as ‘pushing’.

If as I understand it, this allows a WSGI application to return a Link header 
and mod_h2 in Apache then uses HTTP/2 push to deliver up those resources 
straight away.

If I am misunderstanding this, please let me know.

The only problem I do see with this right now if it does what is required, is 
that in mod_wsgi daemon mode, except for select headers such as Set-Cookie and 
WWW-Authenticate, response headers of the same name will be joined together. 
What I don’t know is if mod_h2/nghttp2 will handle where the value of a Link 
header is joined. It is probably going to be safer if I modify mod_wsgi and add 
Link to the white list of headers which aren’t joined together.

If this does solve the push issue, what is there in HTTP/2 then that one 
couldn’t do via the existing WSGI interface?

Graham


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 6 Jan 2016, at 09:48, Graham Dumpleton  wrote:
> 
> If this does solve the push issue, what is there in HTTP/2 then that one 
> couldn’t do via the existing WSGI interface?

Well, plenty, but none that we’d *want* to expose via WSGI with the possible 
exception of long-running bi-directional communications channels like 
Websockets, which you’ve already expressed a desire to expose in a different 
API. =)

Pushing via Link headers is not ideal because it delays the push until after 
the headers are ready to go, and there’s a tricky ordering concern here (RFC 
7540 points out that any PUSH_PROMISE frames should be sent before the response 
headers and body are sent, which means that we temporarily block the response 
that is ready to go from WSGI. This is a minor concern, but worth noting.)

However, I’m happy to say that Pushing via Link headers is the way to go if 
we’d rather not specify a WSGI-specific API for it.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Amber "Hawkie" Brown]

> On 4 Jan 2016, at 20:27, Cory Benfield  wrote:
> 
> All,
> 
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**
> 
> It’s a new year, and that means it’s time for another attempt to get WSGI 2.0 
> off the ground. Many of you may remember that we attempted to do this last 
> year with Rob Collins leading the charge, but unfortunately personal 
> commitments made it impossible for Rob to keep pushing that attempt forward.
> 
> Since then, the need for a revision of WSGI has become even more apparent. 
> Casual discussion on the web has indicated that application developers are 
> uncomfortable with the limitations of WSGI. These limitations are providing 
> an incentive for both application developers and server developers to take an 
> end-run around WSGI in an attempt to get a framework that is more suitable 
> for the modern web. A great example of the result of WSGI’s deficiencies is 
> Andrew Godwin’s channels work[0] for Django, which represents a paradigm 
> shift in application development that takes it far away from what WSGI is 
> today.
> 
> For this reason, I think we need to try again to get WSGI 2.0 off the ground. 
> But I don’t believe we can do this without getting broad consensus from the 
> developer community that a revision to WSGI is needed, and without 
> understanding what developers need from a new revision of WSGI. This should 
> take into account the prior discussions we’d had on this thread: however, I’m 
> also going to actively solicit feedback from some of the more notable WSGI 
> implementers, to ensure that whatever comes out of this SIG is something that 
> they would actually use.
> 
> This WG already had a list of requirements, which are as follows:
> 
> - Support servers speaking HTTP/1.x, HTTP/2 and Websockets (potentially all 
> on a single port).
> - Support graceful degradation for applications that can use HTTP/2 but still 
> support HTTP/1.x requests.
> - Graceful incremental adoption path - no upgrade-all-components requirement 
> baked into the design.
> - Support Python 2.7 and 3.x (where x is not yet discussed)
> - Support the existing ecosystem of containers (such as mod_wsgi) with the 
> new API. We want a clean, fast and approachable API, and we want to ensure 
> that its no less friendly to work with than WSGI, for all that it will expose 
> much more functionality.
> - Apps need to be able to tell what protocol is in use, and what optional 
> features are available. For instance, HTTP/2 PUSH PROMISE is an optional 
> feature that can be disabled by clients. Websockets needs to expose a socket 
> like object, and so on.
> - Support websockets
> - Support HTTP/2
> - Support HTTP/1.x (which may be just 'point at PEP-’.)
> - Continue to support lightweight shims being built on top such as 
> https://github.com/Pylons/webob/blob/master/webob/request.py
> 
> I believe that all of these requirements are up for grabs, and subject to 
> change and consensus discussion. In this thread, then, I’d like to hear from 
> people about these requirements and others. What do you believe WSGI 2.0 
> should do? Just as importantly, what do you believe it should not do? What 
> prior art should we take into account? Should we bother revising WSGI at all, 
> or should we let the wider application ecosystem pursue its own solutions à 
> la Django's channels? Should we simply adopt Andrew Godwin’s ASGI draft[1] on 
> which channels is based and call *that* WSGI 2.0?
> 
> Right now I want this to be very open. I’d like people to come up with a 
> broad statement listing what they believe should and should not be present in 
> WSGI. This first stage of the work is very general: I just want to get a 
> feeling for what the community believes is important. Once we’re done with 
> that, if the consensus is that this work is worth pursuing, I’ll come up with 
> an initial draft that we can start making concrete changes to.
> 
> In the short term, I’m going to keep this consultation open for **at least 
> two weeks**: that is, I will not start working on an initial draft PEP until 
> at least the **18th of January**. If you believe there are application or 
> server developers that should be involved in this discussion, please reach 
> out to them and point them to this list. I personally have CC’d some people 
> that I believe need to be involved in the discussion, but please reach out to 
> others as well.
> 
> I’d really love to come to the end of 2016 with a solid direction for the 
> future of web programming in Python. I’m looking forward to working with you 
> all on achieving that.
> 
> Thanks,
> 
> Cory
> 
> 
> [0]: https://channels.readthedocs.org/en/latest/
> [1]: https://channels.readthedocs.org/en/latest/asgi.html

Hi all! Due to some dubious life choices, I've decided to take part in this 
discussion!

I have very much a vested interest in async making sense in a new WSGI, and 
personally would go so far 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[André Malo]

* Graham Dumpleton wrote:

> > On 6 Jan 2016, at 12:13 AM, Benoit Chesneau 
> > wrote:
> >
> > So for me what should be WSGI 2? WSGI 2 should add against WSGI 1 the
> > following:
> >
> > - tell to the application it is actually an HTTP2 request (maybe
> > populating a wsgi.http2 true env)
>
> In CGI implementations you would for HTTP/1.1 already get:
>
> SERVER_PROTOCOL: 'HTTP/1.1’
>
> Under HTTP/2 when I tested some time back, I recollect it came through as
> one would assume would be expected:
>
> SERVER_PROTOCOL: ‘HTTP/2’
>
> Is there any reason that this existing CGI variable wouldn’t be
> sufficient for this purpose?

The CGI spec. Although only informational, the definition of SERVER_PROTOCOL 
is kinda weird.

https://www.ietf.org/rfc/rfc3875

Cheers,
-- 
"Solides und umfangreiches Buch"
  -- aus einer Rezension


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Damjan Georgievski]

> All,
>
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**
...
> - Support HTTP/2

I've read a bit about HTTP/2 PUSH functionality, and it seems to me
that it can (and probably would be)
supported in web servers similar to how X-SendFile etc work, ie by
sending a specially crafted response header.
I'm suspicious we'll need any special WSGI support for that. Let's
first see what nginx does about it.

-- 
damjan
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 6 Jan 2016, at 10:19 PM, Cory Benfield  wrote:
> 
> 
>> On 6 Jan 2016, at 09:48, Graham Dumpleton  wrote:
>> 
>> If this does solve the push issue, what is there in HTTP/2 then that one 
>> couldn’t do via the existing WSGI interface?
> 
> Well, plenty, but none that we’d *want* to expose via WSGI with the possible 
> exception of long-running bi-directional communications channels like 
> Websockets, which you’ve already expressed a desire to expose in a different 
> API. =)

Can you elaborate more on the ‘plenty’ part.

This was the issue in the past. People appeared to want access to everything. 
Thus why the belief you may as well allow them a separate API purpose built for 
it. Maybe people are becoming more realistic in expectations now as to what 
they really need for a typical web application as HTTP/2 is better understood.

> Pushing via Link headers is not ideal because it delays the push until after 
> the headers are ready to go, and there’s a tricky ordering concern here (RFC 
> 7540 points out that any PUSH_PROMISE frames should be sent before the 
> response headers and body are sent, which means that we temporarily block the 
> response that is ready to go from WSGI. This is a minor concern, but worth 
> noting.)
> 
> However, I’m happy to say that Pushing via Link headers is the way to go if 
> we’d rather not specify a WSGI-specific API for it.


It appears that Link could at least be a fallback.

The idea of a separate callable to push resources in WSGI environ could still 
be dealt with as an extension specification and so not need changes to the WSGI 
specification itself. Worst case is all that callable does is add Link headers 
to the response. This would likely have to be the case in Apache with mod_h2 as 
wouldn’t expect handlers to have direct access to an API to do it any other 
way, plus in mod_wsgi daemon mode is in the wrong process to access any API.

With the possibility that Link header would be a mechanism for use by such a 
callable, then any calling arguments for the callable should perhaps be 
modelled on what is possible via the Link header. I could be talking nonsense 
on that point as I have no idea how server push works in HTTP/2 protocol.

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 5 Jan 2016, at 00:12, Graham Dumpleton  wrote:
> 
> 
>> On 4 Jan 2016, at 11:27 PM, Cory Benfield > > wrote:
>> 
>> All,
>> 
>> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
>> it at all?**
>> 
>> It’s a new year, and that means it’s time for another attempt to get WSGI 
>> 2.0 off the ground. Many of you may remember that we attempted to do this 
>> last year with Rob Collins leading the charge, but unfortunately personal 
>> commitments made it impossible for Rob to keep pushing that attempt forward.
> 
> Although you call this round 2, it isn’t really. Robert’s effort was not the 
> first time someone has pushed a WSGI 2.0 variant. So this is more like being 
> about round 5 or 6.
> 
> In part because of those repeated attempts by people to propose something and 
> label it as WSGI 2.0, I am very cool on reusing the WSGI 2.0 moniker. You 
> will find little or no mention of ‘WSGI 2.0’ as a label in:
> 
> https://github.com/python-web-sig/wsgi-ng 
> 
> 
> That is probably somewhat due to my grumbling about the use of ‘WSGI 2.0’ 
> back then.
> 
> Time has moved on and so the bad feelings and memories associated with the 
> ‘WSGI 2.0’ label due to early failed efforts have faded, but I would still 
> suggest avoiding the label ‘WSGI 2.0’ if at all possible.

Thanks for that feedback. Consider WSGI 2.0 a catch-all name for the purposes 
of this specific discussion (the “what do we want WSGI to be going forward” 
one). As you’ve suggested here, it’s entirely possible that the result of this 
discussion will be several PEPs/APIs, or none at all, and it’s entirely 
possible that none of them would be called WSGI 2.0.

> My general feeling is that if any proposed changes to the existing WSGI (PEP 
> ) specification cannot be technically implemented on all existing WSGI 
> server/adapter implementations that any new specification should not still be 
> called WSGI.
> 
> In other words, even if many of these implementations may not be used much 
> any more, it must be able to work, without needing to mark things as 
> optional, on CGI, FASTCGI, SCGI, mod_wsgi, gunicorn, uWSGI, Waitress, etc etc.
> 
> This is purely to avoid the confusion whereby implementations cannot or 
> choose not to implement any new specification. The last thing any WSGI server 
> author wants is having to deal with a constant stream of questions and bug 
> reports about not supporting an updated specification where technically it 
> was never going to be possible. We have some obligation not to inflict this 
> on what are, in nearly all cases, volunteers in the Open Source world who 
> work on these things in their spare time and who are not doing it as part of 
> their paid employment.

Can I clarify this requirement a bit? Are you wanting to say that any future 
version of WSGI must be entirely compatible with PEP : that is, may not 
introduce optional features or change existing behaviour, only clarify? Please 
don’t mistake this for me challenging the idea: I’m wanting to get a good 
understanding of what you’re suggesting with this, not agreeing or disagreeing 
at this stage.

> For example, mod_wsgi already supports HTTP/2 by virtue of the fact that the 
> mod_h2 module in Apache exists. The existing internal APIs of Apache and how 
> mod_wsgi uses those means that HTTP/2 bridges into the WSGI world with no 
> code changes to mod_wsgi.

Agreed. If all we want is to keep the request/response cycle intact, then WSGI 
supports H2 already. One possibility that has already been suggested here would 
be to define a HTTP/2 extension to WSGI, advertised in the environ dict, that 
allows the application to signal pushes to the server. This would be a fairly 
simple extension to write and implement.

> They are therefore two different APIs and so why WebSocket should be dealt 
> with in a separate specification and not carry the WSGI label at all. A 
> specific WSGI server could still support the new WebSocket API, but purely 
> because it decides to support both in the same process. Not because the 
> WebSocket API makes use of the WSGI specification.

That’s reasonable: I’d be happy to have websocket support either be a WSGI 
extension or, as you suggest here, a wholly new API. One difficulty with 
creating a new API from whole cloth is encouraging server authors to support 
it, but it’s certainly possible to do. I’d like to hear back from the uWSGI, 
gunicorn, and Twisted folks in addition to yourself about whether they’d be 
interested in implementing such a non-WSGI API.

>> - Graceful incremental adoption path - no upgrade-all-components requirement 
>> baked into the design.
> 
> It is hard to see what you expectations are here.
> 
> Prior attempts to force ASYNC into WSGI, and in some respects WebSockets 
> through forcing raw fd access have not been practical. WSGI 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 6 Jan 2016, at 9:27 AM, chris.d...@gmail.com wrote:
> 
> On Wed, 6 Jan 2016, Graham Dumpleton wrote:
> 
>> 
>>> On 6 Jan 2016, at 12:09 AM, chris.d...@gmail.com wrote:
>>> 
>>> As someone who writes their WSGI applications as functions that take
>>> `start_response` and `environ` and doesn't bother with much
>>> framework the things I would like to see in a minor revision to WSGI
>>> are:
>>> 
>>> * A consistent way to access the raw un-decoded request URI. This is
>>> so I can reconstruct a realistic `PATH_INFO` that has not been
>>> subjected to destructive handling by the server (e.g. apache
>>> messing with `%2F`) before continuing on to a route dispatcher.
>> 
>> This is already available in some servers by way of the REQUEST_URI value.
> 
> Yes, and in others (as mentioned by Benoit) as RAW_URI. One
> ("consistent") way would be better.
> 
> [Lots of good information about the challenges associated with using
> that information to do anything useful, deleted.]
> 
> What I've done in one app is this:
> https://github.com/tiddlyweb/tiddlyweb/blob/cc6b67d2855ea4d8d908f1a3e58db0dce7e8d138/tiddlyweb/web/serve.py#L119
> 
> Despite the fact that that is not strictly correct, it does mostly work
> for the situation described in the comment and the context of that
> app. One of the things I want from a light rev of WSGI is not to have
> to jump through those hoops.
> 
> It may be that's not feasible but I reckon we're at the wishing
> stage of the discussion.

Yeah, that code would have problems.

One other thing just remembered is that technically it is allowed that the path 
part of the request line can actually be a URI.

GET http://hostname/a/b/c HTTP/1.0

This would yield:

REQUEST_URI: 'http://hostname/a/b/c' 
SCRIPT_NAME: ‘'
PATH_INFO: '/a/b/c’

Obviously I didn’t even mention the % encoding issues as part of SCRIPT_NAME 
part as you are obviously aware of those being an issue in PATH_INFO at least.

Lots of fun.

Graham___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 6 Jan 2016, at 12:09 AM, chris.d...@gmail.com wrote:
> 
> As someone who writes their WSGI applications as functions that take
> `start_response` and `environ` and doesn't bother with much
> framework the things I would like to see in a minor revision to WSGI
> are:
> 
> * A consistent way to access the raw un-decoded request URI. This is
>  so I can reconstruct a realistic `PATH_INFO` that has not been
>  subjected to destructive handling by the server (e.g. apache
>  messing with `%2F`) before continuing on to a route dispatcher.

This is already available in some servers by way of the REQUEST_URI value.

This is the original first line of any HTTP request and can be split apart to 
get the path.

The problem is that you cannot easily use it unless you want to replicate 
normalisations that the underlying server may do.

The key problem is working out where SCRIPT_NAME ends and PATH_INFO starts with 
the original path given in REQUEST_URI.

Sure if you only deal with a web application mounted at the root of the host it 
is easier because SCRIPT_NAME would be empty, but when mounted at a sub URL it 
gets trickier.

This is because a web server will eliminate things like repeating slashes in 
the part of the path that may match the mount point (sub url) for the web 
application. The sub url here could be dictated by what is defined in a 
configuration file, or could instead be due to matching against a file system 
path.

Further, the web server will eliminate attempts at relative directory traversal 
using ‘..’ and ‘.’.

So an original path may be something like:

/a/b//c/../d/.//e/../f/g/h

If the mount point was ‘/a/b/d’, then that is what gets passed through 
SCRIPT_NAME.

Now if you instead go to the raw path you would need to replicate all the 
normalisations. Only then could you maybe based on length of SCRIPT_NAME, 
number of component parts, or actual components in the path, try and calculate 
where SCRIPT_NAME ended and PATH_INFO started in the raw path.

This will still all fail if a web server does internal rewrites though, as the 
final SCRIPT_NAME may not even match the raw path, although at that point URL 
reconstruction can be a problem as well if what the application is given by way 
of the rewrite isn’t a public path.

I have only looked at SCRIPT_NAME. Even in PATH_INFO servers will apply same 
sort of normalisations.

So even this isn’t so simple to do properly if you want to go back and do it 
yourself using the raw path.

I have never seen anyone trying to extract repeating slashes intact out of a 
raw path even attempt to do it properly. They tend to assume that the raw path 
is pure and doesn’t have stuff in it which needs to be normalised and that 
rewrites aren’t occurring. As a result they assume that they can just strip 
number of characters off raw path based on length of SCRIPT_NAME passed 
through. This will be fragile though if the raw path isn’t pure.

Graham___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[chris . dent]

On Wed, 6 Jan 2016, Graham Dumpleton wrote:




On 6 Jan 2016, at 12:09 AM, chris.d...@gmail.com wrote:

As someone who writes their WSGI applications as functions that take
`start_response` and `environ` and doesn't bother with much
framework the things I would like to see in a minor revision to WSGI
are:

* A consistent way to access the raw un-decoded request URI. This is
 so I can reconstruct a realistic `PATH_INFO` that has not been
 subjected to destructive handling by the server (e.g. apache
 messing with `%2F`) before continuing on to a route dispatcher.


This is already available in some servers by way of the REQUEST_URI value.


Yes, and in others (as mentioned by Benoit) as RAW_URI. One
("consistent") way would be better.

[Lots of good information about the challenges associated with using
that information to do anything useful, deleted.]

What I've done in one app is this:
https://github.com/tiddlyweb/tiddlyweb/blob/cc6b67d2855ea4d8d908f1a3e58db0dce7e8d138/tiddlyweb/web/serve.py#L119

Despite the fact that that is not strictly correct, it does mostly work
for the situation described in the comment and the context of that
app. One of the things I want from a light rev of WSGI is not to have
to jump through those hoops.

It may be that's not feasible but I reckon we're at the wishing
stage of the discussion.

--
Chris Dent   http://burningchrome.com/
[...]
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 6 Jan 2016, at 9:19 AM, Graham Dumpleton  
> wrote:
> 
>> On 6 Jan 2016, at 12:09 AM, chris.d...@gmail.com 
>>  wrote:
>> 
>> As someone who writes their WSGI applications as functions that take
>> `start_response` and `environ` and doesn't bother with much
>> framework the things I would like to see in a minor revision to WSGI
>> are:
>> 
>> * A consistent way to access the raw un-decoded request URI. This is
>>  so I can reconstruct a realistic `PATH_INFO` that has not been
>>  subjected to destructive handling by the server (e.g. apache
>>  messing with `%2F`) before continuing on to a route dispatcher.
> 
> This is already available in some servers by way of the REQUEST_URI value.
> 
> This is the original first line of any HTTP request and can be split apart to 
> get the path.

Whoops. My foggy memory. REQUEST_URI is only raw path part, not the whole 
request line with method, protocol and path.

Graham___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 5 Jan 2016, at 8:40 PM, Cory Benfield  wrote:
> 
> 
>> On 5 Jan 2016, at 00:12, Graham Dumpleton > > wrote:
>> 
>> 
>>> On 4 Jan 2016, at 11:27 PM, Cory Benfield >> > wrote:
>>> 
>>> All,
>>> 
>>> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we 
>>> do it at all?**
>>> 
>>> It’s a new year, and that means it’s time for another attempt to get WSGI 
>>> 2.0 off the ground. Many of you may remember that we attempted to do this 
>>> last year with Rob Collins leading the charge, but unfortunately personal 
>>> commitments made it impossible for Rob to keep pushing that attempt forward.
>> 
>> Although you call this round 2, it isn’t really. Robert’s effort was not the 
>> first time someone has pushed a WSGI 2.0 variant. So this is more like being 
>> about round 5 or 6.
>> 
>> In part because of those repeated attempts by people to propose something 
>> and label it as WSGI 2.0, I am very cool on reusing the WSGI 2.0 moniker. 
>> You will find little or no mention of ‘WSGI 2.0’ as a label in:
>> 
>> https://github.com/python-web-sig/wsgi-ng 
>> 
>> 
>> That is probably somewhat due to my grumbling about the use of ‘WSGI 2.0’ 
>> back then.
>> 
>> Time has moved on and so the bad feelings and memories associated with the 
>> ‘WSGI 2.0’ label due to early failed efforts have faded, but I would still 
>> suggest avoiding the label ‘WSGI 2.0’ if at all possible.
> 
> Thanks for that feedback. Consider WSGI 2.0 a catch-all name for the purposes 
> of this specific discussion (the “what do we want WSGI to be going forward” 
> one). As you’ve suggested here, it’s entirely possible that the result of 
> this discussion will be several PEPs/APIs, or none at all, and it’s entirely 
> possible that none of them would be called WSGI 2.0.
> 
>> My general feeling is that if any proposed changes to the existing WSGI (PEP 
>> ) specification cannot be technically implemented on all existing WSGI 
>> server/adapter implementations that any new specification should not still 
>> be called WSGI.
>> 
>> In other words, even if many of these implementations may not be used much 
>> any more, it must be able to work, without needing to mark things as 
>> optional, on CGI, FASTCGI, SCGI, mod_wsgi, gunicorn, uWSGI, Waitress, etc 
>> etc.
>> 
>> This is purely to avoid the confusion whereby implementations cannot or 
>> choose not to implement any new specification. The last thing any WSGI 
>> server author wants is having to deal with a constant stream of questions 
>> and bug reports about not supporting an updated specification where 
>> technically it was never going to be possible. We have some obligation not 
>> to inflict this on what are, in nearly all cases, volunteers in the Open 
>> Source world who work on these things in their spare time and who are not 
>> doing it as part of their paid employment.
> 
> Can I clarify this requirement a bit? Are you wanting to say that any future 
> version of WSGI must be entirely compatible with PEP : that is, may not 
> introduce optional features or change existing behaviour, only clarify? 
> Please don’t mistake this for me challenging the idea: I’m wanting to get a 
> good understanding of what you’re suggesting with this, not agreeing or 
> disagreeing at this stage.

I am saying that any update to the WSGI specification should still be able to 
be implemented using any of the existing technologies that can already 
implement WSGI.

I would see it as just causing problems to bring out an updated WSGI 
specification which couldn’t be implemented on top of CGI, FASTCGI, SCGI or 
even mod_wsgi.

Further, it does really still need to be compatible with the existing 
specifications/applications. Changes I am talking about are clarifications or 
suggesting better ways of doing stuff like wsgi.file_wrapper to avoid known 
problems or to eliminate the use of assumptions about how something works.

If a framework or application is made dependent on some new aspect of the WSGI 
specification which has no fallback because the specification was changed to 
not really be compatible with prior versions in some way then it is me as the 
author of a WSGI server who would have to endure the constant questions of why 
that framework or application doesn’t now work on mod_wsgi if the changes 
couldn’t be supported.

People will not care what version of WSGI the framework or application adhered 
to. Their attitude will be that it supports WSGI and since mod_wsgi says it 
supports WSGI it must work, but since it doesn’t mod_wsgi must be broken. They 
will ignore version vagaries. 

So I am being selfish in not wanting to have to put up with more users 
complaining about stuff. :-)

As for optional stuff, if they are truly extensions and can work through 
stuffing things in the WSGI 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Armin Ronacher]

Hi,

I just want to reply to this because I think many people seem to be 
missing why things are done in a certain way.  Especially if the appear 
to be odd.


On 05/01/2016 12:26, Cory Benfield wrote:

1. WSGI is prone to header injection vulnerabilities issues by
designdue to the conversion of HTTP headers to CGI-style environment
variables: if the server doesn’t specifically prevent it, X-Foo and
X_Foo both become HTTP_X_Foo. I don’t believe it’s a good choice to
destructively encode headers, expect applications to undo the damage
somehow, and introduce security vulnerabilities in the process. If
mimicking CGI is still considered a must-have — 1% of current Python web
programmers may have heard about it, most of them from PEP  — then
that burden should be pushed onto the server, not the application.
Headers always will have to be encoded destructively if you want any 
form of generic processing.  We need header joining, we need to 
normalize the keys already at least to the extend of the HTTP 
specification.  I'm happy to not perform the conversion of dashes to 
underscores but you will work in environments where this conversion was 
already done so the spec will need to deal with that case anyways.


The WSGI spec currently also does not sufficiently explain how to join 
headers.  In particular the cookie header was written without header 
joining in mind which is why it needs to be joined differently than all 
other headers.  Header joining also comes up as a big topic in HTTP 2

so the spec will need to cover this.


2. More generally, I fail to see how mixing HTTP headers,
server-related inputs, and environment variables in a dict adds
values. It prevents iterating on each collection separately. It only
makes sense if not offering more features than CGI is a design goal;
in that case, this discussion doesn’t serve a purpose anyway. It
would be nicer and possibly more secure if the application received
separately:
I think this is largely a nice to have, not something that has any 
overall benefits.  I rather just clean up the actual stupid things such 
as CONTENT_TYPE and CONTENT_LENGTH which cause a lot more real world 
friction than just the names of keys in general.  This really should not 
turn into meaningless bikeshedding about what information should be 
called.  Also consider how much code out there already assumes CGI/WSGI 
variables so any move off that really should have good reasons or we all 
will just waste enormous amounts just to transpose between the two 
representations.



a. Configuration information, which servers could read from
environment variables by default for backwards compatibility, but could
also get through more secure channels and restrict to what the
application needs in order to better isolate it from the entire OS.
What WSGI traditionally lacked was a setup phase where data could be 
passed to the application that was server specific but not request 
bound.  For instance there is no reason an application cannot get hold 
of wsgi.errors before a request comes in.  I would like to see this 
fixed in a new specification.



3. Stop pretending that HTTP is a unicode protocol, or at least stop
ignoring reality when doing so. WSGI enforces ISO-8859-1-decoded str
objects in the environ, which is just wrong. It’s all the more a
surprising choice since this change was driven by Python 3, that UTF-8
is the correct choice, and that Python 3 defaults to UTF-8. Django has
to re-encode and re-decode before doing anything with HTTP headers:
I agree with this but you will have to have that fight with others.  I 
said many times before that values should never have been unicode values 
in the first place but certain decisions in the Python 3 standard 
library at the time prevented that.  In particular until 3.2 or so it 
was impossible to parse byte URLs.



5. Improve request / response length handling and connection closure.
Armin and Graham have talked about in the past and know the topic
better than I do. There’s also a rejected PEP by Armin which made
sense to me.
I think last time I discussed that with Graham it was not clear what the 
solution is in the context of WSGI.  The idea that there is a 
content-length is laughable in the context of a real application where 
the server is performing conversions on the input and output stream.  We 
would need many more than just one content length and an automatically 
terminated input stream.


However at that point you will quickly realize that you can't have it 
both ways and you either have a WSGI like protocol, or raw access to 
sockets but certainly not both.  This topic has caused a lot of 
bikeshedding in the past and I fail to see how it will be differently 
this time.


My current thinking is that the most realistic approach to most of those 
problems will be the concept of framing on both the input and output 
side.  That's somewhat compatible with both chunked transports well as 
websockets.  But if we do go down 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

Forwarding this message from the django-developers list.

Hi Cory,

I’m not subscribed to web-sig but I read the discussion there. Feel free to 
forward my answer to the group if you think it’s useful.

I have roughly the same convictions as Graham Dumpleton. If you want to support 
HTTP/2 and WebSockets, don’t start with design decisions anchored in CGI. 
Figure out what a simple and flexible API for these new protocols would be, 
specify it, implement it, and make sure it degrades gracefully to HTTP/1. You 
may be able to channel most of the communication through a single generator, 
but it’s unclear to me that this will be the most convenient design.

If you want to improve WSGI, here’s a list of mistakes or shortcomings in PEP 
 that you can take a stab at. There’s a general theme: for a specification 
that looks at the future, I believe that making modern PaaS-based deployments 
secure by default matters more than not implementing anything beyond what’s 
available in legacy CGI-based deployments.

1. WSGI is prone to header injection vulnerabilities issues by design due to 
the conversion of HTTP headers to CGI-style environment variables: if the 
server doesn’t specifically prevent it, X-Foo and X_Foo both become HTTP_X_Foo. 
I don’t believe it’s a good choice to destructively encode headers, expect 
applications to undo the damage somehow, and introduce security vulnerabilities 
in the process. If mimicking CGI is still considered a must-have — 1% of 
current Python web programmers may have heard about it, most of them from PEP 
 — then that burden should be pushed onto the server, not the application.

2. More generally, I fail to see how mixing HTTP headers, server-related 
inputs, and environment variables in a dict adds values. It prevents iterating 
on each collection separately. It only makes sense if not offering more 
features than CGI is a design goal; in that case, this discussion doesn’t serve 
a purpose anyway. It would be nicer and possibly more secure if the application 
received separately:

a. Configuration information, which servers could read from environment 
variables by default for backwards compatibility, but could also get through 
more secure channels and restrict to what the application needs in order to 
better isolate it from the entire OS.
b. Server APIs mandated by the spec, per request.
c. HTTP headers, per request.

3. Stop pretending that HTTP is a unicode protocol, or at least stop ignoring 
reality when doing so. WSGI enforces ISO-8859-1-decoded str objects in the 
environ, which is just wrong. It’s all the more a surprising choice since this 
change was driven by Python 3, that UTF-8 is the correct choice, and that 
Python 3 defaults to UTF-8. Django has to re-encode and re-decode before doing 
anything with HTTP headers: 
https://github.com/django/django/blob/d5b90c8e120687863c1d41cf92a4cdb11413ad7f/django/core/handlers/wsgi.py#L231-L253

4. Normalize the way to tell the application about the original protocol, IP 
address and port. When dev and ops responsibilities are separate, this is 
clearly an ops responsibility, but due to the lack of standardization devs end 
up dealing with this problem in custom middleware, when they do it at all. 
Everyone keeps getting it wrong, which introduces security vulnerabilities. 
Also it always breaks silently on infrastructure changes.

5. Improve request / response length handling and connection closure. Armin and 
Graham have talked about in the past and know the topic better than I do. 
There’s also a rejected PEP by Armin which made sense to me.

As you can see from these comments, I don’t quite share the design choices that 
led to WSGI as it currently stands. I think it will be easier to build a new 
standard than evolve the current one.

I hope this helps!

Aymeric


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 5 Jan 2016, at 10:57 PM, Graham Dumpleton  
> wrote:
> 
> 
>> On 5 Jan 2016, at 10:26 PM, Cory Benfield > > wrote:
>> 
>> Forwarding this message from the django-developers list.
>> 
>> Hi Cory,
>> 
>> I’m not subscribed to web-sig but I read the discussion there. Feel free to 
>> forward my answer to the group if you think it’s useful.
>> 
>> I have roughly the same convictions as Graham Dumpleton. If you want to 
>> support HTTP/2 and WebSockets, don’t start with design decisions anchored in 
>> CGI. Figure out what a simple and flexible API for these new protocols would 
>> be, specify it, implement it, and make sure it degrades gracefully to 
>> HTTP/1. You may be able to channel most of the communication through a 
>> single generator, but it’s unclear to me that this will be the most 
>> convenient design.
>> 
>> If you want to improve WSGI, here’s a list of mistakes or shortcomings in 
>> PEP  that you can take a stab at. There’s a general theme: for a 
>> specification that looks at the future, I believe that making modern 
>> PaaS-based deployments secure by default matters more than not implementing 
>> anything beyond what’s available in legacy CGI-based deployments.
>> 
>> 1. WSGI is prone to header injection vulnerabilities issues by design due to 
>> the conversion of HTTP headers to CGI-style environment variables: if the 
>> server doesn’t specifically prevent it, X-Foo and X_Foo both become 
>> HTTP_X_Foo. I don’t believe it’s a good choice to destructively encode 
>> headers, expect applications to undo the damage somehow, and introduce 
>> security vulnerabilities in the process. If mimicking CGI is still 
>> considered a must-have — 1% of current Python web programmers may have heard 
>> about it, most of them from PEP  — then that burden should be pushed 
>> onto the server, not the application.
> 
> FWIW, Apache 2.4 will discard headers which would use underscore, as well as 
> many other characters. Basically it probably only accepts alphanumeric and 
> ‘-‘ in original name.
> 
> In mod_wsgi, it does the same thing, even for Apache 2.2 where it wasn’t done.
> 
> So with mod_wsgi at least you are safe. Or at least if not still using some 
> ancient mod_wsgi version. (Death to LTS Linux versions and out of date 
> packages) :-)
> 
> The nginx server if used as a front end and where it is populating CGI like 
> variables for passing to a builtin module such as uWSGI will also I believe 
> discard headers which don’t match that requirement as well.
> 
> I can’t remember if gunicorn was updated to do something similar, or whether 
> when uWSGI isn’t used behind nginx via its uwsgi protocol, but instead 
> listens publicly via HTTP whether it does it either. 


I should clarify a point here. Apache 2.4 will discard the headers at the point 
of converting them to a CGI like environment when a handler asks for a CGI like 
set of variables. Raw headers will always be passed through as they were.

Graham___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 5 Jan 2016, at 10:26 PM, Cory Benfield  wrote:
> 
> Forwarding this message from the django-developers list.
> 
> Hi Cory,
> 
> I’m not subscribed to web-sig but I read the discussion there. Feel free to 
> forward my answer to the group if you think it’s useful.
> 
> I have roughly the same convictions as Graham Dumpleton. If you want to 
> support HTTP/2 and WebSockets, don’t start with design decisions anchored in 
> CGI. Figure out what a simple and flexible API for these new protocols would 
> be, specify it, implement it, and make sure it degrades gracefully to HTTP/1. 
> You may be able to channel most of the communication through a single 
> generator, but it’s unclear to me that this will be the most convenient 
> design.
> 
> If you want to improve WSGI, here’s a list of mistakes or shortcomings in PEP 
>  that you can take a stab at. There’s a general theme: for a 
> specification that looks at the future, I believe that making modern 
> PaaS-based deployments secure by default matters more than not implementing 
> anything beyond what’s available in legacy CGI-based deployments.
> 
> 1. WSGI is prone to header injection vulnerabilities issues by design due to 
> the conversion of HTTP headers to CGI-style environment variables: if the 
> server doesn’t specifically prevent it, X-Foo and X_Foo both become 
> HTTP_X_Foo. I don’t believe it’s a good choice to destructively encode 
> headers, expect applications to undo the damage somehow, and introduce 
> security vulnerabilities in the process. If mimicking CGI is still considered 
> a must-have — 1% of current Python web programmers may have heard about it, 
> most of them from PEP  — then that burden should be pushed onto the 
> server, not the application.

FWIW, Apache 2.4 will discard headers which would use underscore, as well as 
many other characters. Basically it probably only accepts alphanumeric and ‘-‘ 
in original name.

In mod_wsgi, it does the same thing, even for Apache 2.2 where it wasn’t done.

So with mod_wsgi at least you are safe. Or at least if not still using some 
ancient mod_wsgi version. (Death to LTS Linux versions and out of date 
packages) :-)

The nginx server if used as a front end and where it is populating CGI like 
variables for passing to a builtin module such as uWSGI will also I believe 
discard headers which don’t match that requirement as well.

I can’t remember if gunicorn was updated to do something similar, or whether 
when uWSGI isn’t used behind nginx via its uwsgi protocol, but instead listens 
publicly via HTTP whether it does it either. 

> 2. More generally, I fail to see how mixing HTTP headers, server-related 
> inputs, and environment variables in a dict adds values. It prevents 
> iterating on each collection separately. It only makes sense if not offering 
> more features than CGI is a design goal; in that case, this discussion 
> doesn’t serve a purpose anyway. It would be nicer and possibly more secure if 
> the application received separately:
> 
> a. Configuration information, which servers could read from environment 
> variables by default for backwards compatibility, but could also get through 
> more secure channels and restrict to what the application needs in order to 
> better isolate it from the entire OS.

I have always had a bit of a beef with the way that the use of environment 
variables for configuration was promoted by the 12 factor manifesto. It grew 
out of how a specific hosting service did things and ignored that various web 
servers used configuration files instead or did things in other ways. Of course 
the hosting service made it difficult to impossible to use some of those 
traditional web servers, so they were safe in their narrow view of things.

Anyway, if environment variables were used where appropriate and with an 
intermediate mapping layer within Python web applications that would have been 
fine. The problem was that you started to see direct lookup of environment 
variables deep in code bases. So people wedded themselves to use of environment 
variables.

The more sensible thing to do would have been to use an intermediate Python 
module/package providing an abstraction layer for getting configuration. Code 
would then use that. The configuration layer could then look up environment 
variables or use other means to get configuration, such as from more 
traditional configuration files, or pulling it done from configuration servers.

As far as I know there is no good Python package out there which serves as such 
a intermediary configuration system which could be plugged into any application 
and which doesn’t carry a huge amount of baggage. Would love to hear about one 
if it exists.

> b. Server APIs mandated by the spec, per request.
> c. HTTP headers, per request.
> 
> 3. Stop pretending that HTTP is a unicode protocol, or at least stop ignoring 
> reality when doing so. WSGI enforces ISO-8859-1-decoded str objects in the 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Luke Plant]

Just to add my 2c - as another Django developer, I agree completely with 
Aymeric here. My own experience was that the HTTP handling done by WSGI 
(especially URL handing, HTTP header mangling, os.environ as a 
destination - all due to CGI compatibility - and semi-broken unicode 
handling) only made things harder for us. We would much rather have 
dealt with raw streams of bytes and done all HTTP parsing ourselves.


Like Graham said, for HTTP/2 let's ignore the history of WSGI and start 
from scratch with a API that actually serves us well.


Regards,

Luke


On 05/01/16 11:26, Cory Benfield wrote:

Forwarding this message from the django-developers list.

Hi Cory,

I’m not subscribed to web-sig but I read the discussion there. Feel free to 
forward my answer to the group if you think it’s useful.

I have roughly the same convictions as Graham Dumpleton. If you want to support 
HTTP/2 and WebSockets, don’t start with design decisions anchored in CGI. 
Figure out what a simple and flexible API for these new protocols would be, 
specify it, implement it, and make sure it degrades gracefully to HTTP/1. You 
may be able to channel most of the communication through a single generator, 
but it’s unclear to me that this will be the most convenient design.

If you want to improve WSGI, here’s a list of mistakes or shortcomings in PEP 
 that you can take a stab at. There’s a general theme: for a specification 
that looks at the future, I believe that making modern PaaS-based deployments 
secure by default matters more than not implementing anything beyond what’s 
available in legacy CGI-based deployments.

1. WSGI is prone to header injection vulnerabilities issues by design due to 
the conversion of HTTP headers to CGI-style environment variables: if the 
server doesn’t specifically prevent it, X-Foo and X_Foo both become HTTP_X_Foo. 
I don’t believe it’s a good choice to destructively encode headers, expect 
applications to undo the damage somehow, and introduce security vulnerabilities 
in the process. If mimicking CGI is still considered a must-have — 1% of 
current Python web programmers may have heard about it, most of them from PEP 
 — then that burden should be pushed onto the server, not the application.

2. More generally, I fail to see how mixing HTTP headers, server-related 
inputs, and environment variables in a dict adds values. It prevents iterating 
on each collection separately. It only makes sense if not offering more 
features than CGI is a design goal; in that case, this discussion doesn’t serve 
a purpose anyway. It would be nicer and possibly more secure if the application 
received separately:

a. Configuration information, which servers could read from environment 
variables by default for backwards compatibility, but could also get through 
more secure channels and restrict to what the application needs in order to 
better isolate it from the entire OS.
b. Server APIs mandated by the spec, per request.
c. HTTP headers, per request.

3. Stop pretending that HTTP is a unicode protocol, or at least stop ignoring 
reality when doing so. WSGI enforces ISO-8859-1-decoded str objects in the 
environ, which is just wrong. It’s all the more a surprising choice since this 
change was driven by Python 3, that UTF-8 is the correct choice, and that 
Python 3 defaults to UTF-8. Django has to re-encode and re-decode before doing 
anything with HTTP headers: 
https://github.com/django/django/blob/d5b90c8e120687863c1d41cf92a4cdb11413ad7f/django/core/handlers/wsgi.py#L231-L253

4. Normalize the way to tell the application about the original protocol, IP 
address and port. When dev and ops responsibilities are separate, this is 
clearly an ops responsibility, but due to the lack of standardization devs end 
up dealing with this problem in custom middleware, when they do it at all. 
Everyone keeps getting it wrong, which introduces security vulnerabilities. 
Also it always breaks silently on infrastructure changes.

5. Improve request / response length handling and connection closure. Armin and 
Graham have talked about in the past and know the topic better than I do. 
There’s also a rejected PEP by Armin which made sense to me.

As you can see from these comments, I don’t quite share the design choices that 
led to WSGI as it currently stands. I think it will be easier to build a new 
standard than evolve the current one.

I hope this helps!

Aymeric


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/l.plant.98%40cantab.net


--
"Trouble: Luck can't last a lifetime, unless you die young."
(despair.com)

Luke Plant || http://lukeplant.me.uk/

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Armin Ronacher]

Hi,

On 05/01/2016 13:09, Luke Plant wrote:

Just to add my 2c - as another Django developer, I agree completely with
Aymeric here. My own experience was that the HTTP handling done by WSGI
(especially URL handing, HTTP header mangling, os.environ as a
destination - all due to CGI compatibility - and semi-broken unicode
handling) only made things harder for us. We would much rather have
dealt with raw streams of bytes and done all HTTP parsing ourselves.

Like Graham said, for HTTP/2 let's ignore the history of WSGI and start
from scratch with a API that actually serves us well.
Alright. I bite: if it would not be done that way you had different 
problems.  In particular a problem that comes up very often is that 
people want the PATH_INFO and SCRIPT_NAME to not be encoded.  That 
however completely breaks any form of routing you would want to do the 
moment they contain unicode characters.


I keep having the argument about PATH_INFO and the header semantics 
constantly with people and i'm absolutely convinced (from the theory 
behind it as well as playing around with ideas for PEP 444 a few years 
ago) that it only gets worse the moment you leave the WSGI territory too 
far.


Likewise I wonder how many people that ask for more low level access 
concerned themselves with chunked requests/responses, transport 
encodings and all the complexity that servers do for you.  Yes, quite a 
bit of this is broken in WSGI but would have been trivial to fix without 
throwing the whole specification into the toilette :)



Regards,
Armin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Graham Dumpleton]

> On 4 Jan 2016, at 11:27 PM, Cory Benfield  wrote:
> 
> All,
> 
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**
> 
> It’s a new year, and that means it’s time for another attempt to get WSGI 2.0 
> off the ground. Many of you may remember that we attempted to do this last 
> year with Rob Collins leading the charge, but unfortunately personal 
> commitments made it impossible for Rob to keep pushing that attempt forward.

Although you call this round 2, it isn’t really. Robert’s effort was not the 
first time someone has pushed a WSGI 2.0 variant. So this is more like being 
about round 5 or 6.

In part because of those repeated attempts by people to propose something and 
label it as WSGI 2.0, I am very cool on reusing the WSGI 2.0 moniker. You will 
find little or no mention of ‘WSGI 2.0’ as a label in:

https://github.com/python-web-sig/wsgi-ng 


That is probably somewhat due to my grumbling about the use of ‘WSGI 2.0’ back 
then.

Time has moved on and so the bad feelings and memories associated with the 
‘WSGI 2.0’ label due to early failed efforts have faded, but I would still 
suggest avoiding the label ‘WSGI 2.0’ if at all possible.

My general feeling is that if any proposed changes to the existing WSGI (PEP 
) specification cannot be technically implemented on all existing WSGI 
server/adapter implementations that any new specification should not still be 
called WSGI.

In other words, even if many of these implementations may not be used much any 
more, it must be able to work, without needing to mark things as optional, on 
CGI, FASTCGI, SCGI, mod_wsgi, gunicorn, uWSGI, Waitress, etc etc.

This is purely to avoid the confusion whereby implementations cannot or choose 
not to implement any new specification. The last thing any WSGI server author 
wants is having to deal with a constant stream of questions and bug reports 
about not supporting an updated specification where technically it was never 
going to be possible. We have some obligation not to inflict this on what are, 
in nearly all cases, volunteers in the Open Source world who work on these 
things in their spare time and who are not doing it as part of their paid 
employment.

> Since then, the need for a revision of WSGI has become even more apparent. 
> Casual discussion on the web has indicated that application developers are 
> uncomfortable with the limitations of WSGI. These limitations are providing 
> an incentive for both application developers and server developers to take an 
> end-run around WSGI in an attempt to get a framework that is more suitable 
> for the modern web. A great example of the result of WSGI’s deficiencies is 
> Andrew Godwin’s channels work[0] for Django, which represents a paradigm 
> shift in application development that takes it far away from what WSGI is 
> today.
> 
> For this reason, I think we need to try again to get WSGI 2.0 off the ground. 
> But I don’t believe we can do this without getting broad consensus from the 
> developer community that a revision to WSGI is needed, and without 
> understanding what developers need from a new revision of WSGI. This should 
> take into account the prior discussions we’d had on this thread: however, I’m 
> also going to actively solicit feedback from some of the more notable WSGI 
> implementers, to ensure that whatever comes out of this SIG is something that 
> they would actually use.
> 
> This WG already had a list of requirements, which are as follows:
> 
> - Support servers speaking HTTP/1.x, HTTP/2 and Websockets (potentially all 
> on a single port).

Any support for implementing WebSockets should though be seen as a separate 
requirement to implementing HTTP/2.

A specific WSGI server implementation may be able to support HTTP/2, but not 
support WebSockets, or it could support WebSockets via HTTP/1.x already. In 
fact basic request/response functionality of HTTP/2 maps into the existing WSGI 
API specification and doesn’t really require any changes be made to the WSGI 
specification.

For example, mod_wsgi already supports HTTP/2 by virtue of the fact that the 
mod_h2 module in Apache exists. The existing internal APIs of Apache and how 
mod_wsgi uses those means that HTTP/2 bridges into the WSGI world with no code 
changes to mod_wsgi.

To support WebSockets is a much bigger problem and is not achievable with CGI, 
FASTCGI, SCGI.

It may be able to be supported within the Apache/mod_wsgi implementation, but 
the major re-architecting required in the mod_wsgi code, and the fact that it 
couldn’t be done by simply exposing a socket, but by requiring a new high level 
abstract API be developed which doesn’t expose the actual socket object, means 
you are really talking about a whole new API.

To me the WebSocket requirement and the need for a completely new API rules out 
ever doing this as part of an updated WSGI specification. 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Andrew Godwin]

On Mon, Jan 4, 2016 at 6:22 PM, Robert Collins 
wrote:
>
> I think that WSGI got many things right - thats why so many things
> support it - but identifying which of its attributes is a factor for
> success, and which isn't is really hard: we're a decade on, more or
> less, and the ecosystem is a lot bigger. w.r.t. ASGI, it sounds like
> mongrel2 just in earlier days. Its a nifty way to build things, but I
> think its missing a fairly important thing: its not conceptually
> simple any more.
>

I appreciate that, though I would argue that the spec makes it seem
more complex than it actually is. You can write a ASGI application in
about 4 lines, in a way that I think is no worse than start_response and
environ in WSGI:

while True:
_, request = layer.receive_many("http.request", block=True)
response = {"content": "Hello World!", "status": 200}
layer.send(request['reply_channel'], response)

There's no reason ASGI couldn't have a WSGI-like layer specified on top for
HTTP requests so it looked more like current WSGI, but with slightly nicer
request and response objects. That is, in fact, mostly what the Django
Channels code does right now.

Mongrel2's design had some other issues, ZeroMQ being top of the list
(it's just not a good queuing system), as well as being way too stateful
and complex.
I could probably go on about the differences at length, but I'll refrain :)


>  - likewise on the application side: we need a Django person engaged
> with this - I reached out for one at the start of the last initiative
> but didn't manage to hook anyone, flask folk being interested would be
> good etc.
>

Hi, I am very interested, obviously.

Django _will probably have_ websocket support by next release, and it
almost certainly will run using ASGI internally; what's up for debate is
how it gets served, either via a WSGI shim, a WSGI-NG shim or native
support in a server.

I'm a big believer in going for implementation and iteration and polish
before jumping to a PEP, which is why I didn't bring ASGI anywhere near
that process yet. Once it's done, tested, implemented and we get it running
well at scale, then I think it would be ready. Django got a big pot of
funding from Mozilla just to get this process done, and several large sites
waiting to beta test things and report back how they work in the real world.

That said, I'm not stuck in some world where Django is the only Python web
framework; I would love to see the input of others. My ideal would be for a
new spec to renew the interest in writing more "raw" web apps in Python,
rather than people turning to other languages which have ostensibly better
WebSocket etc. support.


> On the asyncio / await etc side: I leave this up to the author.
> Thunking to WSGI from awaitable's would be tedious and thread-heavy I
> suspect. Obviously it can be done. AFAIK right now there is no mass
> move to asyncio from the server frameworks around: we're not going to
> solve the problems they have today with WSGI (e.g. the server-specific
> websockets requirements) by  making standardisation be predicated on
> moving something like Django to a new server core. I recall attempts
> to move other big frameworks around years ago now, taking a long long
> time I think the question of 'how can we enable interop of HTTP
> applications in asyncio' is best handled by a dedicated effort
> analogous to WSGI but in that programming model.
>

I agree that tying things to asyncio is a Bad Idea; any proposal should
certainly be more friendly to it, but requiring people to write fully
asynchronous code is a sure way to lose the simplicity of WSGI. We'd have
to do an incredible amount of work to Django to make it fully asynchronous
like that, to the point where it may never happen. What would probably
happen is just serving requests one at a time using thread pools or
something.

Andrew
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Robert Collins]

I should also say - thanks for picking this up. I may have been a tad
on the grumpy side on my prior mail - new years
paging-in-of-everything-after-a-break.

-Rob

On 5 January 2016 at 01:27, Cory Benfield  wrote:
> All,
>
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**
>
> It’s a new year, and that means it’s time for another attempt to get WSGI 2.0 
> off the ground. Many of you may remember that we attempted to do this last 
> year with Rob Collins leading the charge, but unfortunately personal 
> commitments made it impossible for Rob to keep pushing that attempt forward.
>
> Since then, the need for a revision of WSGI has become even more apparent. 
> Casual discussion on the web has indicated that application developers are 
> uncomfortable with the limitations of WSGI. These limitations are providing 
> an incentive for both application developers and server developers to take an 
> end-run around WSGI in an attempt to get a framework that is more suitable 
> for the modern web. A great example of the result of WSGI’s deficiencies is 
> Andrew Godwin’s channels work[0] for Django, which represents a paradigm 
> shift in application development that takes it far away from what WSGI is 
> today.
>
> For this reason, I think we need to try again to get WSGI 2.0 off the ground. 
> But I don’t believe we can do this without getting broad consensus from the 
> developer community that a revision to WSGI is needed, and without 
> understanding what developers need from a new revision of WSGI. This should 
> take into account the prior discussions we’d had on this thread: however, I’m 
> also going to actively solicit feedback from some of the more notable WSGI 
> implementers, to ensure that whatever comes out of this SIG is something that 
> they would actually use.
>
> This WG already had a list of requirements, which are as follows:
>
> - Support servers speaking HTTP/1.x, HTTP/2 and Websockets (potentially all 
> on a single port).
> - Support graceful degradation for applications that can use HTTP/2 but still 
> support HTTP/1.x requests.
> - Graceful incremental adoption path - no upgrade-all-components requirement 
> baked into the design.
> - Support Python 2.7 and 3.x (where x is not yet discussed)
> - Support the existing ecosystem of containers (such as mod_wsgi) with the 
> new API. We want a clean, fast and approachable API, and we want to ensure 
> that its no less friendly to work with than WSGI, for all that it will expose 
> much more functionality.
> - Apps need to be able to tell what protocol is in use, and what optional 
> features are available. For instance, HTTP/2 PUSH PROMISE is an optional 
> feature that can be disabled by clients. Websockets needs to expose a socket 
> like object, and so on.
> - Support websockets
> - Support HTTP/2
> - Support HTTP/1.x (which may be just 'point at PEP-’.)
> - Continue to support lightweight shims being built on top such as 
> https://github.com/Pylons/webob/blob/master/webob/request.py
>
> I believe that all of these requirements are up for grabs, and subject to 
> change and consensus discussion. In this thread, then, I’d like to hear from 
> people about these requirements and others. What do you believe WSGI 2.0 
> should do? Just as importantly, what do you believe it should not do? What 
> prior art should we take into account? Should we bother revising WSGI at all, 
> or should we let the wider application ecosystem pursue its own solutions à 
> la Django's channels? Should we simply adopt Andrew Godwin’s ASGI draft[1] on 
> which channels is based and call *that* WSGI 2.0?
>
> Right now I want this to be very open. I’d like people to come up with a 
> broad statement listing what they believe should and should not be present in 
> WSGI. This first stage of the work is very general: I just want to get a 
> feeling for what the community believes is important. Once we’re done with 
> that, if the consensus is that this work is worth pursuing, I’ll come up with 
> an initial draft that we can start making concrete changes to.
>
> In the short term, I’m going to keep this consultation open for **at least 
> two weeks**: that is, I will not start working on an initial draft PEP until 
> at least the **18th of January**. If you believe there are application or 
> server developers that should be involved in this discussion, please reach 
> out to them and point them to this list. I personally have CC’d some people 
> that I believe need to be involved in the discussion, but please reach out to 
> others as well.
>
> I’d really love to come to the end of 2016 with a solid direction for the 
> future of web programming in Python. I’m looking forward to working with you 
> all on achieving that.
>
> Thanks,
>
> Cory
>
>
> [0]: https://channels.readthedocs.org/en/latest/
> [1]: https://channels.readthedocs.org/en/latest/asgi.html



-- 
Robert Collins 

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Robert Collins]

So, as Graham said, I think calling it 2.0 is a bit of an issue -
HTTP/2.0 and WSGI 2.0 are not synonymous, given the diverse
requirements we have.



On 5 January 2016 at 01:27, Cory Benfield  wrote:
> All,
>
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**
>
> It’s a new year, and that means it’s time for another attempt to get WSGI 2.0 
> off the ground. Many of you may remember that we attempted to do this last 
> year with Rob Collins leading the charge, but unfortunately personal 
> commitments made it impossible for Rob to keep pushing that attempt forward.
>
> Since then, the need for a revision of WSGI has become even more apparent. 
> Casual discussion on the web has indicated that application developers are 
> uncomfortable with the limitations of WSGI. These limitations are providing 
> an incentive for both application developers and server developers to take an 
> end-run around WSGI in an attempt to get a framework that is more suitable 
> for the modern web. A great example of the result of WSGI’s deficiencies is 
> Andrew Godwin’s channels work[0] for Django, which represents a paradigm 
> shift in application development that takes it far away from what WSGI is 
> today.
>
> For this reason, I think we need to try again to get WSGI 2.0 off the ground. 
> But I don’t believe we can do this without getting broad consensus from the 
> developer community that a revision to WSGI is needed, and without 
> understanding what developers need from a new revision of WSGI. This should 
> take into account the prior discussions we’d had on this thread: however, I’m 
> also going to actively solicit feedback from some of the more notable WSGI 
> implementers, to ensure that whatever comes out of this SIG is something that 
> they would actually use.
>
> This WG already had a list of requirements, which are as follows:
>
> - Support servers speaking HTTP/1.x, HTTP/2 and Websockets (potentially all 
> on a single port).
> - Support graceful degradation for applications that can use HTTP/2 but still 
> support HTTP/1.x requests.
> - Graceful incremental adoption path - no upgrade-all-components requirement 
> baked into the design.
> - Support Python 2.7 and 3.x (where x is not yet discussed)
> - Support the existing ecosystem of containers (such as mod_wsgi) with the 
> new API. We want a clean, fast and approachable API, and we want to ensure 
> that its no less friendly to work with than WSGI, for all that it will expose 
> much more functionality.
> - Apps need to be able to tell what protocol is in use, and what optional 
> features are available. For instance, HTTP/2 PUSH PROMISE is an optional 
> feature that can be disabled by clients. Websockets needs to expose a socket 
> like object, and so on.
> - Support websockets
> - Support HTTP/2
> - Support HTTP/1.x (which may be just 'point at PEP-’.)
> - Continue to support lightweight shims being built on top such as 
> https://github.com/Pylons/webob/blob/master/webob/request.py
>
> I believe that all of these requirements are up for grabs, and subject to 
> change and consensus discussion. In this thread, then, I’d like to hear from 
> people about these requirements and others. What do you believe WSGI 2.0 
> should do? Just as importantly, what do you believe it should not do? What 
> prior art should we take into account? Should we bother revising WSGI at all, 
> or should we let the wider application ecosystem pursue its own solutions à 
> la Django's channels? Should we simply adopt Andrew Godwin’s ASGI draft[1] on 
> which channels is based and call *that* WSGI 2.0?
>
> Right now I want this to be very open. I’d like people to come up with a 
> broad statement listing what they believe should and should not be present in 
> WSGI. This first stage of the work is very general: I just want to get a 
> feeling for what the community believes is important. Once we’re done with 
> that, if the consensus is that this work is worth pursuing, I’ll come up with 
> an initial draft that we can start making concrete changes to.
>
> In the short term, I’m going to keep this consultation open for **at least 
> two weeks**: that is, I will not start working on an initial draft PEP until 
> at least the **18th of January**. If you believe there are application or 
> server developers that should be involved in this discussion, please reach 
> out to them and point them to this list. I personally have CC’d some people 
> that I believe need to be involved in the discussion, but please reach out to 
> others as well.
>
> I’d really love to come to the end of 2016 with a solid direction for the 
> future of web programming in Python. I’m looking forward to working with you 
> all on achieving that.

Whats the goal? The goal I had when I started was to enable a
portable-across-servers websockets + HTTP/2 capability in the Python
world. A PEP is one route to doing that. Just putting some code

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 4 Jan 2016, at 14:48, Damjan Georgievski  wrote:
> 
>> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
>> it at all?**
> …
>> - Support websockets
>> - Support HTTP/2
> 
> What does HTTP/2 support mean? What features of HTTP/2 need to be
> exposed in the wsgi api?

(CC-ing the list)

The current WSGI API does not provide any consensus method for doing server 
push. Such a thing could absolutely be done as an extension to WSGI in its 
current form, and we should consider that.

More generally, HTTP/2 is a bit more generous with what can be done with a 
stream than is the case in HTTP/1.1. For example, a stream could in principle 
be kept open indefinitely and used as a bi-directional communications channel: 
WSGI in its current form does not make that easy to do.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Armin Ronacher]

Hi,

On 04/01/2016 16:15, Cory Benfield wrote:

I don’t believe that will work.

Correct.  This cannot be done except for very simplistic servers.


Regards,
Armin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 4 Jan 2016, at 12:27, Cory Benfield  wrote:
> 
> All,
> 
> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do 
> it at all?**

Having set up the conversation, I also want to take part in it. So let me 
outline what I think we need from WSGI 2.

In my opinion, right now WSGI suffers from the following problems:

1. No out-of-the-box support for the concurrent programming techniques that are 
becoming extremely popular in the Python community and that should be 
encouraged.
2. No support for WebSockets, HTTP/2 server push, or any kind of generalised 
“extension” to the HTTP/1.1 request-response model.
3. A lack of an “escape hatch” that allows an application to take 
responsibility for a connection from the server, thereby ensuring that servers 
WSGI servers are the bottleneck for innovative and unusual programming 
techniques.
4. A confused bytes/unicode model that makes it mandatory to mix bytes and 
unicode in confusing ways at different parts of the stack.
5. A backward-looking design that favours applications doing in-memory 
buffering of responses over flushing response data to disk as it becomes 
available.

For this reason, my ideal WSGI 2.0 would be a relatively minor revision of PEP 
, focusing on addressing these core concerns. PEP  actually contains a 
great deal of what we already need, but either does not make it mandatory or 
fails to provide examples of how the tools can be used to solve the problem.

Problem (4) is easily resolved by coming to a consensus position on what fields 
should be byte strings, what ones should be unicode strings, and then very 
clearly setting out the rules for handling each side of that interaction.

I’d like to remove the `write` callable and instead **require** that the return 
value from the application callable be an iterable that produces the body. This 
should help encourage applications to use generators (or coroutines, as 
discussed later) to iteratively generate their output. This should help with 
problem (5) because it will allow us to propagate backpressure through a WSGI 
server to its application, based on how swiftly the server is able to flush 
data to the network.

Problem (2) is not actually entirely true: uWSGI has demonstrated that it’s 
entirely possible to extend WSGI to handle websockets, and such a model could 
presumably also be used for HTTP/2 server push. I’d like to take the approach 
of essentially creating ‘blessed’ extensions for websockets and HTTP/2 server 
push that tie in with the requirement for concurrency in problem (1). These 
will serve double duty, firstly ensuring that there’s a widely applicable 
general approach to supporting these features, while also providing a useful 
example of how WSGI 2 can be extended to support similar HTTP protocol 
extensions in future.

Fixing problem (3) seems useful to me, but I definitely need feedback from 
server implementers to see how they feel about becoming essentially dumb TCP 
proxies for applications in some scenarios.

Fixing problem (1) is the hardest, and for that reason I’ve reached out to a 
few Python concurrency experts. The best idea I have right now is based on 
trying to standardise on the ‘awaitable’ type and the ‘async for’ logic from 
PEP 492 to base WSGI 2 on asynchronous iterators. Essentially, the WSGI 2 
callable would return a PEP 493 asynchronous iterable.

I have some concerns about this approach, however. Firstly, I’m nervous about 
how well this works when called from C (e.g. for servers like uWSGI). Secondly, 
I want to make sure that it continues to function well with other event loop 
implementations (e.g. Twisted). For that reason, I’m open to alternative 
suggestions here. However, I do believe we need to support Python applications 
running concurrently in an event loop (e.g. asyncio-based or Twisted-based).

I have one further thing that I believe is important: I believe we need to 
support Python 2.7. It’s my sincere belief that if WSGI 2 is Python 3 only, we 
will struggle to get anyone to implement it: both servers and application 
frameworks will wait for people to take up Python 3 before implementing it, and 
users will not take up Python 3 until servers and application frameworks 
support WSGI 2.

What do people think of these goals?

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Damjan Georgievski]

>>> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we 
>>> do it at all?**
>> …
>>> - Support websockets
>>> - Support HTTP/2
>>
>> What does HTTP/2 support mean? What features of HTTP/2 need to be
>> exposed in the wsgi api?
>
> (CC-ing the list)
>
> The current WSGI API does not provide any consensus method for doing server 
> push. Such a thing could absolutely be done as an extension to WSGI in its 
> current form, and we should consider that.
>
> More generally, HTTP/2 is a bit more generous with what can be done with a 
> stream than is the case in HTTP/1.1. For example, a stream could in principle 
> be kept open indefinitely and used as a bi-directional communications 
> channel: WSGI in its current form does not make that easy to do.
>

So will a general solution for both HTTP/2 and Websockets be exposing
the underlaying socket as an 'wsgi.fd' environment variable?



-- 
damjan
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Armin Ronacher]

Hi,

I personally probably do not want to participate in this discussion much 
but I want to leave some thoughts in case someone finds them useful.


I personally think that fundamentally "concurrent programming" and just 
getting access to a socket is not something that fits into a generically 
deployable container specification which is what WSGI largely is.  WSGI 
was quite trivially specified as what happens from request to response 
and even in that area it already suffered from significant limitations 
in regards to where the specification did not consider what servers 
would do with it.


I do not want to go into detail too much but WSGI the spec never really 
concerned itself with the vast complexity that is HTTP in practice 
(chunked requests, transfer encodings, stream termination signalling etc.)


I heavily doubt that dragging concurrency into the spec will make it any 
less problematic for real world situations.


Why do we need concurrency on the spec level?  I honestly do not see the 
point because in practical terms we might just make a spec that then 
cannot really be deployed in practice just because nobody would want to. 
 Making a server that gracefully shuts down when things are purely 
request/response is already tricky enough, but finding a method to shut 
down a server with active stream connections is something that does not 
even have enough agreement between implementations yet (which also needs 
a lot of client support) that I don't think will fit into a specification.


I honestly do not think that you can have it both ways: a WSGI 
specification and a raw socket.  Maybe we reached the point where WSGI 
should just be deprecated and frameworks themselves will fill the gap. 
We would only specify a data exchange layer so that frameworks can 
interoperate in some way or another.



Regards,
Armin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 4 Jan 2016, at 14:56, Damjan Georgievski  wrote:
> 
 **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we 
 do it at all?**
>>> …
 - Support websockets
 - Support HTTP/2
>>> 
>>> What does HTTP/2 support mean? What features of HTTP/2 need to be
>>> exposed in the wsgi api?
>> 
>> (CC-ing the list)
>> 
>> The current WSGI API does not provide any consensus method for doing server 
>> push. Such a thing could absolutely be done as an extension to WSGI in its 
>> current form, and we should consider that.
>> 
>> More generally, HTTP/2 is a bit more generous with what can be done with a 
>> stream than is the case in HTTP/1.1. For example, a stream could in 
>> principle be kept open indefinitely and used as a bi-directional 
>> communications channel: WSGI in its current form does not make that easy to 
>> do.
>> 
> 
> So will a general solution for both HTTP/2 and Websockets be exposing
> the underlaying socket as an 'wsgi.fd' environment variable?

I don’t believe that will work.

Both HTTP/2 and Websockets have framing logic, and HTTP/2 also has a 
moderately-complex connection-level state machine that makes passing off the FD 
to an application a fraught endeavour. I suspect in both cases they will be 
best handled by having a extension protocol that allows the protocol-specific 
logic to function sensibly. In both cases, standard Python generators used as 
coroutines would be totally suitable, with concurrency being the only fly in 
the ointment to getting this right.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 4 Jan 2016, at 15:08, Armin Ronacher  wrote:
> 
> I honestly do not think that you can have it both ways: a WSGI specification 
> and a raw socket.  Maybe we reached the point where WSGI should just be 
> deprecated and frameworks themselves will fill the gap. We would only specify 
> a data exchange layer so that frameworks can interoperate in some way or 
> another.

This is one of the bits of feedback I expected we’d get, and it’s one we really 
do need to consider. It’s possible that the time for WSGI is coming to a close.

However, I’d like to try not to give up as a first step. =)

Your core question seems to be: “why do we need a spec that specifies 
concurrency?” I think this is reasonable. One way out might be to take the 
route of ASGI[0], which essentially uses a message broker to act as the 
interface between server and application. This lets applications handle their 
own concurrency without needing to co-ordinate with the server. From there the 
spec doesn’t need to handle concurrency, as the application and server are in 
separate processes.

However, if the application and server run together (as with WSGI today), I 
don’t think we can get out of needing to talk about concurrency, because *not* 
talking about it essentially forces the application to assume that each 
request/response cycle runs in a new process and that it cannot share memory or 
resources. That’s what we have at the moment, and using something like asyncio 
or Twisted in such an environment is very tricky and essentially requires that 
you run on top a server that also uses them (see Hendrix or Klein).

Let’s step back for a moment and consider the most simple case: one request, 
one response, both sent across WSGI as a complete entity. Could we not specify 
that the application callable has to return an object that is analogous to a 
future/promise/Deferred: essentially, an object that can be waited on until the 
response is ready. In that situation, a WSGI server could repeatedly call the 
application callable and then wait until any of the future/promise/Deferreds is 
ready, then send that data back in the response. This would allow the 
application to use its own concurrency model. Setting aside whether it’s a good 
model (it’s not), do you agree that it would work?

If it would, I think it’s reasonable to consider whether we can come up with a 
different, sufficiently-general approach to this problem, potentially based on 
that kind of approach.

Cory


[0]: https://channels.readthedocs.org/en/latest/asgi.html


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Armin Ronacher]

Hi,

On 04/01/2016 16:30, Cory Benfield wrote:

Your core question seems to be: “why do we need a spec that specifies
concurrency?” I think this is reasonable. One way out might be to
take the route of ASGI[0], which essentially uses a message broker to
act as the interface between server and application. This lets
applications handle their own concurrency without needing to
co-ordinate with the server. From there the spec doesn’t need to
handle concurrency, as the application and server are in separate
processes.
I think the *only* way to scale websockets is to use an RPC system to 
dispatch commands and to handle fan out somewhere centralized.  This for 
instance works really well with redis as a broker.  All larger 
deployments of websockets I have worked with so far involved a simple 
redis to websocket server that barely restarts and dispatches commands 
(and receives messages) via redis.


That's a simple an straightforward way that still keeps deployments work 
well because you never restart the actual connections unless you need to 
pull a cable or you have a bug in the websocket server.


That's why I'm personally also not really interested in this topic as 
for large scale deployments this is not really an issue and for toy 
applications I do not use websockets ;)



Regards,
Armin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Andrew Godwin]

Thought I should weigh in on this, as I got mentioned by name in it. Sorry
about maybe not getting the threading right, I wasn't subscribed to the
list still it sprang from the grave this morning!

So, to quote the reply I just sent to Cory in django-developers:


I don't think ASGI would be a suitable replacement for WSGI in the current
form; in particular, I suspect it will have a performance disadvantage,
though I've not quantified yet.

That said, if Django Channels does become the primary method for
communication with web clients, the only thing we would want out of WSGI 2
would be something that could easily plug into ASGI - that is, something
that supports WebSockets, can handle simultaneous connections with many
clients in the same thread via cooperative multitasking or similar, and
allows raw access to sockets. Given those things, an ASGI HTTP/WebSocket
protocol server could simply be a WSGI 2 application, allowing much better
code reuse.


In particular, ASGI is never going to be low-level enough for people who
want to do crazy things; there's no access to input and output socket
streams, for starters. While I've tried to maintain some of the WSGI
familiarity, the whole thing is one step higher a level of abstraction, and
with that brings inflexibility and slight performance decreases that I
suspect people will get mad about if it becomes WSGI 2.

Now, I do think that the majority of web frameworks out there right now
could be ported to ASGI in no time at all; the asgiref package I'm working
on will ship with an ASGI-to-WSGI wrapper that just makes that work, as
long as the WSGI application doesn't try and be too clever with output

I don't want to express an opinion on if this means WSGI 2 is unnecessary
since I'm pretty biased; we need _some_ sort of standard that covers
WebSockets going forward, or we have the uwsgi problem where everyone
invents their own API for it.

My wishlist, though, is basically:

- WebSocket support
- Concurrent client handling support
- WebSockets and HTTP mixed-mode (i.e. on the same port and URLs)
- Fix the bytes/unicode issue that's all over the place in the current spec

These would all work super well in making ASGI itself pluggable into WSGI 2.

HTTP/2 features also need some place (like server push), and there's other
things cropping up like WebRTC that we might need to eventually support.
This is one of the reasons ASGI separates transport from protocol encoding;
I know people will invent new crazy web stuff in the next few years, and I
wanted a pattern that could extend to support most protocol types without
changing the abstraction completely.

Andrew
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com