Re: CORS performance proposal

2015-06-08 Thread Martin Thomson
On 8 June 2015 at 21:30, Nottingham, Mark mnott...@akamai.com wrote:
 A header denoting site-wide metadata would work for this too, of course, if 
 folks were comfortable with the security properties of doing that (as well as 
 the potential response overhead).


The security properties bother me a little.  Alt-Svc is showing us
that we can't just define a header field like that without some
serious analysis.



Re: CORS performance proposal

2015-06-08 Thread Nottingham, Mark
Picking up an old thread...

On 20 Feb 2015, at 12:54 pm, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 
 * Martin Thomson wrote:
 On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`.
 
 I missed that.  In which case I'd point out that `OPTIONS *` is very
 poorly supported.  Some people (myself included) want it to die a
 flaming death.
 
 Evidence for poorly supported would certainly be helpful (web hosting
 packages without TLS support, for instance, do not count, though).

FWIW, this is old but I think still true, at least regarding OPTIONS *:
  https://www.mnot.net/blog/2005/04/03/options

I wrote up my thoughts about using OPTIONS after looking at it for many years:
  https://www.mnot.net/blog/2012/10/29/NO_OPTIONS

Also - it's becoming more common for content providers to ask CDNs for CORS 
preflight caching. Since OPTIONS doesn't fit into HTTP caching, something 
custom has to be designed, which means dealing with all of the corner cases, 
interop issues, etc. 

Using a .well-known would avoid all of that, since it can be cached just like 
any other response. If there's concern about extra requests against origins 
that don't use this mechanism, I'd suggest defining a default cacheability for 
404 responses on the resource (e.g., 3 hours). That said, I suspect that if 
this were defined, take-up would be good; it's much easier than supporting 
pre-flights.

A header denoting site-wide metadata would work for this too, of course, if 
folks were comfortable with the security properties of doing that (as well as 
the potential response overhead). 

Cheers,

--
Mark Nottinghamm...@akamai.comhttps://www.mnot.net/







Re: CORS performance proposal

2015-06-08 Thread Anne van Kesteren
On Tue, Jun 9, 2015 at 6:42 AM, Martin Thomson martin.thom...@gmail.com wrote:
 The security properties bother me a little.  Alt-Svc is showing us
 that we can't just define a header field like that without some
 serious analysis.

Same goes for a site-wide file. See crossdomain.xml. However, either
coupled with credentials mode = omit seems okayish... Mark, do these
CDN requests mention credentials?


-- 
https://annevankesteren.nl/



Re: CORS performance proposal

2015-06-08 Thread Nottingham, Mark

 On 9 Jun 2015, at 2:42 pm, Martin Thomson martin.thom...@gmail.com wrote:
 
 On 8 June 2015 at 21:30, Nottingham, Mark mnott...@akamai.com wrote:
 A header denoting site-wide metadata would work for this too, of course, if 
 folks were comfortable with the security properties of doing that (as well 
 as the potential response overhead).
 
 The security properties bother me a little.  Alt-Svc is showing us
 that we can't just define a header field like that without some
 serious analysis.

Indeed. Also, an intermediary cache (whether a proxy or a CDN) would need to 
monitor all of the headers sent back for a given origin to figure out the 
applicable policy, and rewrite responses accordingly. It wouldn't just work out 
of the box like a .well-known would.

Cheers,


--
Mark Nottinghamm...@akamai.comhttps://www.mnot.net/







Re: CORS performance proposal

2015-06-08 Thread Nottingham, Mark

 On 9 Jun 2015, at 2:54 pm, Anne van Kesteren ann...@annevk.nl wrote:
 
 On Tue, Jun 9, 2015 at 6:42 AM, Martin Thomson martin.thom...@gmail.com 
 wrote:
 The security properties bother me a little.  Alt-Svc is showing us
 that we can't just define a header field like that without some
 serious analysis.
 
 Same goes for a site-wide file. See crossdomain.xml. However, either
 coupled with credentials mode = omit seems okayish... Mark, do these
 CDN requests mention credentials?

Will look into it. Supporting without credentials (and leaving future 
extensibility for the possibility) would certainly be a good start.

Cheers,


--
Mark Nottinghamm...@akamai.comhttps://www.mnot.net/







Re: CORS performance

2015-03-05 Thread Austin William Wright
On Mon, Feb 23, 2015 at 12:42 PM, Jonas Sicking jo...@sicking.cc wrote:

 Do we have any data on how common it is for people to use CORS with
 credentials? My impression is that it's far less common than CORS
 without credentials.

 If that's the case then I think we'd get most of the functionality,
 with essentially none of the risk, by only allowing server-wide
 cookie-less preflights.

 But data would help for sure.



The credentials issue is a great concern. I've seen two cases I've been
able to exploit where the Origin header is always passed back to
Access-Control-Allow-Origin with Access-Control-Allow-Credentials; not
realizing this exposes CSRF tokens and other sensitive information. Though
this is not so much data as anecdote. (One was on a company-internal
project, one briefly appeared in a software library we are using in between
stable releases.) It's not hard to find additional examples [1][2][3][4].
I've never used the credentials functionality of CORS, instead passing an
Authorization header explicitly. I think we could survive without it.

I also recently ran into the CORS performance issue. CORS seems very biased
against hypermedia services, in that it doubles the roundtrip time to an
already very chatty API design.

To work around these two issues, I began work on a proxy to accept
`message/http` POST requests which returns `message/http` responses
(actually for now, application/json to avoid the full RFC7230
implementation). (Existing proxies solving this problem don't have the
performance characteristics or flexibility, or both, desired around HTTP
headers, hypermedia, and streaming.) The only downside is this bypasses the
user-agent cache, but for many kinds of services the user-agent cache is
rarely utilized (for instance, jQuery's AJAX actively disables caching).

I suggest that any request I can make with this proxy (which does not
itself verify CORS before passing the request), I should be able to make
without the CORS preflight request.

The semantics of the proposed header then becomes a declaration that This
server is accessible from the Internet (as opposed to a request on an
intranet server made from an Internet-served page); and would then allow
the making of requests that never utilize stored user-agent data
(credentials). (With the exception of the Date header, could this be
called deterministic, since the same XHR call will always produce the same
HTTP request?)

This definition would have the following desirable effects:

1. HTTP requests become stateless
2. Removes the possibility of exposing sensitive, stored user information
(a TRACE response, if honored, would not divulge anything the sender
doesn't already know)
3. Brings performance in line with non-browser user-agents that don't
require CORS checks
4. Eliminates the privacy and security concerns associated with CORS
work-arounds, like JSONP. As seen in [5] [6] [7]

This wouldn't introduce new security concerns, because by definition it
would not allow requests that couldn't already be made by proxy.

The `OPTIONS *` request seems appropriate for this; as it is defined to be
a request for metadata about the server itself, which is all that is
necessary.

Austin Wright.

[1]
https://github.com/primiano/blink-gitcs/blob/0b5424070e3006102e0036deea1e2e263b871eaa/LayoutTests/http/tests/security/resources/cors-redirect.php
[2]
https://github.com/wohlig/sergybackend/blob/3df9e42dccf620e53253c746022fbe1e53a97a3a/application/views/json.php
[3]
https://github.com/AslakNiclasen/ProjektOpgave/blob/7d5de466398371ab1481973bf54074ab005fffdc/Kode/ad_controller/.htaccess2.txt
[4]
https://github.com/datfinesoul/starphleet-logviewer/blob/af42371daad52fa7e7ba759e94b20024077a30bb/nginx.conf
[5] http://blog.javascripting.com/2015/01/17/dont-hassle-with-cors/
[6] https://jsonp.nodejitsu.com/
[7] https://github.com/Rob--W/cors-anywhere


Re: CORS performance

2015-02-25 Thread Anne van Kesteren
On Tue, Feb 24, 2015 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote:
 On Tue, Feb 24, 2015 at 3:25 AM, Anne van Kesteren ann...@annevk.nl wrote:
 If we only do it for this, could we combine that feature with the
 existing preflight then? Support a Access-Control-Allow-Origin-Wide:
 true header or some such that's mutually exclusive with
 Access-Control-Allow-Credentials: true.

 I don't have opinions on this.

I filed https://www.w3.org/Bugs/Public/show_bug.cgi?id=28099

It would still be great to see data, hear from web developers, and
hear from other browsers. I feel like we don't really have enough to
go on yet, but at least we have a somewhat concrete plan for the case
that does not involve credentials.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-24 Thread Anne van Kesteren
On Mon, Feb 23, 2015 at 8:42 PM, Jonas Sicking jo...@sicking.cc wrote:
 On Mon, Feb 23, 2015 at 11:06 AM, Anne van Kesteren ann...@annevk.nl wrote:
 That combined with requiring to list
 the explicit origin has worked well for CORS so far.

 This could potentially help.

 I don't remember the details of how/why people screwed up with
 crosssite.xml. But if the problem was that people hosted multiple
 services on the same server and only thought of one of them when
 writing a policy, then this won't really help very much.

http://www.jamesward.com/2009/11/08/how-bad-crossdomain-policies-expose-protected-data-to-malicious-applications/
seems to support that.


 Do we have any data on how common it is for people to use CORS with
 credentials? My impression is that it's far less common than CORS
 without credentials.

I don't have data. It seems we don't have telemetry for this in Gecko.
Anyone else? I would also suspect that Access-Control-Allow-Origin: *
is more common.


 If that's the case then I think we'd get most of the functionality,
 with essentially none of the risk, by only allowing server-wide
 cookie-less preflights.

If we only do it for this, could we combine that feature with the
existing preflight then? Support a Access-Control-Allow-Origin-Wide:
true header or some such that's mutually exclusive with
Access-Control-Allow-Credentials: true.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-24 Thread Jonas Sicking
On Tue, Feb 24, 2015 at 3:25 AM, Anne van Kesteren ann...@annevk.nl wrote:
 If that's the case then I think we'd get most of the functionality,
 with essentially none of the risk, by only allowing server-wide
 cookie-less preflights.

 If we only do it for this, could we combine that feature with the
 existing preflight then? Support a Access-Control-Allow-Origin-Wide:
 true header or some such that's mutually exclusive with
 Access-Control-Allow-Credentials: true.

I don't have opinions on this.

/ Jonas



Re: CORS performance

2015-02-23 Thread Jonas Sicking
On Mon, Feb 23, 2015 at 7:15 AM, Henri Sivonen hsivo...@hsivonen.fi wrote:
 On Tue, Feb 17, 2015 at 9:31 PM, Brad Hill hillb...@gmail.com wrote:
 I think it is at least worth discussing the relative merits of using a
 resource published under /.well-known for such use cases, vs. sending
 pinned headers with every single resource.

 FWIW, when CORS was designed, the Flash crossdomain.xml design (which
 uses a well-known URL though not under /.well-known) already existed
 and CORS deliberately opted for a different design.

 It's been a while, so I don't recall what the reasons against adopting
 crossdomain.xml or something very similar to it were, but considering
 that the crossdomain.xml design was knowingly rejected, it's probably
 worthwhile to pay attention to why.

A lot websites accidentally enabled cross-origin requests with
cookies. Not realizing that that enabled attackers to make requests
that had side-effects as well as read personal user data without user
permission.

In short, it was very easy to misconfigure a server, and people did.

This is why I would feel dramatically more comfortable if we only
enabled server-wide opt-in for credential-less requests. Those are
many orders of magnitude easier to make secure.

/ Jonas



Re: CORS performance proposal

2015-02-23 Thread Jonas Sicking
On Sat, Feb 21, 2015 at 11:18 PM, Anne van Kesteren ann...@annevk.nl wrote:
 On Sat, Feb 21, 2015 at 10:17 AM, Martin Thomson
 martin.thom...@gmail.com wrote:
 On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote:
 High-byte of what? A URL is within ASCII range when it reaches the
 server. This is the first time I hear of this.

 Apparently, all sorts of muck floats around the Internet.  When we did
 HTTP/2 we were forced to accept that header field values (URLs in
 particular) were a sequence of octets.  Those are often interpreted as
 strings in various interesting ways.

 But in this particular case it must be the browser that generates said
 muck, no? Other than Internet Explorer (and that's a couple versions
 ago, so wouldn't support this protocol anyway), there's no browser
 that does this as far as I know.

All browsers support sending %xx stuff to the server. Decoding those
is likely more often than not happening in a server-specific way
still. Despite specs defining how they should do it.

/ Jonas



Re: CORS performance proposal

2015-02-23 Thread Jonas Sicking
On Fri, Feb 20, 2015 at 11:43 PM, Anne van Kesteren ann...@annevk.nl wrote:
 On Fri, Feb 20, 2015 at 9:38 PM, Jonas Sicking jo...@sicking.cc wrote:
 On Fri, Feb 20, 2015 at 1:05 AM, Anne van Kesteren ann...@annevk.nl wrote:
 An alternative is that we attempt to introduce
 Access-Control-Policy-Path again from 2008. The problems you raised
 https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html
 seem surmountable. URL parsing is defined in more detail these days
 and we could simply ban URLs containing escaped \ and /.

 I do remember that another issue that came up back then was that
 servers would treat more than just '\', or the escaped version
 thereof, as a /. But also any character whose low-byte was equal to
 the ascii code for '\' or '/'. I.e. the server would just cut the
 high-byte when doing some internal 2byte-string to 1byte-string
 conversion. Potentially this conversion is affected by what character
 encodings the server is configured for too, but i'm less sure about
 that.

 High-byte of what? A URL is within ASCII range when it reaches the
 server. This is the first time I hear of this.

I really don't remember the details. I'd recommend talking to
microsoft since I believe they had done most research into this at the
time.

Keep in mind though that just because URL parsing is defined a
particular way, doesn't mean that software implements it that way.

/ Jonas



Re: CORS performance

2015-02-23 Thread Jonas Sicking
On Mon, Feb 23, 2015 at 11:06 AM, Anne van Kesteren ann...@annevk.nl wrote:
 On Mon, Feb 23, 2015 at 7:55 PM, Jonas Sicking jo...@sicking.cc wrote:
 A lot websites accidentally enabled cross-origin requests with
 cookies. Not realizing that that enabled attackers to make requests
 that had side-effects as well as read personal user data without user
 permission.

 In short, it was very easy to misconfigure a server, and people did.

 This is why I would feel dramatically more comfortable if we only
 enabled server-wide opt-in for credential-less requests. Those are
 many orders of magnitude easier to make secure.

 Why is that not served by requiring an additional header that
 explicitly opts into that case?

I don't think an extra header is that much harder to deploy than
crosssite.xml is. I.e. I don't see strong reasons to think that people
won't misconfigure.

 That combined with requiring to list
 the explicit origin has worked well for CORS so far.

This could potentially help.

I don't remember the details of how/why people screwed up with
crosssite.xml. But if the problem was that people hosted multiple
services on the same server and only thought of one of them when
writing a policy, then this won't really help very much.

Do we have any data on how common it is for people to use CORS with
credentials? My impression is that it's far less common than CORS
without credentials.

If that's the case then I think we'd get most of the functionality,
with essentially none of the risk, by only allowing server-wide
cookie-less preflights.

But data would help for sure.

/ Jonas



Re: CORS performance

2015-02-23 Thread Anne van Kesteren
On Mon, Feb 23, 2015 at 7:55 PM, Jonas Sicking jo...@sicking.cc wrote:
 A lot websites accidentally enabled cross-origin requests with
 cookies. Not realizing that that enabled attackers to make requests
 that had side-effects as well as read personal user data without user
 permission.

 In short, it was very easy to misconfigure a server, and people did.

 This is why I would feel dramatically more comfortable if we only
 enabled server-wide opt-in for credential-less requests. Those are
 many orders of magnitude easier to make secure.

Why is that not served by requiring an additional header that
explicitly opts into that case? That combined with requiring to list
the explicit origin has worked well for CORS so far.


-- 
https://annevankesteren.nl/



Re: CORS performance proposal

2015-02-21 Thread Anne van Kesteren
On Sat, Feb 21, 2015 at 10:17 AM, Martin Thomson
martin.thom...@gmail.com wrote:
 On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote:
 High-byte of what? A URL is within ASCII range when it reaches the
 server. This is the first time I hear of this.

 Apparently, all sorts of muck floats around the Internet.  When we did
 HTTP/2 we were forced to accept that header field values (URLs in
 particular) were a sequence of octets.  Those are often interpreted as
 strings in various interesting ways.

But in this particular case it must be the browser that generates said
muck, no? Other than Internet Explorer (and that's a couple versions
ago, so wouldn't support this protocol anyway), there's no browser
that does this as far as I know.


 I wouldn't *completely* discount the potential for the conversions
 Jonas mentions here.  A Java server might parse UTF-8 into the
 internal UTF-16 representation and then who knows what happens next.

There's no utf-8 either.


-- 
https://annevankesteren.nl/



Re: CORS performance proposal

2015-02-21 Thread Martin Thomson
On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote:
 High-byte of what? A URL is within ASCII range when it reaches the
 server. This is the first time I hear of this.

Apparently, all sorts of muck floats around the Internet.  When we did
HTTP/2 we were forced to accept that header field values (URLs in
particular) were a sequence of octets.  Those are often interpreted as
strings in various interesting ways.

I wouldn't *completely* discount the potential for the conversions
Jonas mentions here.  A Java server might parse UTF-8 into the
internal UTF-16 representation and then who knows what happens next.



Re: CORS performance proposal

2015-02-20 Thread Anne van Kesteren
On Thu, Feb 19, 2015 at 9:22 PM, Jonas Sicking jo...@sicking.cc wrote:
 Would this be allowed for both requests with credentials and requests
 without credentials? The security implications of the two are very
 different.

Yes, but the latter requires the Access-Control-Allow-Credentials
header to be included in the response.


An alternative is that we attempt to introduce
Access-Control-Policy-Path again from 2008. The problems you raised
https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html
seem surmountable. URL parsing is defined in more detail these days
and we could simply ban URLs containing escaped \ and /.


-- 
https://annevankesteren.nl/



Re: CORS performance proposal

2015-02-20 Thread Anne van Kesteren
On Fri, Feb 20, 2015 at 9:38 PM, Jonas Sicking jo...@sicking.cc wrote:
 On Fri, Feb 20, 2015 at 1:05 AM, Anne van Kesteren ann...@annevk.nl wrote:
 An alternative is that we attempt to introduce
 Access-Control-Policy-Path again from 2008. The problems you raised
 https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html
 seem surmountable. URL parsing is defined in more detail these days
 and we could simply ban URLs containing escaped \ and /.

 I do remember that another issue that came up back then was that
 servers would treat more than just '\', or the escaped version
 thereof, as a /. But also any character whose low-byte was equal to
 the ascii code for '\' or '/'. I.e. the server would just cut the
 high-byte when doing some internal 2byte-string to 1byte-string
 conversion. Potentially this conversion is affected by what character
 encodings the server is configured for too, but i'm less sure about
 that.

High-byte of what? A URL is within ASCII range when it reaches the
server. This is the first time I hear of this.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread Jonas Sicking
On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote:
 so presumably it is OK to set the Content-Type to text/plain

 Thats not ok, but may explain my confusion, is Content-Type considered a
 Custom Header that will always trigger a preflight? if so then none of the
 caching will apply, CouchDB requires sending the appropriate content-type

We most likely can consider the content-type header as *not* custom.
I was one of the people way back when that pointed out that there's a
theoretical chance that allowing arbitrary content-type headers could
cause security issues. But it seems highly theoretical.

I suspect that the mozilla security team would be fine with allowing
arbitrary content-types to be POSTed though. Worth asking. I can't
speak for other browser vendors of course.

/ Jonas



Re: CORS performance proposal

2015-02-19 Thread Jonas Sicking
Would this be allowed for both requests with credentials and requests
without credentials? The security implications of the two are very
different.

/ Jonas

On Thu, Feb 19, 2015 at 5:29 AM, Anne van Kesteren ann...@annevk.nl wrote:
 When the user agent is about to make its first preflight to an origin
 (timeout up to the user agent), it first makes a preflight that looks
 like:

   OPTIONS *
   Access-Control-Request-Origin-Wide-Cache: [origin]
   Access-Control-Request-Method: *
   Access-Control-Request-Headers: *

 If the response is

   2xx XX
   Access-Control-Allow-Origin-Wide-Cache: [origin]
   Access-Control-Allow-Methods: *
   Access-Control-Allow-Headers: *
   Access-Control-Max-Age: [max-age]

 then no more preflights will be made for the duration of [max-age] (or
 shortened per user agent preference). If the response includes

   Access-Control-Allow-Credentials: true

 the cache scope is increased to requests that include credentials.

 I think this has a reasonable tradeoff between security and opening up
 all the power of the HTTP APIs on the server without the performance
 hit. It still makes the developer very conscious about the various
 features involved.

 The cache would be on a per requesting origin basis as per the headers
 above. The Origin and Access-Control-Allow-Origin would not take part
 in this exchange, to make it very clear what this is about.

 (This does not affect Access-Control-Expose-Headers or any of the
 other headers required as part of non-preflight responses.)


 --
 https://annevankesteren.nl/




Re: CORS performance

2015-02-19 Thread Jonas Sicking
On Thu, Feb 19, 2015 at 3:30 AM, Anne van Kesteren ann...@annevk.nl wrote:
 On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote:
 With Couch / PouchDB we are working with an existing REST API wherein every
 request is to a different url (which is unlikely to change), the performance
 impact is significant since most of the time is used up by latency, the CORS
 preflight request essentially double the time it takes to do anything

 Yeah, also, it should not be up to us how people design their HTTP
 APIs. Limiting HTTP in that way because it is hard to make CORS scale
 seems bad.


 I think we've been too conservative when introducing CORS. It's
 effectively protecting content behind a firewall,

...and content that uses user credentials like cookies.

/ Jonas



Re: CORS performance

2015-02-19 Thread Brad Hill
I think that POSTing JSON would probably expose to CSRF a lot of things
that work over HTTP but don't expect to be interacted with by web browsers
in that manner.  That's why the recent JSON encoding for forms mandates
that it be same-origin only.

On Thu Feb 19 2015 at 12:23:48 PM Jonas Sicking jo...@sicking.cc wrote:

 On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote:
  so presumably it is OK to set the Content-Type to text/plain
 
  Thats not ok, but may explain my confusion, is Content-Type considered a
  Custom Header that will always trigger a preflight? if so then none of
 the
  caching will apply, CouchDB requires sending the appropriate content-type

 We most likely can consider the content-type header as *not* custom.
 I was one of the people way back when that pointed out that there's a
 theoretical chance that allowing arbitrary content-type headers could
 cause security issues. But it seems highly theoretical.

 I suspect that the mozilla security team would be fine with allowing
 arbitrary content-types to be POSTed though. Worth asking. I can't
 speak for other browser vendors of course.

 / Jonas




Re: CORS performance proposal

2015-02-19 Thread Martin Thomson
On 20 February 2015 at 00:29, Anne van Kesteren ann...@annevk.nl wrote:
   Access-Control-Allow-Origin-Wide-Cache: [origin]


This has some pretty implications for server deployments that host
mutual distrustful applications.  Now, these servers are already
pretty well hosed from other directions, but I don't believe that
there is any pre-existing case where a header field set in a request
to /x could affect future requests to /y.

An alternative would be to use /.well-known for site wide policies.



Re: CORS performance

2015-02-19 Thread Martin Thomson
On 18 February 2015 at 06:31, Brad Hill hillb...@gmail.com wrote:
 Some of the things that argue against /.well-known are:

 1) Added latency of fetching the resource.

It's not available everywhere yet, but you could push it, based on the below.

 2) Clients hammering servers for non-existent /.well-known resources (the
 favicon issue)

You could avoid that by Link:-ing to the /.well-known and only hitting
it if the link appears.



Re: CORS performance

2015-02-19 Thread Brian Smith
Anne van Kesteren ann...@annevk.nl wrote:
 Concerns raised by Monsur
 https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0260.html
 and others before him are still valid.

 When you have an HTTP API on another origin you effectively get a huge
 performance penalty. Even with caching of preflights, as each fetch is
 likely to go to a distinct URL.

Definitely there is a huge performance penalty per-request when the
preflight isn't cached.

But:

1. Preflight is only necessary for a subset of CORS requests.
Preflight is never done for GET or HEAD, and you can avoid preflight
for POST requests by making your API accept data in a format that
matches what HTML forms post. Therefore, we're only talking about PUT,
DELETE, less common forms of POST, and other less commonly-used
methods.

2. It seems very wasteful to design an API that requires multiple
PUT/DELETE/POST requests to complete a transaction (I'm using the word
transaction loosely here; I don't mean ACID, necessarily). A lot of
people think that REST means that you should only change a resource by
PUT/DELETE/POSTing to its URL, however that's a misunderstanding of
what REST is really about. Regardless of CORS, it is a good idea to
design APIs in such a way that every modification transaction can be
done in one or as few requests as possible, and this can be done a way
that is in line with RESTful design. This type of design is
more-or-less required when you need ACID transactions anyway.

3. Often you can predict which resources need preflight. With HTTP/2
and SPDY, the server can use the server push mechanism to push
preflight responses to the client before the client even sends them.

Given #1, #2, and #3, I'm a little bit unsure how bad the performance
problem really is, and how bad it will be going forward. It would be
good to see a concrete example to get a better understanding of the
issue.

Cheers,
Brian



Re: CORS performance

2015-02-19 Thread Anne van Kesteren
On Thu, Feb 19, 2015 at 11:45 AM, Anne van Kesteren ann...@annevk.nl wrote:
 On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote:
 1. Preflight is only necessary for a subset of CORS requests.
 Preflight is never done for GET or HEAD, and you can avoid preflight
 for POST requests by making your API accept data in a format that
 matches what HTML forms post. Therefore, we're only talking about PUT,
 DELETE, less common forms of POST, and other less commonly-used
 methods.

 Euh, if you completely ignore headers, sure. But most HTTP APIs will
 use some amount of custom headers, meaning *all* methods require a
 preflight.

And you seem to forget that an HTTP API typically covers a large set
of URLs (e.g. if you use remote database such as PouchDB). With CORS
that requires at least one preflight per URL.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread Brian Smith
On Thu, Feb 19, 2015 at 2:45 AM, Anne van Kesteren ann...@annevk.nl wrote:
 On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote:
 1. Preflight is only necessary for a subset of CORS requests.
 Preflight is never done for GET or HEAD, and you can avoid preflight
 for POST requests by making your API accept data in a format that
 matches what HTML forms post. Therefore, we're only talking about PUT,
 DELETE, less common forms of POST, and other less commonly-used
 methods.

 Euh, if you completely ignore headers, sure. But most HTTP APIs will
 use some amount of custom headers, meaning *all* methods require a
 preflight.

Is it really true that most HTTP APIs will sue some amount of custom
headers? And, is is it necessary for these APIs to be designed such
that the custom headers are required?

Cheers,
Brian



Re: CORS performance

2015-02-19 Thread Dale Harvey
With Couch / PouchDB we are working with an existing REST API wherein every
request is to a different url (which is unlikely to change), the
performance impact is significant since most of the time is used up by
latency, the CORS preflight request essentially double the time it takes to
do anything

On 19 February 2015 at 10:50, Brian Smith br...@briansmith.org wrote:

 On Thu, Feb 19, 2015 at 2:45 AM, Anne van Kesteren ann...@annevk.nl
 wrote:
  On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org
 wrote:
  1. Preflight is only necessary for a subset of CORS requests.
  Preflight is never done for GET or HEAD, and you can avoid preflight
  for POST requests by making your API accept data in a format that
  matches what HTML forms post. Therefore, we're only talking about PUT,
  DELETE, less common forms of POST, and other less commonly-used
  methods.
 
  Euh, if you completely ignore headers, sure. But most HTTP APIs will
  use some amount of custom headers, meaning *all* methods require a
  preflight.

 Is it really true that most HTTP APIs will sue some amount of custom
 headers? And, is is it necessary for these APIs to be designed such
 that the custom headers are required?

 Cheers,
 Brian



Re: CORS performance

2015-02-19 Thread Brian Smith
Dale Harvey d...@arandomurl.com wrote:
 The REST api pretty much by design means a unique url per request

CouchDB has http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API,
which allows you to fetch or edit and create multiple documents at
once, with one HTTP request. CouchDB's documentation says you're
supposed to POST a JSON document for editing, but the example doesn't
set the Content-Type on the request so presumably it is OK to set the
Content-Type to text/plain. This means that you'd have ONE request and
ZERO preflights to edit N documents.

 in this case a lot of the requests look like

   GET origin/_change?since=0
   GET origin/_change?since=the last id

A GET like this won't require preflight unless you set custom header
fields on the request. Are you setting custom headers? If so, which
ones and why? I looked at the CouchDB documentation and it doesn't
mention any custom header fields. Thus, it seems to me like none of
the GET requests should require preflight.

Also, if your server is SPDY or HTTP/2, you should be able to
configure it so that when the server receives a request GET
/whatever/123, it replies with the response for that request AND
pushes the response for the not-even-yet-sent OPTIONS /whatever/123
request. In that case, even if you don't use the preflight-less bulk
document API and insist on using PUT, there's zero added latency from
the preflight.

Cheers,
Brian



Re: CORS performance

2015-02-19 Thread Mike West
On Tue, Feb 17, 2015 at 8:43 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote:

 * Anne van Kesteren wrote:
 On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net
 wrote:
  Individual resources should not be able to declare policy for the whole
  server, ...
 
 With HSTS we gave up on that.

 Well, HSTS essentially removes communication options, while the intent
 of CORS is to add communication options. I don't think you can compare
 them like that. HSTS is more like a redirect and misconfiguration may
 result in denial of service, while CORS misconfiguration can have more
 far-reaching consequences like exposing user information.


I share this concern. Note that CSP pinning as we're discussing it is also
purely negative in nature. It can block you from loading resources you'd
otherwise have access to, but can't force your host into exposing resources
you otherwise wouldn't.

Brad's .well-known suggestion is interesting. I'm worried about the latency
impacts, but it's probably worth exploring what it would take to add this
kind of thing to the Manifest spec (or some same-origin-limited version
thereof).

-mike

--
Mike West mk...@google.com, @mikewest

Google Germany GmbH, Dienerstrasse 12, 80331 München,
Germany, Registergericht und -nummer: Hamburg, HRB 86891, Sitz der
Gesellschaft: Hamburg, Geschäftsführer: Graham Law, Christine Elizabeth
Flores
(Sorry; I'm legally required to add this exciting detail to emails. Bleh.)


Re: CORS performance

2015-02-19 Thread Anne van Kesteren
On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote:
 1. Preflight is only necessary for a subset of CORS requests.
 Preflight is never done for GET or HEAD, and you can avoid preflight
 for POST requests by making your API accept data in a format that
 matches what HTML forms post. Therefore, we're only talking about PUT,
 DELETE, less common forms of POST, and other less commonly-used
 methods.

Euh, if you completely ignore headers, sure. But most HTTP APIs will
use some amount of custom headers, meaning *all* methods require a
preflight.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread Dale Harvey
 What is it about PouchDB and CouchDB that causes them to require
 preflight for all of these requests in the first place? What is
 difficult about changing them to not require preflight for all of
 these requests?

The REST api pretty much by design means a unique url per request, in this
case a lot of the requests look like

  GET origin/_change?since=0
  GET origin/_change?since=the last id

Its unlikely to change since its 10 years old across standardized across
several different products that works well in most cases aside for just
being kinda slow when you try to use it over CORS.

 If declaring this policy through a header is not acceptable, we could
 attempt a double preflight fetch for the very first CORS fetch against
 an origin (that requires a preflight). Try OPTIONS * before OPTIONS
 /actual-request. If that handshake succeeds (details TBD) no more
 preflights necessary for the entire origin.

This is very much what I expected when I first used CORS, similiar to the
flash cross-domain.xml file, I would just like to mark an origin I control
as being accessible from any host, as the only things CORS protects is data
behind a firewall I think it should be a simple mechanism to say this
domain is not behind a firewall, have at it


On 19 February 2015 at 11:30, Brian Smith br...@briansmith.org wrote:

 Dale Harvey d...@arandomurl.com wrote:
  With Couch / PouchDB we are working with an existing REST API wherein
 every
  request is to a different url (which is unlikely to change), the
 performance
  impact is significant since most of the time is used up by latency, the
 CORS
  preflight request essentially double the time it takes to do anything

 I understand that currently the cost of this API is 2*N and you want
 to reduce the 2 to 1 instead of reducing the N, even though N is
 usually much larger than 2.

 What is it about PouchDB and CouchDB that causes them to require
 preflight for all of these requests in the first place? What is
 difficult about changing them to not require preflight for all of
 these requests?

 Cheers,
 Brian



Re: CORS performance

2015-02-19 Thread Anne van Kesteren
On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote:
 With Couch / PouchDB we are working with an existing REST API wherein every
 request is to a different url (which is unlikely to change), the performance
 impact is significant since most of the time is used up by latency, the CORS
 preflight request essentially double the time it takes to do anything

Yeah, also, it should not be up to us how people design their HTTP
APIs. Limiting HTTP in that way because it is hard to make CORS scale
seems bad.


I think we've been too conservative when introducing CORS. It's
effectively protecting content behind a firewall, but we added all
these additional opt in mechanism beyond protecting content behind a
firewall due to unease about the potential risks. Figuring out to what
extent that actually serves a purpose would be good.

If declaring this policy through a header is not acceptable, we could
attempt a double preflight fetch for the very first CORS fetch against
an origin (that requires a preflight). Try OPTIONS * before OPTIONS
/actual-request. If that handshake succeeds (details TBD) no more
preflights necessary for the entire origin.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread Brian Smith
On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote:
 so presumably it is OK to set the Content-Type to text/plain

 Thats not ok, but may explain my confusion, is Content-Type considered a
 Custom Header that will always trigger a preflight?

To be clear, my comment was about POST requests to the bulk document
API, not about other requests.

I ran your demo and observed the network traffic using Wireshark.
Indeed, OPTIONS requests are being sent for every GET. But, that is
because you are setting the Content-Type header field on your GET
requests. Since GET requests don't have a request body, you shouldn't
set the Content-Type header field on them. And, if you do, then
browsers will treat it as a custom header field. That is what forces
the preflight for those requests.

Compare the network traffic for these two scripts:

  script
xhr=new XMLHttpRequest();
xhr.open(GET,
http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4;,
true);
xhr.setRequestHeader(Accept,application/json);
xhr.setRequestHeader(Content-Type,application/json);
xhr.send();
  /script

  script
xhr=new XMLHttpRequest();
xhr.open(GET,
http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4;,
true);
xhr.setRequestHeader(Accept,application/json);
xhr.send();
  /script

They are the same, except the second one doesn't set the Content-Type
header, and thus it doesn't cause the preflight to be sent.

 if so then none of the
 caching will apply, CouchDB requires sending the appropriate content-type

CouchDB may require sending Accept: application/json, but that isn't
considered a custom header field, so it doesn't trigger preflight.

 The /_changes requests are only part of the problem, once we receive the
 changes information we then have to request information about individual
 documents which all have a unique id

   GET /registry/mypackagename

 We do one of those per document (70,000 npm docs), all trigger a preflight
 (whether or not custom headers are involved)

I believe none of these require preflight unless a mistake is being
made (probably setting Content-Type on GET requests).

Also, regardless, you can use the CouchDB bulk document API to fetch
all these documents in one request, instead of 70,000 requests.

 Also performance details aside every week somebody has a library or proxy
 that sends some custom header or they just missed a step when configuring
 CORS, its a constant source of confusion for our users. We try to get around
 it by providing helper scripts but Anne's proposal mirroring flashes cross
 domain.xml sounds vastly superior to the current implementation from the
 developers perspective.

I agree that things can be improved here. I think the solution may be
better developer tools. In particular, devtools should tell you
exactly why a request triggered preflight.

Cheers,
Brian



Re: CORS performance

2015-02-19 Thread Dale Harvey
Will take a look at the content-type on GET requests, thanks

 I believe none of these require preflight unless a mistake is being
 made (probably setting Content-Type on GET requests).

http://www.w3.org/TR/cors/#preflight-result-cache-0

If the cache is against the url, and we are sending requests to different
urls, wont requests to different urls always trigger a preflight?

 Also, regardless, you can use the CouchDB bulk document API to fetch
 all these documents in one request, instead of 70,000 requests.

CouchDB has no bulk document fetch api, it has all_docs but that isnt
appropriate for this case, there is a talk about introducing it
https://issues.apache.org/jira/browse/COUCHDB-2310, however its going to
take a while (I would personally rather we replace it with a streaming api)

 I agree that things can be improved here. I think the solution may be
 better developer tools. In particular, devtools should tell you
 exactly why a request triggered preflight.

Whats wrong with 'This origin is part of the public internet and doesnt
need any complications or restrictions due to CORS' ie Anne proposal?


On 19 February 2015 at 13:21, Brian Smith br...@briansmith.org wrote:

 On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote:
  so presumably it is OK to set the Content-Type to text/plain
 
  Thats not ok, but may explain my confusion, is Content-Type considered a
  Custom Header that will always trigger a preflight?

 To be clear, my comment was about POST requests to the bulk document
 API, not about other requests.

 I ran your demo and observed the network traffic using Wireshark.
 Indeed, OPTIONS requests are being sent for every GET. But, that is
 because you are setting the Content-Type header field on your GET
 requests. Since GET requests don't have a request body, you shouldn't
 set the Content-Type header field on them. And, if you do, then
 browsers will treat it as a custom header field. That is what forces
 the preflight for those requests.

 Compare the network traffic for these two scripts:

   script
 xhr=new XMLHttpRequest();
 xhr.open(GET,
 
 http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4
 ,
 true);
 xhr.setRequestHeader(Accept,application/json);
 xhr.setRequestHeader(Content-Type,application/json);
 xhr.send();
   /script

   script
 xhr=new XMLHttpRequest();
 xhr.open(GET,
 
 http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4
 ,
 true);
 xhr.setRequestHeader(Accept,application/json);
 xhr.send();
   /script

 They are the same, except the second one doesn't set the Content-Type
 header, and thus it doesn't cause the preflight to be sent.

  if so then none of the
  caching will apply, CouchDB requires sending the appropriate content-type

 CouchDB may require sending Accept: application/json, but that isn't
 considered a custom header field, so it doesn't trigger preflight.

  The /_changes requests are only part of the problem, once we receive the
  changes information we then have to request information about individual
  documents which all have a unique id
 
GET /registry/mypackagename
 
  We do one of those per document (70,000 npm docs), all trigger a
 preflight
  (whether or not custom headers are involved)

 I believe none of these require preflight unless a mistake is being
 made (probably setting Content-Type on GET requests).

 Also, regardless, you can use the CouchDB bulk document API to fetch
 all these documents in one request, instead of 70,000 requests.

  Also performance details aside every week somebody has a library or proxy
  that sends some custom header or they just missed a step when configuring
  CORS, its a constant source of confusion for our users. We try to get
 around
  it by providing helper scripts but Anne's proposal mirroring flashes
 cross
  domain.xml sounds vastly superior to the current implementation from the
  developers perspective.

 I agree that things can be improved here. I think the solution may be
 better developer tools. In particular, devtools should tell you
 exactly why a request triggered preflight.

 Cheers,
 Brian



Re: CORS performance

2015-02-19 Thread Brian Smith
Dale Harvey d...@arandomurl.com wrote:
 I believe none of these require preflight unless a mistake is being
 made (probably setting Content-Type on GET requests).

 http://www.w3.org/TR/cors/#preflight-result-cache-0

 If the cache is against the url, and we are sending requests to different
 urls, wont requests to different urls always trigger a preflight?

In general, if your GET requests don't set custom headers, preflight
isn't necessary, because CORS has an optimization for GET (and POST)
that avoids preflight, for exactly the cases like yours..

 Also, regardless, you can use the CouchDB bulk document API to fetch
 all these documents in one request, instead of 70,000 requests.

 CouchDB has no bulk document fetch api

Sorry. I was reading
http://docs.couchdb.org/en/latest/api/database/bulk-api.html#db-bulk-docs
and assumed it had been implemented already. But I see that maybe you
are trying to do something slightly different anyway with PouchDB.
Regardless, no preflight should be necessary for this and so Anne's
proposal won't help with it.

 I agree that things can be improved here. I think the solution may be
 better developer tools. In particular, devtools should tell you
 exactly why a request triggered preflight.

 Whats wrong with 'This origin is part of the public internet and doesnt need
 any complications or restrictions due to CORS' ie Anne proposal?

I didn't say anything was wrong with Anne's proposal. What I said is
that it would help to have somebody present a concrete example of
where it would be useful.

Cheers,
Brian



Re: CORS performance proposal

2015-02-19 Thread Dale Harvey
 The cache would be on a per requesting origin basis as per the headers
 above. The Origin and Access-Control-Allow-Origin would not take part
 in this exchange, to make it very clear what this is about.

I dont want to conflate what could be seperate proposals, but they seem
closely related, this would improve the situation for easing the number of
preflight requests to be made, however still requires servers to follow
what is a fairly complicated process of setting up the appropriate headers

What if we allowed one of the response fields to denote this url is on the
public internet, please dont bother with cors restrictions. This means the
process of setting up cors could be to ensure a single response returns
with the appropriate headers and servers no longer need to worry about
every possible headers clients can send to each particular url.

(Clients would have to set a custom header to ensure the preflight
optimisation was skipped I believe)

This would be very much in line with how it was implemented for flash -
http://www.adobe.com/devnet/articles/crossdomain_policy_file_spec.html


On 19 February 2015 at 13:29, Anne van Kesteren ann...@annevk.nl wrote:

 When the user agent is about to make its first preflight to an origin
 (timeout up to the user agent), it first makes a preflight that looks
 like:

   OPTIONS *
   Access-Control-Request-Origin-Wide-Cache: [origin]
   Access-Control-Request-Method: *
   Access-Control-Request-Headers: *

 If the response is

   2xx XX
   Access-Control-Allow-Origin-Wide-Cache: [origin]
   Access-Control-Allow-Methods: *
   Access-Control-Allow-Headers: *
   Access-Control-Max-Age: [max-age]

 then no more preflights will be made for the duration of [max-age] (or
 shortened per user agent preference). If the response includes

   Access-Control-Allow-Credentials: true

 the cache scope is increased to requests that include credentials.

 I think this has a reasonable tradeoff between security and opening up
 all the power of the HTTP APIs on the server without the performance
 hit. It still makes the developer very conscious about the various
 features involved.

 The cache would be on a per requesting origin basis as per the headers
 above. The Origin and Access-Control-Allow-Origin would not take part
 in this exchange, to make it very clear what this is about.

 (This does not affect Access-Control-Expose-Headers or any of the
 other headers required as part of non-preflight responses.)


 --
 https://annevankesteren.nl/




Re: CORS performance

2015-02-19 Thread Dale Harvey
 so presumably it is OK to set the Content-Type to text/plain

Thats not ok, but may explain my confusion, is Content-Type considered a
Custom Header that will always trigger a preflight? if so then none of the
caching will apply, CouchDB requires sending the appropriate content-type

I tried setting up a little demo here, it will replicate the npm registry
for 5 seconds - http://paste.pouchdb.com/paste/q8n610/#output

You can see in the network logs various OPTIONS requests for
http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4
http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=311limit=100_nonce=UIZRQHrUG1Gjbm6S
etc etc

The /_changes requests are only part of the problem, once we receive the
changes information we then have to request information about individual
documents which all have a unique id

  GET /registry/mypackagename

We do one of those per document (70,000 npm docs), all trigger a preflight
(whether or not custom headers are involved)

We can and are doing a lot of thing to try and improve performance / reduce
the number of HTTP requests, but for our particular case its dealing with
10 years of established server protocols, there isnt 'a server', theres at
least 10 server implementations across all platforms by various projects /
companies that all need / try to interoperate, we cant just make ad hoc
changes to the protocol to get around CORS limitations.

Also performance details aside every week somebody has a library or proxy
that sends some custom header or they just missed a step when configuring
CORS, its a constant source of confusion for our users. We try to get
around it by providing helper scripts but Anne's proposal mirroring flashes
cross domain.xml sounds vastly superior to the current implementation from
the developers perspective.

On 19 February 2015 at 12:05, Brian Smith br...@briansmith.org wrote:

 Dale Harvey d...@arandomurl.com wrote:
  The REST api pretty much by design means a unique url per request

 CouchDB has http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API,
 which allows you to fetch or edit and create multiple documents at
 once, with one HTTP request. CouchDB's documentation says you're
 supposed to POST a JSON document for editing, but the example doesn't
 set the Content-Type on the request so presumably it is OK to set the
 Content-Type to text/plain. This means that you'd have ONE request and
 ZERO preflights to edit N documents.

  in this case a lot of the requests look like
 
GET origin/_change?since=0
GET origin/_change?since=the last id

 A GET like this won't require preflight unless you set custom header
 fields on the request. Are you setting custom headers? If so, which
 ones and why? I looked at the CouchDB documentation and it doesn't
 mention any custom header fields. Thus, it seems to me like none of
 the GET requests should require preflight.

 Also, if your server is SPDY or HTTP/2, you should be able to
 configure it so that when the server receives a request GET
 /whatever/123, it replies with the response for that request AND
 pushes the response for the not-even-yet-sent OPTIONS /whatever/123
 request. In that case, even if you don't use the preflight-less bulk
 document API and insist on using PUT, there's zero added latency from
 the preflight.

 Cheers,
 Brian



CORS performance proposal

2015-02-19 Thread Anne van Kesteren
When the user agent is about to make its first preflight to an origin
(timeout up to the user agent), it first makes a preflight that looks
like:

  OPTIONS *
  Access-Control-Request-Origin-Wide-Cache: [origin]
  Access-Control-Request-Method: *
  Access-Control-Request-Headers: *

If the response is

  2xx XX
  Access-Control-Allow-Origin-Wide-Cache: [origin]
  Access-Control-Allow-Methods: *
  Access-Control-Allow-Headers: *
  Access-Control-Max-Age: [max-age]

then no more preflights will be made for the duration of [max-age] (or
shortened per user agent preference). If the response includes

  Access-Control-Allow-Credentials: true

the cache scope is increased to requests that include credentials.

I think this has a reasonable tradeoff between security and opening up
all the power of the HTTP APIs on the server without the performance
hit. It still makes the developer very conscious about the various
features involved.

The cache would be on a per requesting origin basis as per the headers
above. The Origin and Access-Control-Allow-Origin would not take part
in this exchange, to make it very clear what this is about.

(This does not affect Access-Control-Expose-Headers or any of the
other headers required as part of non-preflight responses.)


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread James M Snell
On Feb 19, 2015 3:33 AM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote:
  With Couch / PouchDB we are working with an existing REST API wherein
every
  request is to a different url (which is unlikely to change), the
performance
  impact is significant since most of the time is used up by latency, the
CORS
  preflight request essentially double the time it takes to do anything

 Yeah, also, it should not be up to us how people design their HTTP
 APIs. Limiting HTTP in that way because it is hard to make CORS scale
 seems bad.



+1. Forcing developers to change their APIs would be bad form at this
stage. Not to mention just plain silly.

Optimizing with an OPTIONS * preflight is a good option but won't be as
broadly available to developers as a response header. Perhaps another
approach would be to allow a resource to declare a CORS policy only for
subordinate resources, rather than the entire origin.

For instance, an OPTIONS sent to http://example.org/api/ can return CORS
headers that cover every URL prefixed with http://example.org/api/. That
would logically extend all the way up to OPTIONS * in order to set a policy
that covers the entire origin.

- James

 I think we've been too conservative when introducing CORS. It's
 effectively protecting content behind a firewall, but we added all
 these additional opt in mechanism beyond protecting content behind a
 firewall due to unease about the potential risks. Figuring out to what
 extent that actually serves a purpose would be good.

 If declaring this policy through a header is not acceptable, we could
 attempt a double preflight fetch for the very first CORS fetch against
 an origin (that requires a preflight). Try OPTIONS * before OPTIONS
 /actual-request. If that handshake succeeds (details TBD) no more
 preflights necessary for the entire origin.


 --
 https://annevankesteren.nl/



Re: CORS performance

2015-02-19 Thread Dale Harvey
 If the cache is against the url, and we are sending requests to different
urls, wont
 requests to different urls always trigger a preflight?

I just realised my mistake, GETS without custom headers should need to
trigger preflight requests, sorry

On 19 February 2015 at 13:31, Dale Harvey d...@arandomurl.com wrote:

 Will take a look at the content-type on GET requests, thanks

  I believe none of these require preflight unless a mistake is being
  made (probably setting Content-Type on GET requests).

 http://www.w3.org/TR/cors/#preflight-result-cache-0

 If the cache is against the url, and we are sending requests to different
 urls, wont requests to different urls always trigger a preflight?

  Also, regardless, you can use the CouchDB bulk document API to fetch
  all these documents in one request, instead of 70,000 requests.

 CouchDB has no bulk document fetch api, it has all_docs but that isnt
 appropriate for this case, there is a talk about introducing it
 https://issues.apache.org/jira/browse/COUCHDB-2310, however its going to
 take a while (I would personally rather we replace it with a streaming api)

  I agree that things can be improved here. I think the solution may be
  better developer tools. In particular, devtools should tell you
  exactly why a request triggered preflight.

 Whats wrong with 'This origin is part of the public internet and doesnt
 need any complications or restrictions due to CORS' ie Anne proposal?


 On 19 February 2015 at 13:21, Brian Smith br...@briansmith.org wrote:

 On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote:
  so presumably it is OK to set the Content-Type to text/plain
 
  Thats not ok, but may explain my confusion, is Content-Type considered a
  Custom Header that will always trigger a preflight?

 To be clear, my comment was about POST requests to the bulk document
 API, not about other requests.

 I ran your demo and observed the network traffic using Wireshark.
 Indeed, OPTIONS requests are being sent for every GET. But, that is
 because you are setting the Content-Type header field on your GET
 requests. Since GET requests don't have a request body, you shouldn't
 set the Content-Type header field on them. And, if you do, then
 browsers will treat it as a custom header field. That is what forces
 the preflight for those requests.

 Compare the network traffic for these two scripts:

   script
 xhr=new XMLHttpRequest();
 xhr.open(GET,
 
 http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4
 ,
 true);
 xhr.setRequestHeader(Accept,application/json);
 xhr.setRequestHeader(Content-Type,application/json);
 xhr.send();
   /script

   script
 xhr=new XMLHttpRequest();
 xhr.open(GET,
 
 http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4
 ,
 true);
 xhr.setRequestHeader(Accept,application/json);
 xhr.send();
   /script

 They are the same, except the second one doesn't set the Content-Type
 header, and thus it doesn't cause the preflight to be sent.

  if so then none of the
  caching will apply, CouchDB requires sending the appropriate
 content-type

 CouchDB may require sending Accept: application/json, but that isn't
 considered a custom header field, so it doesn't trigger preflight.

  The /_changes requests are only part of the problem, once we receive the
  changes information we then have to request information about individual
  documents which all have a unique id
 
GET /registry/mypackagename
 
  We do one of those per document (70,000 npm docs), all trigger a
 preflight
  (whether or not custom headers are involved)

 I believe none of these require preflight unless a mistake is being
 made (probably setting Content-Type on GET requests).

 Also, regardless, you can use the CouchDB bulk document API to fetch
 all these documents in one request, instead of 70,000 requests.

  Also performance details aside every week somebody has a library or
 proxy
  that sends some custom header or they just missed a step when
 configuring
  CORS, its a constant source of confusion for our users. We try to get
 around
  it by providing helper scripts but Anne's proposal mirroring flashes
 cross
  domain.xml sounds vastly superior to the current implementation from the
  developers perspective.

 I agree that things can be improved here. I think the solution may be
 better developer tools. In particular, devtools should tell you
 exactly why a request triggered preflight.

 Cheers,
 Brian





Re: CORS performance

2015-02-19 Thread henry.st...@bblfish.net

 On 19 Feb 2015, at 22:04, Martin Thomson martin.thom...@gmail.com wrote:
 
 On 18 February 2015 at 06:31, Brad Hill hillb...@gmail.com wrote:
 Some of the things that argue against /.well-known are:
 
 1) Added latency of fetching the resource.
 
 It's not available everywhere yet, but you could push it, based on the below.
 
 2) Clients hammering servers for non-existent /.well-known resources (the
 favicon issue)
 
 You could avoid that by Link:-ing to the /.well-known and only hitting
 it if the link appears.

I assume you mean the Link: header. In that case I like the idea.

Well the client could even cache the document and only hit it once for the whole
server. Furthermore there would then be no need for the url to be in a 
.well-known
location. It could be any resource whatsoever. 

That is the way that the Web Access Control system functions. See link from 
this page

http://www.w3.org/2005/Incubator/webid/spec/

Every resource has a link to those resources that require access.
Aslo see the curl examples 
https://github.com/read-write-web/rww-play/wiki/Curl-Interactions


Henry

Social Web Architect
http://bblfish.net/




Re: CORS performance

2015-02-19 Thread Bjoern Hoehrmann
* Jonas Sicking wrote:
We most likely can consider the content-type header as *not* custom.
I was one of the people way back when that pointed out that there's a
theoretical chance that allowing arbitrary content-type headers could
cause security issues. But it seems highly theoretical.

I suspect that the mozilla security team would be fine with allowing
arbitrary content-types to be POSTed though. Worth asking. I can't
speak for other browser vendors of course.

I think the situation might well be worse now than it was when we first
started discussing what is now CORS. In any case, this would be an ex-
periment that cannot easily be undone, browser vendors would not pay the
bill if there are actually large scale security vulnerabilities opened
up by such a change, and I do not really see notable benefits in con-
ducting such an experiment.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/ 



Re: CORS performance proposal

2015-02-19 Thread Brian Smith
On Thu, Feb 19, 2015 at 5:29 AM, Anne van Kesteren ann...@annevk.nl wrote:
 When the user agent is about to make its first preflight to an origin
 (timeout up to the user agent), it first makes a preflight that looks
 like:

   OPTIONS *
   Access-Control-Request-Origin-Wide-Cache: [origin]
   Access-Control-Request-Method: *
   Access-Control-Request-Headers: *

This would make CORS preflight even slower for every server that
doesn't implement the new proposal (i.e. every currently-deployed
server and probably most servers deployed in the future). Perhaps the
OPTIONS * request could be made in parallel with the initial
preflight requests.

But, then, what happens when the information in the OPTIONS *
response conflicts with the information in the normal preflight
request?

 I think this has a reasonable tradeoff between security and opening up
 all the power of the HTTP APIs on the server without the performance
 hit. It still makes the developer very conscious about the various
 features involved.

I think developer consciousness is exactly the issue here:

1. Let's say you want to add OPTIONS * preflight support to an
existing web application. How do you go about finding all the things
that need to change to make that safe to do? It seems very difficult
to successfully find every place the app assumes it is protected by
the fact that it doesn't do CORS.

2. Similar to #1, let's say that two teams develop two parts of a
website. One of the teams follows normal CORS rules and the other
depends on the proposed OPTIONS * mechanism. This would be a
disaster if/when both apps are deployed on the same origin.

3. Because of these issues, an organization forces its developers to
develop every app as though every resource is CORS-enabled, to
future-proof against the scenerio where OPTIONS * is deployed in the
future. This makes the development of the web app more difficult and
slower.

4. In the discussion of Entry Point Regulation (EPR) on WebAppSec, the
main argument in favor of it is that it is impossible for developers
to do things like #3 correctly and it is unreasonable for us to expect
them to. I'm don't buy the EPR argument completely, but I do see some
merit in the underlying secure by default argument behind EPR.

Because of these concerns, I think that it would be worthwhile to
study a concrete example of the problem, to make sure we correctly
understand the use case we're trying to solve. As we saw yesterday
with the PouchDB/CouchDB example, it is easy to accidentally and
unnecessarily force a preflight. It may also be the case that we can
find other, safer, ways to avoid preflights and/or optimize how they
are done, such as by optimizing CORS for use with HTTP/2 server push
mechanisms. But, we need to see real instances of the problem first.

Cheers,
Brian



Re: CORS performance proposal

2015-02-19 Thread Bjoern Hoehrmann
* Martin Thomson wrote:
On 20 February 2015 at 00:29, Anne van Kesteren ann...@annevk.nl wrote:
   Access-Control-Allow-Origin-Wide-Cache: [origin]

This has some pretty implications for server deployments that host
mutual distrustful applications.  Now, these servers are already
pretty well hosed from other directions, but I don't believe that
there is any pre-existing case where a header field set in a request
to /x could affect future requests to /y.

An alternative would be to use /.well-known for site wide policies.

The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/ 



Re: CORS performance

2015-02-19 Thread Jonas Sicking
On Thu, Feb 19, 2015 at 12:38 PM, Brad Hill hillb...@gmail.com wrote:
 I think that POSTing JSON would probably expose to CSRF a lot of things that
 work over HTTP but don't expect to be interacted with by web browsers in
 that manner.  That's why the recent JSON encoding for forms mandates that it
 be same-origin only.

Note that you can already POST JSON cross-origin. Without any
preflight. The only thing you can't do is to set the Content-Type
header to the official JSON mimetype.

So the question is, does the server check that the Content-Type header
is set to application/json and if not abort any processing?

/ Jonas



Re: CORS performance proposal

2015-02-19 Thread Bjoern Hoehrmann
* Martin Thomson wrote:
On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`.

I missed that.  In which case I'd point out that `OPTIONS *` is very
poorly supported.  Some people (myself included) want it to die a
flaming death.

Evidence for poorly supported would certainly be helpful (web hosting
packages without TLS support, for instance, do not count, though).
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/ 



Re: CORS performance proposal

2015-02-19 Thread Martin Thomson
On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`.

I missed that.  In which case I'd point out that `OPTIONS *` is very
poorly supported.  Some people (myself included) want it to die a
flaming death.



Re: CORS performance

2015-02-17 Thread Eric Mill
On Tue, Feb 17, 2015 at 2:43 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote:

 * Anne van Kesteren wrote:
 On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net
 wrote:
  Individual resources should not be able to declare policy for the whole
  server, ...
 
 With HSTS we gave up on that.


FWIW, this dynamic is why you can't set HSTS on an S3 bucket (or a
CloudFront distribution backed by an S3 bucket). Amazon isn't willing to
let you set a HSTS header for a file that might also be served at
s3.amazonaws.com. And so any website backed by S3, even if you never use
the s3.amazonaws.com URLs, is restricted from setting HSTS headers.

-- Eric

-- 
konklone.com | @konklone https://twitter.com/konklone


Re: CORS performance

2015-02-17 Thread Devdatta Akhawe
+1 to Anne's suggestion. The current design is pretty terrible for API
performance. I think a request to / with OPTIONS or something, with a
response that requires some server side logic (like return the random
number UA just sent) is pretty darn secure.

cheers
dev


On 17 February 2015 at 11:24, Anne van Kesteren ann...@annevk.nl wrote:
 On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 Individual resources should not be able to declare policy for the whole
 server, ...

 With HSTS we gave up on that.


 HTTP/1.1 rather has `OPTIONS *` for that, which would require a
 new kind of preflight request. And if the whole server is fine with
 cross-origin requests, I am not sure there is much of a point trying to
 lock it down by restricting request headers or methods.

 Yeah, I wasn't sure whether those should all be listed. Maybe simply
 declaring you're fluent in CORS in a unique way is sufficient.


 --
 https://annevankesteren.nl/




CORS performance

2015-02-17 Thread Anne van Kesteren
Concerns raised by Monsur
https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0260.html
and others before him are still valid.

When you have an HTTP API on another origin you effectively get a huge
performance penalty. Even with caching of preflights, as each fetch is
likely to go to a distinct URL.

With the recent introduction of CSP pinning, I was wondering whether
something like CORS pinning would be feasible. A way for a server to
declare that it speaks CORS across an entire origin.

The CORS preflight in effect is a rather complicated way for the
server to announce that it can handle CORS. We made it rather tricky
to avoid footgun scenarios, but I'm wondering whether that is still
the right tradeoff.

Something like:

  CORS: max-age=31415926; allow-origin=*; allow-credentials=true;
allow-headers=*; allow-methods=*; expose-headers=*


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-17 Thread Brad Hill
On both this, and CSP pinning, I find myself getting nervous about adding
an increasing number of headers which, when sent by any resource, impact
the security posture and functioning of an entire origin.   HSTS and HPKP
are somewhat special in that: they convey only a few bits of information.
are somewhat self-verifying (e.g. an https pin cannot be applied over http,
and a public key pin cannot be applied from a certificate without a
matching key) and unlikely to change.

For more complex, application-layer issues, are pinning headers really the
best and most manageable approach?

I think it is at least worth discussing the relative merits of using a
resource published under /.well-known for such use cases, vs. sending
pinned headers with every single resource.

Some of the things that argue against /.well-known are:

1) Added latency of fetching the resource.

For CORS pinning, this probably isn't an issue.  A client that has never
seen a pinned policy (by whatever mechanism it is delivered) will always
have to make a preflight, and servers will always have to respond
appropriately (rather than relying on the pinned policy) for legacy
clients.  So a client that knows about pinning can make a request to
/.well-known in parallel the first time it makes a CORS request to a given
host, and speed up all subsequent requests without any additional latency
on the first.

2) Clients hammering servers for non-existent /.well-known resources (the
favicon issue)

Again, probably not an issue for CORS, or not nearly as large an issue.
Clients can be configured to never ask for the CORS policy at an origin
unless/until they make a CORS request to that origin.  There is a question
of how long to treat a 404 as authoritative before re-polling, but it can
be fairly long without breaking anything.  We might also devise a hint
header that suggests a pinned policy is available, which would only be sent
along with preflight responses.


On disadvantages of headers:

1) Individual resources declaring policy for an entire origin can be
problematic and difficult to manage.

Doing this right, both to set a correct policy and police lower-privileged
resources from setting incorrect ones, in practice would require an
administrator to set up a filtering proxy in front of the entire origin,
which is more difficult and costly than simply locking down access to a
specific resource path to only administrators.

2) Lots of redundant chit-chat.

HTTP/2 header compression reduces the impact of this somewhat, and CORS
specifically could avoid sending pinning headers except on preflight
responses, but it does seem like a real concern.  How much long-term bloat
are we willing to accept on the network in order to save a few ms of
latency the first time you connect to a site?

-Brad



On Tue Feb 17 2015 at 10:42:20 AM Anne van Kesteren ann...@annevk.nl
wrote:

 Concerns raised by Monsur
 https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0260.html
 and others before him are still valid.

 When you have an HTTP API on another origin you effectively get a huge
 performance penalty. Even with caching of preflights, as each fetch is
 likely to go to a distinct URL.

 With the recent introduction of CSP pinning, I was wondering whether
 something like CORS pinning would be feasible. A way for a server to
 declare that it speaks CORS across an entire origin.

 The CORS preflight in effect is a rather complicated way for the
 server to announce that it can handle CORS. We made it rather tricky
 to avoid footgun scenarios, but I'm wondering whether that is still
 the right tradeoff.

 Something like:

   CORS: max-age=31415926; allow-origin=*; allow-credentials=true;
 allow-headers=*; allow-methods=*; expose-headers=*


 --
 https://annevankesteren.nl/




Re: CORS performance

2015-02-17 Thread Bjoern Hoehrmann
* Anne van Kesteren wrote:
With the recent introduction of CSP pinning, I was wondering whether
something like CORS pinning would be feasible. A way for a server to
declare that it speaks CORS across an entire origin.

The CORS preflight in effect is a rather complicated way for the
server to announce that it can handle CORS. We made it rather tricky
to avoid footgun scenarios, but I'm wondering whether that is still
the right tradeoff.

Something like:

  CORS: max-age=31415926; allow-origin=*; allow-credentials=true;
allow-headers=*; allow-methods=*; expose-headers=*

Individual resources should not be able to declare policy for the whole
server, HTTP/1.1 rather has `OPTIONS *` for that, which would require a
new kind of pre-flight request. And if the whole server is fine with
cross-origin requests, I am not sure there is much of a point trying to
lock it down by restricting request headers or methods. I suppose some-
thing like this could be implemented, but I don't think CORS pinning
is quite the right analogy.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/ 



Re: CORS performance

2015-02-17 Thread Anne van Kesteren
On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 Individual resources should not be able to declare policy for the whole
 server, ...

With HSTS we gave up on that.


 HTTP/1.1 rather has `OPTIONS *` for that, which would require a
 new kind of preflight request. And if the whole server is fine with
 cross-origin requests, I am not sure there is much of a point trying to
 lock it down by restricting request headers or methods.

Yeah, I wasn't sure whether those should all be listed. Maybe simply
declaring you're fluent in CORS in a unique way is sufficient.


-- 
https://annevankesteren.nl/



Re: CORS performance

2015-02-17 Thread Bjoern Hoehrmann
* Anne van Kesteren wrote:
On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 Individual resources should not be able to declare policy for the whole
 server, ...

With HSTS we gave up on that.

Well, HSTS essentially removes communication options, while the intent
of CORS is to add communication options. I don't think you can compare
them like that. HSTS is more like a redirect and misconfiguration may
result in denial of service, while CORS misconfiguration can have more
far-reaching consequences like exposing user information.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/