Re: CORS performance proposal
On 8 June 2015 at 21:30, Nottingham, Mark mnott...@akamai.com wrote: A header denoting site-wide metadata would work for this too, of course, if folks were comfortable with the security properties of doing that (as well as the potential response overhead). The security properties bother me a little. Alt-Svc is showing us that we can't just define a header field like that without some serious analysis.
Re: CORS performance proposal
Picking up an old thread... On 20 Feb 2015, at 12:54 pm, Bjoern Hoehrmann derhoe...@gmx.net wrote: * Martin Thomson wrote: On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote: The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`. I missed that. In which case I'd point out that `OPTIONS *` is very poorly supported. Some people (myself included) want it to die a flaming death. Evidence for poorly supported would certainly be helpful (web hosting packages without TLS support, for instance, do not count, though). FWIW, this is old but I think still true, at least regarding OPTIONS *: https://www.mnot.net/blog/2005/04/03/options I wrote up my thoughts about using OPTIONS after looking at it for many years: https://www.mnot.net/blog/2012/10/29/NO_OPTIONS Also - it's becoming more common for content providers to ask CDNs for CORS preflight caching. Since OPTIONS doesn't fit into HTTP caching, something custom has to be designed, which means dealing with all of the corner cases, interop issues, etc. Using a .well-known would avoid all of that, since it can be cached just like any other response. If there's concern about extra requests against origins that don't use this mechanism, I'd suggest defining a default cacheability for 404 responses on the resource (e.g., 3 hours). That said, I suspect that if this were defined, take-up would be good; it's much easier than supporting pre-flights. A header denoting site-wide metadata would work for this too, of course, if folks were comfortable with the security properties of doing that (as well as the potential response overhead). Cheers, -- Mark Nottinghamm...@akamai.comhttps://www.mnot.net/
Re: CORS performance proposal
On Tue, Jun 9, 2015 at 6:42 AM, Martin Thomson martin.thom...@gmail.com wrote: The security properties bother me a little. Alt-Svc is showing us that we can't just define a header field like that without some serious analysis. Same goes for a site-wide file. See crossdomain.xml. However, either coupled with credentials mode = omit seems okayish... Mark, do these CDN requests mention credentials? -- https://annevankesteren.nl/
Re: CORS performance proposal
On 9 Jun 2015, at 2:42 pm, Martin Thomson martin.thom...@gmail.com wrote: On 8 June 2015 at 21:30, Nottingham, Mark mnott...@akamai.com wrote: A header denoting site-wide metadata would work for this too, of course, if folks were comfortable with the security properties of doing that (as well as the potential response overhead). The security properties bother me a little. Alt-Svc is showing us that we can't just define a header field like that without some serious analysis. Indeed. Also, an intermediary cache (whether a proxy or a CDN) would need to monitor all of the headers sent back for a given origin to figure out the applicable policy, and rewrite responses accordingly. It wouldn't just work out of the box like a .well-known would. Cheers, -- Mark Nottinghamm...@akamai.comhttps://www.mnot.net/
Re: CORS performance proposal
On 9 Jun 2015, at 2:54 pm, Anne van Kesteren ann...@annevk.nl wrote: On Tue, Jun 9, 2015 at 6:42 AM, Martin Thomson martin.thom...@gmail.com wrote: The security properties bother me a little. Alt-Svc is showing us that we can't just define a header field like that without some serious analysis. Same goes for a site-wide file. See crossdomain.xml. However, either coupled with credentials mode = omit seems okayish... Mark, do these CDN requests mention credentials? Will look into it. Supporting without credentials (and leaving future extensibility for the possibility) would certainly be a good start. Cheers, -- Mark Nottinghamm...@akamai.comhttps://www.mnot.net/
Re: CORS performance
On Mon, Feb 23, 2015 at 12:42 PM, Jonas Sicking jo...@sicking.cc wrote: Do we have any data on how common it is for people to use CORS with credentials? My impression is that it's far less common than CORS without credentials. If that's the case then I think we'd get most of the functionality, with essentially none of the risk, by only allowing server-wide cookie-less preflights. But data would help for sure. The credentials issue is a great concern. I've seen two cases I've been able to exploit where the Origin header is always passed back to Access-Control-Allow-Origin with Access-Control-Allow-Credentials; not realizing this exposes CSRF tokens and other sensitive information. Though this is not so much data as anecdote. (One was on a company-internal project, one briefly appeared in a software library we are using in between stable releases.) It's not hard to find additional examples [1][2][3][4]. I've never used the credentials functionality of CORS, instead passing an Authorization header explicitly. I think we could survive without it. I also recently ran into the CORS performance issue. CORS seems very biased against hypermedia services, in that it doubles the roundtrip time to an already very chatty API design. To work around these two issues, I began work on a proxy to accept `message/http` POST requests which returns `message/http` responses (actually for now, application/json to avoid the full RFC7230 implementation). (Existing proxies solving this problem don't have the performance characteristics or flexibility, or both, desired around HTTP headers, hypermedia, and streaming.) The only downside is this bypasses the user-agent cache, but for many kinds of services the user-agent cache is rarely utilized (for instance, jQuery's AJAX actively disables caching). I suggest that any request I can make with this proxy (which does not itself verify CORS before passing the request), I should be able to make without the CORS preflight request. The semantics of the proposed header then becomes a declaration that This server is accessible from the Internet (as opposed to a request on an intranet server made from an Internet-served page); and would then allow the making of requests that never utilize stored user-agent data (credentials). (With the exception of the Date header, could this be called deterministic, since the same XHR call will always produce the same HTTP request?) This definition would have the following desirable effects: 1. HTTP requests become stateless 2. Removes the possibility of exposing sensitive, stored user information (a TRACE response, if honored, would not divulge anything the sender doesn't already know) 3. Brings performance in line with non-browser user-agents that don't require CORS checks 4. Eliminates the privacy and security concerns associated with CORS work-arounds, like JSONP. As seen in [5] [6] [7] This wouldn't introduce new security concerns, because by definition it would not allow requests that couldn't already be made by proxy. The `OPTIONS *` request seems appropriate for this; as it is defined to be a request for metadata about the server itself, which is all that is necessary. Austin Wright. [1] https://github.com/primiano/blink-gitcs/blob/0b5424070e3006102e0036deea1e2e263b871eaa/LayoutTests/http/tests/security/resources/cors-redirect.php [2] https://github.com/wohlig/sergybackend/blob/3df9e42dccf620e53253c746022fbe1e53a97a3a/application/views/json.php [3] https://github.com/AslakNiclasen/ProjektOpgave/blob/7d5de466398371ab1481973bf54074ab005fffdc/Kode/ad_controller/.htaccess2.txt [4] https://github.com/datfinesoul/starphleet-logviewer/blob/af42371daad52fa7e7ba759e94b20024077a30bb/nginx.conf [5] http://blog.javascripting.com/2015/01/17/dont-hassle-with-cors/ [6] https://jsonp.nodejitsu.com/ [7] https://github.com/Rob--W/cors-anywhere
Re: CORS performance
On Tue, Feb 24, 2015 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Feb 24, 2015 at 3:25 AM, Anne van Kesteren ann...@annevk.nl wrote: If we only do it for this, could we combine that feature with the existing preflight then? Support a Access-Control-Allow-Origin-Wide: true header or some such that's mutually exclusive with Access-Control-Allow-Credentials: true. I don't have opinions on this. I filed https://www.w3.org/Bugs/Public/show_bug.cgi?id=28099 It would still be great to see data, hear from web developers, and hear from other browsers. I feel like we don't really have enough to go on yet, but at least we have a somewhat concrete plan for the case that does not involve credentials. -- https://annevankesteren.nl/
Re: CORS performance
On Mon, Feb 23, 2015 at 8:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Feb 23, 2015 at 11:06 AM, Anne van Kesteren ann...@annevk.nl wrote: That combined with requiring to list the explicit origin has worked well for CORS so far. This could potentially help. I don't remember the details of how/why people screwed up with crosssite.xml. But if the problem was that people hosted multiple services on the same server and only thought of one of them when writing a policy, then this won't really help very much. http://www.jamesward.com/2009/11/08/how-bad-crossdomain-policies-expose-protected-data-to-malicious-applications/ seems to support that. Do we have any data on how common it is for people to use CORS with credentials? My impression is that it's far less common than CORS without credentials. I don't have data. It seems we don't have telemetry for this in Gecko. Anyone else? I would also suspect that Access-Control-Allow-Origin: * is more common. If that's the case then I think we'd get most of the functionality, with essentially none of the risk, by only allowing server-wide cookie-less preflights. If we only do it for this, could we combine that feature with the existing preflight then? Support a Access-Control-Allow-Origin-Wide: true header or some such that's mutually exclusive with Access-Control-Allow-Credentials: true. -- https://annevankesteren.nl/
Re: CORS performance
On Tue, Feb 24, 2015 at 3:25 AM, Anne van Kesteren ann...@annevk.nl wrote: If that's the case then I think we'd get most of the functionality, with essentially none of the risk, by only allowing server-wide cookie-less preflights. If we only do it for this, could we combine that feature with the existing preflight then? Support a Access-Control-Allow-Origin-Wide: true header or some such that's mutually exclusive with Access-Control-Allow-Credentials: true. I don't have opinions on this. / Jonas
Re: CORS performance
On Mon, Feb 23, 2015 at 7:15 AM, Henri Sivonen hsivo...@hsivonen.fi wrote: On Tue, Feb 17, 2015 at 9:31 PM, Brad Hill hillb...@gmail.com wrote: I think it is at least worth discussing the relative merits of using a resource published under /.well-known for such use cases, vs. sending pinned headers with every single resource. FWIW, when CORS was designed, the Flash crossdomain.xml design (which uses a well-known URL though not under /.well-known) already existed and CORS deliberately opted for a different design. It's been a while, so I don't recall what the reasons against adopting crossdomain.xml or something very similar to it were, but considering that the crossdomain.xml design was knowingly rejected, it's probably worthwhile to pay attention to why. A lot websites accidentally enabled cross-origin requests with cookies. Not realizing that that enabled attackers to make requests that had side-effects as well as read personal user data without user permission. In short, it was very easy to misconfigure a server, and people did. This is why I would feel dramatically more comfortable if we only enabled server-wide opt-in for credential-less requests. Those are many orders of magnitude easier to make secure. / Jonas
Re: CORS performance proposal
On Sat, Feb 21, 2015 at 11:18 PM, Anne van Kesteren ann...@annevk.nl wrote: On Sat, Feb 21, 2015 at 10:17 AM, Martin Thomson martin.thom...@gmail.com wrote: On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote: High-byte of what? A URL is within ASCII range when it reaches the server. This is the first time I hear of this. Apparently, all sorts of muck floats around the Internet. When we did HTTP/2 we were forced to accept that header field values (URLs in particular) were a sequence of octets. Those are often interpreted as strings in various interesting ways. But in this particular case it must be the browser that generates said muck, no? Other than Internet Explorer (and that's a couple versions ago, so wouldn't support this protocol anyway), there's no browser that does this as far as I know. All browsers support sending %xx stuff to the server. Decoding those is likely more often than not happening in a server-specific way still. Despite specs defining how they should do it. / Jonas
Re: CORS performance proposal
On Fri, Feb 20, 2015 at 11:43 PM, Anne van Kesteren ann...@annevk.nl wrote: On Fri, Feb 20, 2015 at 9:38 PM, Jonas Sicking jo...@sicking.cc wrote: On Fri, Feb 20, 2015 at 1:05 AM, Anne van Kesteren ann...@annevk.nl wrote: An alternative is that we attempt to introduce Access-Control-Policy-Path again from 2008. The problems you raised https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html seem surmountable. URL parsing is defined in more detail these days and we could simply ban URLs containing escaped \ and /. I do remember that another issue that came up back then was that servers would treat more than just '\', or the escaped version thereof, as a /. But also any character whose low-byte was equal to the ascii code for '\' or '/'. I.e. the server would just cut the high-byte when doing some internal 2byte-string to 1byte-string conversion. Potentially this conversion is affected by what character encodings the server is configured for too, but i'm less sure about that. High-byte of what? A URL is within ASCII range when it reaches the server. This is the first time I hear of this. I really don't remember the details. I'd recommend talking to microsoft since I believe they had done most research into this at the time. Keep in mind though that just because URL parsing is defined a particular way, doesn't mean that software implements it that way. / Jonas
Re: CORS performance
On Mon, Feb 23, 2015 at 11:06 AM, Anne van Kesteren ann...@annevk.nl wrote: On Mon, Feb 23, 2015 at 7:55 PM, Jonas Sicking jo...@sicking.cc wrote: A lot websites accidentally enabled cross-origin requests with cookies. Not realizing that that enabled attackers to make requests that had side-effects as well as read personal user data without user permission. In short, it was very easy to misconfigure a server, and people did. This is why I would feel dramatically more comfortable if we only enabled server-wide opt-in for credential-less requests. Those are many orders of magnitude easier to make secure. Why is that not served by requiring an additional header that explicitly opts into that case? I don't think an extra header is that much harder to deploy than crosssite.xml is. I.e. I don't see strong reasons to think that people won't misconfigure. That combined with requiring to list the explicit origin has worked well for CORS so far. This could potentially help. I don't remember the details of how/why people screwed up with crosssite.xml. But if the problem was that people hosted multiple services on the same server and only thought of one of them when writing a policy, then this won't really help very much. Do we have any data on how common it is for people to use CORS with credentials? My impression is that it's far less common than CORS without credentials. If that's the case then I think we'd get most of the functionality, with essentially none of the risk, by only allowing server-wide cookie-less preflights. But data would help for sure. / Jonas
Re: CORS performance
On Mon, Feb 23, 2015 at 7:55 PM, Jonas Sicking jo...@sicking.cc wrote: A lot websites accidentally enabled cross-origin requests with cookies. Not realizing that that enabled attackers to make requests that had side-effects as well as read personal user data without user permission. In short, it was very easy to misconfigure a server, and people did. This is why I would feel dramatically more comfortable if we only enabled server-wide opt-in for credential-less requests. Those are many orders of magnitude easier to make secure. Why is that not served by requiring an additional header that explicitly opts into that case? That combined with requiring to list the explicit origin has worked well for CORS so far. -- https://annevankesteren.nl/
Re: CORS performance proposal
On Sat, Feb 21, 2015 at 10:17 AM, Martin Thomson martin.thom...@gmail.com wrote: On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote: High-byte of what? A URL is within ASCII range when it reaches the server. This is the first time I hear of this. Apparently, all sorts of muck floats around the Internet. When we did HTTP/2 we were forced to accept that header field values (URLs in particular) were a sequence of octets. Those are often interpreted as strings in various interesting ways. But in this particular case it must be the browser that generates said muck, no? Other than Internet Explorer (and that's a couple versions ago, so wouldn't support this protocol anyway), there's no browser that does this as far as I know. I wouldn't *completely* discount the potential for the conversions Jonas mentions here. A Java server might parse UTF-8 into the internal UTF-16 representation and then who knows what happens next. There's no utf-8 either. -- https://annevankesteren.nl/
Re: CORS performance proposal
On 21 February 2015 at 20:43, Anne van Kesteren ann...@annevk.nl wrote: High-byte of what? A URL is within ASCII range when it reaches the server. This is the first time I hear of this. Apparently, all sorts of muck floats around the Internet. When we did HTTP/2 we were forced to accept that header field values (URLs in particular) were a sequence of octets. Those are often interpreted as strings in various interesting ways. I wouldn't *completely* discount the potential for the conversions Jonas mentions here. A Java server might parse UTF-8 into the internal UTF-16 representation and then who knows what happens next.
Re: CORS performance proposal
On Thu, Feb 19, 2015 at 9:22 PM, Jonas Sicking jo...@sicking.cc wrote: Would this be allowed for both requests with credentials and requests without credentials? The security implications of the two are very different. Yes, but the latter requires the Access-Control-Allow-Credentials header to be included in the response. An alternative is that we attempt to introduce Access-Control-Policy-Path again from 2008. The problems you raised https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html seem surmountable. URL parsing is defined in more detail these days and we could simply ban URLs containing escaped \ and /. -- https://annevankesteren.nl/
Re: CORS performance proposal
On Fri, Feb 20, 2015 at 9:38 PM, Jonas Sicking jo...@sicking.cc wrote: On Fri, Feb 20, 2015 at 1:05 AM, Anne van Kesteren ann...@annevk.nl wrote: An alternative is that we attempt to introduce Access-Control-Policy-Path again from 2008. The problems you raised https://lists.w3.org/Archives/Public/public-appformats/2008May/0037.html seem surmountable. URL parsing is defined in more detail these days and we could simply ban URLs containing escaped \ and /. I do remember that another issue that came up back then was that servers would treat more than just '\', or the escaped version thereof, as a /. But also any character whose low-byte was equal to the ascii code for '\' or '/'. I.e. the server would just cut the high-byte when doing some internal 2byte-string to 1byte-string conversion. Potentially this conversion is affected by what character encodings the server is configured for too, but i'm less sure about that. High-byte of what? A URL is within ASCII range when it reaches the server. This is the first time I hear of this. -- https://annevankesteren.nl/
Re: CORS performance
On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote: so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? if so then none of the caching will apply, CouchDB requires sending the appropriate content-type We most likely can consider the content-type header as *not* custom. I was one of the people way back when that pointed out that there's a theoretical chance that allowing arbitrary content-type headers could cause security issues. But it seems highly theoretical. I suspect that the mozilla security team would be fine with allowing arbitrary content-types to be POSTed though. Worth asking. I can't speak for other browser vendors of course. / Jonas
Re: CORS performance proposal
Would this be allowed for both requests with credentials and requests without credentials? The security implications of the two are very different. / Jonas On Thu, Feb 19, 2015 at 5:29 AM, Anne van Kesteren ann...@annevk.nl wrote: When the user agent is about to make its first preflight to an origin (timeout up to the user agent), it first makes a preflight that looks like: OPTIONS * Access-Control-Request-Origin-Wide-Cache: [origin] Access-Control-Request-Method: * Access-Control-Request-Headers: * If the response is 2xx XX Access-Control-Allow-Origin-Wide-Cache: [origin] Access-Control-Allow-Methods: * Access-Control-Allow-Headers: * Access-Control-Max-Age: [max-age] then no more preflights will be made for the duration of [max-age] (or shortened per user agent preference). If the response includes Access-Control-Allow-Credentials: true the cache scope is increased to requests that include credentials. I think this has a reasonable tradeoff between security and opening up all the power of the HTTP APIs on the server without the performance hit. It still makes the developer very conscious about the various features involved. The cache would be on a per requesting origin basis as per the headers above. The Origin and Access-Control-Allow-Origin would not take part in this exchange, to make it very clear what this is about. (This does not affect Access-Control-Expose-Headers or any of the other headers required as part of non-preflight responses.) -- https://annevankesteren.nl/
Re: CORS performance
On Thu, Feb 19, 2015 at 3:30 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote: With Couch / PouchDB we are working with an existing REST API wherein every request is to a different url (which is unlikely to change), the performance impact is significant since most of the time is used up by latency, the CORS preflight request essentially double the time it takes to do anything Yeah, also, it should not be up to us how people design their HTTP APIs. Limiting HTTP in that way because it is hard to make CORS scale seems bad. I think we've been too conservative when introducing CORS. It's effectively protecting content behind a firewall, ...and content that uses user credentials like cookies. / Jonas
Re: CORS performance
I think that POSTing JSON would probably expose to CSRF a lot of things that work over HTTP but don't expect to be interacted with by web browsers in that manner. That's why the recent JSON encoding for forms mandates that it be same-origin only. On Thu Feb 19 2015 at 12:23:48 PM Jonas Sicking jo...@sicking.cc wrote: On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote: so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? if so then none of the caching will apply, CouchDB requires sending the appropriate content-type We most likely can consider the content-type header as *not* custom. I was one of the people way back when that pointed out that there's a theoretical chance that allowing arbitrary content-type headers could cause security issues. But it seems highly theoretical. I suspect that the mozilla security team would be fine with allowing arbitrary content-types to be POSTed though. Worth asking. I can't speak for other browser vendors of course. / Jonas
Re: CORS performance proposal
On 20 February 2015 at 00:29, Anne van Kesteren ann...@annevk.nl wrote: Access-Control-Allow-Origin-Wide-Cache: [origin] This has some pretty implications for server deployments that host mutual distrustful applications. Now, these servers are already pretty well hosed from other directions, but I don't believe that there is any pre-existing case where a header field set in a request to /x could affect future requests to /y. An alternative would be to use /.well-known for site wide policies.
Re: CORS performance
On 18 February 2015 at 06:31, Brad Hill hillb...@gmail.com wrote: Some of the things that argue against /.well-known are: 1) Added latency of fetching the resource. It's not available everywhere yet, but you could push it, based on the below. 2) Clients hammering servers for non-existent /.well-known resources (the favicon issue) You could avoid that by Link:-ing to the /.well-known and only hitting it if the link appears.
Re: CORS performance
Anne van Kesteren ann...@annevk.nl wrote: Concerns raised by Monsur https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0260.html and others before him are still valid. When you have an HTTP API on another origin you effectively get a huge performance penalty. Even with caching of preflights, as each fetch is likely to go to a distinct URL. Definitely there is a huge performance penalty per-request when the preflight isn't cached. But: 1. Preflight is only necessary for a subset of CORS requests. Preflight is never done for GET or HEAD, and you can avoid preflight for POST requests by making your API accept data in a format that matches what HTML forms post. Therefore, we're only talking about PUT, DELETE, less common forms of POST, and other less commonly-used methods. 2. It seems very wasteful to design an API that requires multiple PUT/DELETE/POST requests to complete a transaction (I'm using the word transaction loosely here; I don't mean ACID, necessarily). A lot of people think that REST means that you should only change a resource by PUT/DELETE/POSTing to its URL, however that's a misunderstanding of what REST is really about. Regardless of CORS, it is a good idea to design APIs in such a way that every modification transaction can be done in one or as few requests as possible, and this can be done a way that is in line with RESTful design. This type of design is more-or-less required when you need ACID transactions anyway. 3. Often you can predict which resources need preflight. With HTTP/2 and SPDY, the server can use the server push mechanism to push preflight responses to the client before the client even sends them. Given #1, #2, and #3, I'm a little bit unsure how bad the performance problem really is, and how bad it will be going forward. It would be good to see a concrete example to get a better understanding of the issue. Cheers, Brian
Re: CORS performance
On Thu, Feb 19, 2015 at 11:45 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote: 1. Preflight is only necessary for a subset of CORS requests. Preflight is never done for GET or HEAD, and you can avoid preflight for POST requests by making your API accept data in a format that matches what HTML forms post. Therefore, we're only talking about PUT, DELETE, less common forms of POST, and other less commonly-used methods. Euh, if you completely ignore headers, sure. But most HTTP APIs will use some amount of custom headers, meaning *all* methods require a preflight. And you seem to forget that an HTTP API typically covers a large set of URLs (e.g. if you use remote database such as PouchDB). With CORS that requires at least one preflight per URL. -- https://annevankesteren.nl/
Re: CORS performance
On Thu, Feb 19, 2015 at 2:45 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote: 1. Preflight is only necessary for a subset of CORS requests. Preflight is never done for GET or HEAD, and you can avoid preflight for POST requests by making your API accept data in a format that matches what HTML forms post. Therefore, we're only talking about PUT, DELETE, less common forms of POST, and other less commonly-used methods. Euh, if you completely ignore headers, sure. But most HTTP APIs will use some amount of custom headers, meaning *all* methods require a preflight. Is it really true that most HTTP APIs will sue some amount of custom headers? And, is is it necessary for these APIs to be designed such that the custom headers are required? Cheers, Brian
Re: CORS performance
With Couch / PouchDB we are working with an existing REST API wherein every request is to a different url (which is unlikely to change), the performance impact is significant since most of the time is used up by latency, the CORS preflight request essentially double the time it takes to do anything On 19 February 2015 at 10:50, Brian Smith br...@briansmith.org wrote: On Thu, Feb 19, 2015 at 2:45 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote: 1. Preflight is only necessary for a subset of CORS requests. Preflight is never done for GET or HEAD, and you can avoid preflight for POST requests by making your API accept data in a format that matches what HTML forms post. Therefore, we're only talking about PUT, DELETE, less common forms of POST, and other less commonly-used methods. Euh, if you completely ignore headers, sure. But most HTTP APIs will use some amount of custom headers, meaning *all* methods require a preflight. Is it really true that most HTTP APIs will sue some amount of custom headers? And, is is it necessary for these APIs to be designed such that the custom headers are required? Cheers, Brian
Re: CORS performance
Dale Harvey d...@arandomurl.com wrote: The REST api pretty much by design means a unique url per request CouchDB has http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API, which allows you to fetch or edit and create multiple documents at once, with one HTTP request. CouchDB's documentation says you're supposed to POST a JSON document for editing, but the example doesn't set the Content-Type on the request so presumably it is OK to set the Content-Type to text/plain. This means that you'd have ONE request and ZERO preflights to edit N documents. in this case a lot of the requests look like GET origin/_change?since=0 GET origin/_change?since=the last id A GET like this won't require preflight unless you set custom header fields on the request. Are you setting custom headers? If so, which ones and why? I looked at the CouchDB documentation and it doesn't mention any custom header fields. Thus, it seems to me like none of the GET requests should require preflight. Also, if your server is SPDY or HTTP/2, you should be able to configure it so that when the server receives a request GET /whatever/123, it replies with the response for that request AND pushes the response for the not-even-yet-sent OPTIONS /whatever/123 request. In that case, even if you don't use the preflight-less bulk document API and insist on using PUT, there's zero added latency from the preflight. Cheers, Brian
Re: CORS performance
On Tue, Feb 17, 2015 at 8:43 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: * Anne van Kesteren wrote: On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: Individual resources should not be able to declare policy for the whole server, ... With HSTS we gave up on that. Well, HSTS essentially removes communication options, while the intent of CORS is to add communication options. I don't think you can compare them like that. HSTS is more like a redirect and misconfiguration may result in denial of service, while CORS misconfiguration can have more far-reaching consequences like exposing user information. I share this concern. Note that CSP pinning as we're discussing it is also purely negative in nature. It can block you from loading resources you'd otherwise have access to, but can't force your host into exposing resources you otherwise wouldn't. Brad's .well-known suggestion is interesting. I'm worried about the latency impacts, but it's probably worth exploring what it would take to add this kind of thing to the Manifest spec (or some same-origin-limited version thereof). -mike -- Mike West mk...@google.com, @mikewest Google Germany GmbH, Dienerstrasse 12, 80331 München, Germany, Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg, Geschäftsführer: Graham Law, Christine Elizabeth Flores (Sorry; I'm legally required to add this exciting detail to emails. Bleh.)
Re: CORS performance
On Thu, Feb 19, 2015 at 11:43 AM, Brian Smith br...@briansmith.org wrote: 1. Preflight is only necessary for a subset of CORS requests. Preflight is never done for GET or HEAD, and you can avoid preflight for POST requests by making your API accept data in a format that matches what HTML forms post. Therefore, we're only talking about PUT, DELETE, less common forms of POST, and other less commonly-used methods. Euh, if you completely ignore headers, sure. But most HTTP APIs will use some amount of custom headers, meaning *all* methods require a preflight. -- https://annevankesteren.nl/
Re: CORS performance
What is it about PouchDB and CouchDB that causes them to require preflight for all of these requests in the first place? What is difficult about changing them to not require preflight for all of these requests? The REST api pretty much by design means a unique url per request, in this case a lot of the requests look like GET origin/_change?since=0 GET origin/_change?since=the last id Its unlikely to change since its 10 years old across standardized across several different products that works well in most cases aside for just being kinda slow when you try to use it over CORS. If declaring this policy through a header is not acceptable, we could attempt a double preflight fetch for the very first CORS fetch against an origin (that requires a preflight). Try OPTIONS * before OPTIONS /actual-request. If that handshake succeeds (details TBD) no more preflights necessary for the entire origin. This is very much what I expected when I first used CORS, similiar to the flash cross-domain.xml file, I would just like to mark an origin I control as being accessible from any host, as the only things CORS protects is data behind a firewall I think it should be a simple mechanism to say this domain is not behind a firewall, have at it On 19 February 2015 at 11:30, Brian Smith br...@briansmith.org wrote: Dale Harvey d...@arandomurl.com wrote: With Couch / PouchDB we are working with an existing REST API wherein every request is to a different url (which is unlikely to change), the performance impact is significant since most of the time is used up by latency, the CORS preflight request essentially double the time it takes to do anything I understand that currently the cost of this API is 2*N and you want to reduce the 2 to 1 instead of reducing the N, even though N is usually much larger than 2. What is it about PouchDB and CouchDB that causes them to require preflight for all of these requests in the first place? What is difficult about changing them to not require preflight for all of these requests? Cheers, Brian
Re: CORS performance
On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote: With Couch / PouchDB we are working with an existing REST API wherein every request is to a different url (which is unlikely to change), the performance impact is significant since most of the time is used up by latency, the CORS preflight request essentially double the time it takes to do anything Yeah, also, it should not be up to us how people design their HTTP APIs. Limiting HTTP in that way because it is hard to make CORS scale seems bad. I think we've been too conservative when introducing CORS. It's effectively protecting content behind a firewall, but we added all these additional opt in mechanism beyond protecting content behind a firewall due to unease about the potential risks. Figuring out to what extent that actually serves a purpose would be good. If declaring this policy through a header is not acceptable, we could attempt a double preflight fetch for the very first CORS fetch against an origin (that requires a preflight). Try OPTIONS * before OPTIONS /actual-request. If that handshake succeeds (details TBD) no more preflights necessary for the entire origin. -- https://annevankesteren.nl/
Re: CORS performance
On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote: so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? To be clear, my comment was about POST requests to the bulk document API, not about other requests. I ran your demo and observed the network traffic using Wireshark. Indeed, OPTIONS requests are being sent for every GET. But, that is because you are setting the Content-Type header field on your GET requests. Since GET requests don't have a request body, you shouldn't set the Content-Type header field on them. And, if you do, then browsers will treat it as a custom header field. That is what forces the preflight for those requests. Compare the network traffic for these two scripts: script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4;, true); xhr.setRequestHeader(Accept,application/json); xhr.setRequestHeader(Content-Type,application/json); xhr.send(); /script script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4;, true); xhr.setRequestHeader(Accept,application/json); xhr.send(); /script They are the same, except the second one doesn't set the Content-Type header, and thus it doesn't cause the preflight to be sent. if so then none of the caching will apply, CouchDB requires sending the appropriate content-type CouchDB may require sending Accept: application/json, but that isn't considered a custom header field, so it doesn't trigger preflight. The /_changes requests are only part of the problem, once we receive the changes information we then have to request information about individual documents which all have a unique id GET /registry/mypackagename We do one of those per document (70,000 npm docs), all trigger a preflight (whether or not custom headers are involved) I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. Also performance details aside every week somebody has a library or proxy that sends some custom header or they just missed a step when configuring CORS, its a constant source of confusion for our users. We try to get around it by providing helper scripts but Anne's proposal mirroring flashes cross domain.xml sounds vastly superior to the current implementation from the developers perspective. I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Cheers, Brian
Re: CORS performance
Will take a look at the content-type on GET requests, thanks I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). http://www.w3.org/TR/cors/#preflight-result-cache-0 If the cache is against the url, and we are sending requests to different urls, wont requests to different urls always trigger a preflight? Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. CouchDB has no bulk document fetch api, it has all_docs but that isnt appropriate for this case, there is a talk about introducing it https://issues.apache.org/jira/browse/COUCHDB-2310, however its going to take a while (I would personally rather we replace it with a streaming api) I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Whats wrong with 'This origin is part of the public internet and doesnt need any complications or restrictions due to CORS' ie Anne proposal? On 19 February 2015 at 13:21, Brian Smith br...@briansmith.org wrote: On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote: so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? To be clear, my comment was about POST requests to the bulk document API, not about other requests. I ran your demo and observed the network traffic using Wireshark. Indeed, OPTIONS requests are being sent for every GET. But, that is because you are setting the Content-Type header field on your GET requests. Since GET requests don't have a request body, you shouldn't set the Content-Type header field on them. And, if you do, then browsers will treat it as a custom header field. That is what forces the preflight for those requests. Compare the network traffic for these two scripts: script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4 , true); xhr.setRequestHeader(Accept,application/json); xhr.setRequestHeader(Content-Type,application/json); xhr.send(); /script script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4 , true); xhr.setRequestHeader(Accept,application/json); xhr.send(); /script They are the same, except the second one doesn't set the Content-Type header, and thus it doesn't cause the preflight to be sent. if so then none of the caching will apply, CouchDB requires sending the appropriate content-type CouchDB may require sending Accept: application/json, but that isn't considered a custom header field, so it doesn't trigger preflight. The /_changes requests are only part of the problem, once we receive the changes information we then have to request information about individual documents which all have a unique id GET /registry/mypackagename We do one of those per document (70,000 npm docs), all trigger a preflight (whether or not custom headers are involved) I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. Also performance details aside every week somebody has a library or proxy that sends some custom header or they just missed a step when configuring CORS, its a constant source of confusion for our users. We try to get around it by providing helper scripts but Anne's proposal mirroring flashes cross domain.xml sounds vastly superior to the current implementation from the developers perspective. I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Cheers, Brian
Re: CORS performance
Dale Harvey d...@arandomurl.com wrote: I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). http://www.w3.org/TR/cors/#preflight-result-cache-0 If the cache is against the url, and we are sending requests to different urls, wont requests to different urls always trigger a preflight? In general, if your GET requests don't set custom headers, preflight isn't necessary, because CORS has an optimization for GET (and POST) that avoids preflight, for exactly the cases like yours.. Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. CouchDB has no bulk document fetch api Sorry. I was reading http://docs.couchdb.org/en/latest/api/database/bulk-api.html#db-bulk-docs and assumed it had been implemented already. But I see that maybe you are trying to do something slightly different anyway with PouchDB. Regardless, no preflight should be necessary for this and so Anne's proposal won't help with it. I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Whats wrong with 'This origin is part of the public internet and doesnt need any complications or restrictions due to CORS' ie Anne proposal? I didn't say anything was wrong with Anne's proposal. What I said is that it would help to have somebody present a concrete example of where it would be useful. Cheers, Brian
Re: CORS performance proposal
The cache would be on a per requesting origin basis as per the headers above. The Origin and Access-Control-Allow-Origin would not take part in this exchange, to make it very clear what this is about. I dont want to conflate what could be seperate proposals, but they seem closely related, this would improve the situation for easing the number of preflight requests to be made, however still requires servers to follow what is a fairly complicated process of setting up the appropriate headers What if we allowed one of the response fields to denote this url is on the public internet, please dont bother with cors restrictions. This means the process of setting up cors could be to ensure a single response returns with the appropriate headers and servers no longer need to worry about every possible headers clients can send to each particular url. (Clients would have to set a custom header to ensure the preflight optimisation was skipped I believe) This would be very much in line with how it was implemented for flash - http://www.adobe.com/devnet/articles/crossdomain_policy_file_spec.html On 19 February 2015 at 13:29, Anne van Kesteren ann...@annevk.nl wrote: When the user agent is about to make its first preflight to an origin (timeout up to the user agent), it first makes a preflight that looks like: OPTIONS * Access-Control-Request-Origin-Wide-Cache: [origin] Access-Control-Request-Method: * Access-Control-Request-Headers: * If the response is 2xx XX Access-Control-Allow-Origin-Wide-Cache: [origin] Access-Control-Allow-Methods: * Access-Control-Allow-Headers: * Access-Control-Max-Age: [max-age] then no more preflights will be made for the duration of [max-age] (or shortened per user agent preference). If the response includes Access-Control-Allow-Credentials: true the cache scope is increased to requests that include credentials. I think this has a reasonable tradeoff between security and opening up all the power of the HTTP APIs on the server without the performance hit. It still makes the developer very conscious about the various features involved. The cache would be on a per requesting origin basis as per the headers above. The Origin and Access-Control-Allow-Origin would not take part in this exchange, to make it very clear what this is about. (This does not affect Access-Control-Expose-Headers or any of the other headers required as part of non-preflight responses.) -- https://annevankesteren.nl/
Re: CORS performance
so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? if so then none of the caching will apply, CouchDB requires sending the appropriate content-type I tried setting up a little demo here, it will replicate the npm registry for 5 seconds - http://paste.pouchdb.com/paste/q8n610/#output You can see in the network logs various OPTIONS requests for http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4 http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=311limit=100_nonce=UIZRQHrUG1Gjbm6S etc etc The /_changes requests are only part of the problem, once we receive the changes information we then have to request information about individual documents which all have a unique id GET /registry/mypackagename We do one of those per document (70,000 npm docs), all trigger a preflight (whether or not custom headers are involved) We can and are doing a lot of thing to try and improve performance / reduce the number of HTTP requests, but for our particular case its dealing with 10 years of established server protocols, there isnt 'a server', theres at least 10 server implementations across all platforms by various projects / companies that all need / try to interoperate, we cant just make ad hoc changes to the protocol to get around CORS limitations. Also performance details aside every week somebody has a library or proxy that sends some custom header or they just missed a step when configuring CORS, its a constant source of confusion for our users. We try to get around it by providing helper scripts but Anne's proposal mirroring flashes cross domain.xml sounds vastly superior to the current implementation from the developers perspective. On 19 February 2015 at 12:05, Brian Smith br...@briansmith.org wrote: Dale Harvey d...@arandomurl.com wrote: The REST api pretty much by design means a unique url per request CouchDB has http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API, which allows you to fetch or edit and create multiple documents at once, with one HTTP request. CouchDB's documentation says you're supposed to POST a JSON document for editing, but the example doesn't set the Content-Type on the request so presumably it is OK to set the Content-Type to text/plain. This means that you'd have ONE request and ZERO preflights to edit N documents. in this case a lot of the requests look like GET origin/_change?since=0 GET origin/_change?since=the last id A GET like this won't require preflight unless you set custom header fields on the request. Are you setting custom headers? If so, which ones and why? I looked at the CouchDB documentation and it doesn't mention any custom header fields. Thus, it seems to me like none of the GET requests should require preflight. Also, if your server is SPDY or HTTP/2, you should be able to configure it so that when the server receives a request GET /whatever/123, it replies with the response for that request AND pushes the response for the not-even-yet-sent OPTIONS /whatever/123 request. In that case, even if you don't use the preflight-less bulk document API and insist on using PUT, there's zero added latency from the preflight. Cheers, Brian
Re: CORS performance
On Feb 19, 2015 3:33 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Feb 19, 2015 at 12:17 PM, Dale Harvey d...@arandomurl.com wrote: With Couch / PouchDB we are working with an existing REST API wherein every request is to a different url (which is unlikely to change), the performance impact is significant since most of the time is used up by latency, the CORS preflight request essentially double the time it takes to do anything Yeah, also, it should not be up to us how people design their HTTP APIs. Limiting HTTP in that way because it is hard to make CORS scale seems bad. +1. Forcing developers to change their APIs would be bad form at this stage. Not to mention just plain silly. Optimizing with an OPTIONS * preflight is a good option but won't be as broadly available to developers as a response header. Perhaps another approach would be to allow a resource to declare a CORS policy only for subordinate resources, rather than the entire origin. For instance, an OPTIONS sent to http://example.org/api/ can return CORS headers that cover every URL prefixed with http://example.org/api/. That would logically extend all the way up to OPTIONS * in order to set a policy that covers the entire origin. - James I think we've been too conservative when introducing CORS. It's effectively protecting content behind a firewall, but we added all these additional opt in mechanism beyond protecting content behind a firewall due to unease about the potential risks. Figuring out to what extent that actually serves a purpose would be good. If declaring this policy through a header is not acceptable, we could attempt a double preflight fetch for the very first CORS fetch against an origin (that requires a preflight). Try OPTIONS * before OPTIONS /actual-request. If that handshake succeeds (details TBD) no more preflights necessary for the entire origin. -- https://annevankesteren.nl/
Re: CORS performance
If the cache is against the url, and we are sending requests to different urls, wont requests to different urls always trigger a preflight? I just realised my mistake, GETS without custom headers should need to trigger preflight requests, sorry On 19 February 2015 at 13:31, Dale Harvey d...@arandomurl.com wrote: Will take a look at the content-type on GET requests, thanks I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). http://www.w3.org/TR/cors/#preflight-result-cache-0 If the cache is against the url, and we are sending requests to different urls, wont requests to different urls always trigger a preflight? Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. CouchDB has no bulk document fetch api, it has all_docs but that isnt appropriate for this case, there is a talk about introducing it https://issues.apache.org/jira/browse/COUCHDB-2310, however its going to take a while (I would personally rather we replace it with a streaming api) I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Whats wrong with 'This origin is part of the public internet and doesnt need any complications or restrictions due to CORS' ie Anne proposal? On 19 February 2015 at 13:21, Brian Smith br...@briansmith.org wrote: On Thu, Feb 19, 2015 at 4:49 AM, Dale Harvey d...@arandomurl.com wrote: so presumably it is OK to set the Content-Type to text/plain Thats not ok, but may explain my confusion, is Content-Type considered a Custom Header that will always trigger a preflight? To be clear, my comment was about POST requests to the bulk document API, not about other requests. I ran your demo and observed the network traffic using Wireshark. Indeed, OPTIONS requests are being sent for every GET. But, that is because you are setting the Content-Type header field on your GET requests. Since GET requests don't have a request body, you shouldn't set the Content-Type header field on them. And, if you do, then browsers will treat it as a custom header field. That is what forces the preflight for those requests. Compare the network traffic for these two scripts: script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4 , true); xhr.setRequestHeader(Accept,application/json); xhr.setRequestHeader(Content-Type,application/json); xhr.send(); /script script xhr=new XMLHttpRequest(); xhr.open(GET, http://skimdb.iriscouch.com/registry/_changes?timeout=25000style=all_docssince=209limit=100_nonce=xhGtdb3XqOaYCWh4 , true); xhr.setRequestHeader(Accept,application/json); xhr.send(); /script They are the same, except the second one doesn't set the Content-Type header, and thus it doesn't cause the preflight to be sent. if so then none of the caching will apply, CouchDB requires sending the appropriate content-type CouchDB may require sending Accept: application/json, but that isn't considered a custom header field, so it doesn't trigger preflight. The /_changes requests are only part of the problem, once we receive the changes information we then have to request information about individual documents which all have a unique id GET /registry/mypackagename We do one of those per document (70,000 npm docs), all trigger a preflight (whether or not custom headers are involved) I believe none of these require preflight unless a mistake is being made (probably setting Content-Type on GET requests). Also, regardless, you can use the CouchDB bulk document API to fetch all these documents in one request, instead of 70,000 requests. Also performance details aside every week somebody has a library or proxy that sends some custom header or they just missed a step when configuring CORS, its a constant source of confusion for our users. We try to get around it by providing helper scripts but Anne's proposal mirroring flashes cross domain.xml sounds vastly superior to the current implementation from the developers perspective. I agree that things can be improved here. I think the solution may be better developer tools. In particular, devtools should tell you exactly why a request triggered preflight. Cheers, Brian
Re: CORS performance
On 19 Feb 2015, at 22:04, Martin Thomson martin.thom...@gmail.com wrote: On 18 February 2015 at 06:31, Brad Hill hillb...@gmail.com wrote: Some of the things that argue against /.well-known are: 1) Added latency of fetching the resource. It's not available everywhere yet, but you could push it, based on the below. 2) Clients hammering servers for non-existent /.well-known resources (the favicon issue) You could avoid that by Link:-ing to the /.well-known and only hitting it if the link appears. I assume you mean the Link: header. In that case I like the idea. Well the client could even cache the document and only hit it once for the whole server. Furthermore there would then be no need for the url to be in a .well-known location. It could be any resource whatsoever. That is the way that the Web Access Control system functions. See link from this page http://www.w3.org/2005/Incubator/webid/spec/ Every resource has a link to those resources that require access. Aslo see the curl examples https://github.com/read-write-web/rww-play/wiki/Curl-Interactions Henry Social Web Architect http://bblfish.net/
Re: CORS performance
* Jonas Sicking wrote: We most likely can consider the content-type header as *not* custom. I was one of the people way back when that pointed out that there's a theoretical chance that allowing arbitrary content-type headers could cause security issues. But it seems highly theoretical. I suspect that the mozilla security team would be fine with allowing arbitrary content-types to be POSTed though. Worth asking. I can't speak for other browser vendors of course. I think the situation might well be worse now than it was when we first started discussing what is now CORS. In any case, this would be an ex- periment that cannot easily be undone, browser vendors would not pay the bill if there are actually large scale security vulnerabilities opened up by such a change, and I do not really see notable benefits in con- ducting such an experiment. -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de Available for hire in Berlin (early 2015) · http://www.websitedev.de/
Re: CORS performance proposal
On Thu, Feb 19, 2015 at 5:29 AM, Anne van Kesteren ann...@annevk.nl wrote: When the user agent is about to make its first preflight to an origin (timeout up to the user agent), it first makes a preflight that looks like: OPTIONS * Access-Control-Request-Origin-Wide-Cache: [origin] Access-Control-Request-Method: * Access-Control-Request-Headers: * This would make CORS preflight even slower for every server that doesn't implement the new proposal (i.e. every currently-deployed server and probably most servers deployed in the future). Perhaps the OPTIONS * request could be made in parallel with the initial preflight requests. But, then, what happens when the information in the OPTIONS * response conflicts with the information in the normal preflight request? I think this has a reasonable tradeoff between security and opening up all the power of the HTTP APIs on the server without the performance hit. It still makes the developer very conscious about the various features involved. I think developer consciousness is exactly the issue here: 1. Let's say you want to add OPTIONS * preflight support to an existing web application. How do you go about finding all the things that need to change to make that safe to do? It seems very difficult to successfully find every place the app assumes it is protected by the fact that it doesn't do CORS. 2. Similar to #1, let's say that two teams develop two parts of a website. One of the teams follows normal CORS rules and the other depends on the proposed OPTIONS * mechanism. This would be a disaster if/when both apps are deployed on the same origin. 3. Because of these issues, an organization forces its developers to develop every app as though every resource is CORS-enabled, to future-proof against the scenerio where OPTIONS * is deployed in the future. This makes the development of the web app more difficult and slower. 4. In the discussion of Entry Point Regulation (EPR) on WebAppSec, the main argument in favor of it is that it is impossible for developers to do things like #3 correctly and it is unreasonable for us to expect them to. I'm don't buy the EPR argument completely, but I do see some merit in the underlying secure by default argument behind EPR. Because of these concerns, I think that it would be worthwhile to study a concrete example of the problem, to make sure we correctly understand the use case we're trying to solve. As we saw yesterday with the PouchDB/CouchDB example, it is easy to accidentally and unnecessarily force a preflight. It may also be the case that we can find other, safer, ways to avoid preflights and/or optimize how they are done, such as by optimizing CORS for use with HTTP/2 server push mechanisms. But, we need to see real instances of the problem first. Cheers, Brian
Re: CORS performance proposal
* Martin Thomson wrote: On 20 February 2015 at 00:29, Anne van Kesteren ann...@annevk.nl wrote: Access-Control-Allow-Origin-Wide-Cache: [origin] This has some pretty implications for server deployments that host mutual distrustful applications. Now, these servers are already pretty well hosed from other directions, but I don't believe that there is any pre-existing case where a header field set in a request to /x could affect future requests to /y. An alternative would be to use /.well-known for site wide policies. The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`. -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de Available for hire in Berlin (early 2015) · http://www.websitedev.de/
Re: CORS performance
On Thu, Feb 19, 2015 at 12:38 PM, Brad Hill hillb...@gmail.com wrote: I think that POSTing JSON would probably expose to CSRF a lot of things that work over HTTP but don't expect to be interacted with by web browsers in that manner. That's why the recent JSON encoding for forms mandates that it be same-origin only. Note that you can already POST JSON cross-origin. Without any preflight. The only thing you can't do is to set the Content-Type header to the official JSON mimetype. So the question is, does the server check that the Content-Type header is set to application/json and if not abort any processing? / Jonas
Re: CORS performance proposal
* Martin Thomson wrote: On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote: The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`. I missed that. In which case I'd point out that `OPTIONS *` is very poorly supported. Some people (myself included) want it to die a flaming death. Evidence for poorly supported would certainly be helpful (web hosting packages without TLS support, for instance, do not count, though). -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de Available for hire in Berlin (early 2015) · http://www.websitedev.de/
Re: CORS performance proposal
On 20 February 2015 at 11:39, Bjoern Hoehrmann derhoe...@gmx.net wrote: The proposal is to use `OPTIONS * HTTP/1.1` not `OPTIONS /x HTTP/1.1`. I missed that. In which case I'd point out that `OPTIONS *` is very poorly supported. Some people (myself included) want it to die a flaming death.
Re: CORS performance
On Tue, Feb 17, 2015 at 2:43 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: * Anne van Kesteren wrote: On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: Individual resources should not be able to declare policy for the whole server, ... With HSTS we gave up on that. FWIW, this dynamic is why you can't set HSTS on an S3 bucket (or a CloudFront distribution backed by an S3 bucket). Amazon isn't willing to let you set a HSTS header for a file that might also be served at s3.amazonaws.com. And so any website backed by S3, even if you never use the s3.amazonaws.com URLs, is restricted from setting HSTS headers. -- Eric -- konklone.com | @konklone https://twitter.com/konklone
Re: CORS performance
+1 to Anne's suggestion. The current design is pretty terrible for API performance. I think a request to / with OPTIONS or something, with a response that requires some server side logic (like return the random number UA just sent) is pretty darn secure. cheers dev On 17 February 2015 at 11:24, Anne van Kesteren ann...@annevk.nl wrote: On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: Individual resources should not be able to declare policy for the whole server, ... With HSTS we gave up on that. HTTP/1.1 rather has `OPTIONS *` for that, which would require a new kind of preflight request. And if the whole server is fine with cross-origin requests, I am not sure there is much of a point trying to lock it down by restricting request headers or methods. Yeah, I wasn't sure whether those should all be listed. Maybe simply declaring you're fluent in CORS in a unique way is sufficient. -- https://annevankesteren.nl/
Re: CORS performance
On both this, and CSP pinning, I find myself getting nervous about adding an increasing number of headers which, when sent by any resource, impact the security posture and functioning of an entire origin. HSTS and HPKP are somewhat special in that: they convey only a few bits of information. are somewhat self-verifying (e.g. an https pin cannot be applied over http, and a public key pin cannot be applied from a certificate without a matching key) and unlikely to change. For more complex, application-layer issues, are pinning headers really the best and most manageable approach? I think it is at least worth discussing the relative merits of using a resource published under /.well-known for such use cases, vs. sending pinned headers with every single resource. Some of the things that argue against /.well-known are: 1) Added latency of fetching the resource. For CORS pinning, this probably isn't an issue. A client that has never seen a pinned policy (by whatever mechanism it is delivered) will always have to make a preflight, and servers will always have to respond appropriately (rather than relying on the pinned policy) for legacy clients. So a client that knows about pinning can make a request to /.well-known in parallel the first time it makes a CORS request to a given host, and speed up all subsequent requests without any additional latency on the first. 2) Clients hammering servers for non-existent /.well-known resources (the favicon issue) Again, probably not an issue for CORS, or not nearly as large an issue. Clients can be configured to never ask for the CORS policy at an origin unless/until they make a CORS request to that origin. There is a question of how long to treat a 404 as authoritative before re-polling, but it can be fairly long without breaking anything. We might also devise a hint header that suggests a pinned policy is available, which would only be sent along with preflight responses. On disadvantages of headers: 1) Individual resources declaring policy for an entire origin can be problematic and difficult to manage. Doing this right, both to set a correct policy and police lower-privileged resources from setting incorrect ones, in practice would require an administrator to set up a filtering proxy in front of the entire origin, which is more difficult and costly than simply locking down access to a specific resource path to only administrators. 2) Lots of redundant chit-chat. HTTP/2 header compression reduces the impact of this somewhat, and CORS specifically could avoid sending pinning headers except on preflight responses, but it does seem like a real concern. How much long-term bloat are we willing to accept on the network in order to save a few ms of latency the first time you connect to a site? -Brad On Tue Feb 17 2015 at 10:42:20 AM Anne van Kesteren ann...@annevk.nl wrote: Concerns raised by Monsur https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0260.html and others before him are still valid. When you have an HTTP API on another origin you effectively get a huge performance penalty. Even with caching of preflights, as each fetch is likely to go to a distinct URL. With the recent introduction of CSP pinning, I was wondering whether something like CORS pinning would be feasible. A way for a server to declare that it speaks CORS across an entire origin. The CORS preflight in effect is a rather complicated way for the server to announce that it can handle CORS. We made it rather tricky to avoid footgun scenarios, but I'm wondering whether that is still the right tradeoff. Something like: CORS: max-age=31415926; allow-origin=*; allow-credentials=true; allow-headers=*; allow-methods=*; expose-headers=* -- https://annevankesteren.nl/
Re: CORS performance
* Anne van Kesteren wrote: With the recent introduction of CSP pinning, I was wondering whether something like CORS pinning would be feasible. A way for a server to declare that it speaks CORS across an entire origin. The CORS preflight in effect is a rather complicated way for the server to announce that it can handle CORS. We made it rather tricky to avoid footgun scenarios, but I'm wondering whether that is still the right tradeoff. Something like: CORS: max-age=31415926; allow-origin=*; allow-credentials=true; allow-headers=*; allow-methods=*; expose-headers=* Individual resources should not be able to declare policy for the whole server, HTTP/1.1 rather has `OPTIONS *` for that, which would require a new kind of pre-flight request. And if the whole server is fine with cross-origin requests, I am not sure there is much of a point trying to lock it down by restricting request headers or methods. I suppose some- thing like this could be implemented, but I don't think CORS pinning is quite the right analogy. -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de Available for hire in Berlin (early 2015) · http://www.websitedev.de/
Re: CORS performance
On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: Individual resources should not be able to declare policy for the whole server, ... With HSTS we gave up on that. HTTP/1.1 rather has `OPTIONS *` for that, which would require a new kind of preflight request. And if the whole server is fine with cross-origin requests, I am not sure there is much of a point trying to lock it down by restricting request headers or methods. Yeah, I wasn't sure whether those should all be listed. Maybe simply declaring you're fluent in CORS in a unique way is sufficient. -- https://annevankesteren.nl/
Re: CORS performance
* Anne van Kesteren wrote: On Tue, Feb 17, 2015 at 8:18 PM, Bjoern Hoehrmann derhoe...@gmx.net wrote: Individual resources should not be able to declare policy for the whole server, ... With HSTS we gave up on that. Well, HSTS essentially removes communication options, while the intent of CORS is to add communication options. I don't think you can compare them like that. HSTS is more like a redirect and misconfiguration may result in denial of service, while CORS misconfiguration can have more far-reaching consequences like exposing user information. -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de Available for hire in Berlin (early 2015) · http://www.websitedev.de/
