Re: [XMLHttpRequest] Last Call
We still do not have anyway to advice user agents of long-lived responses in order to avoid the problem of indefinitely queued pipelined requests/responses. With both pipelining and long-lived responses becoming more common, this seems to be an envitable problem. It doesn't seem acceptable to simply ignore this issue. I am not asking for anything increasing the XHR implementation burden, simply a standard way of advising so that authors and user agents can communicate, so user agents have an approach for implementing proper delegation of pipelined requests when users start complaining of problems. Thank you, Kris - Original Message - From: Anne van Kesteren [EMAIL PROTECTED] To: Web API WG (public) public-webapi@w3.org Sent: Tuesday, April 15, 2008 12:40 PM Subject: [XMLHttpRequest] Last Call Hi, The Web API WG resolved yesterday to publish a Last Call Working Draft of The XMLHttpRequest Object specification. Thanks to the webmasters at the W3C it was published earlier today. Last Call comments can be made until 2 June 2008. (Please don't comment about the befor typo there. My mistake.) The draft is located here: http://www.w3.org/TR/2008/WD-XMLHttpRequest-20080415/ This is a rough list of the changes since the last published draft: * Removed dependency on DOM Level 3 Events * Removed dependency on Window Object 1.0 (we already depended on HTML 5 so we could use that instead) * Clearly marked which HTTP methods are to raise SECURITY_ERR * Mention HttpOnly in the security section * Accept-Language is set conditionally * Requests using a GET method will not have an entity body * responseText can no longer return null * responseText and responseXML no longer throw * Headers starting with Sec- will raise * getResponseHeader() will return null for invalid headers * getAllResponseHeaderS() returns empty string in case of failure A more detailed log of these changes, including links to the actual text changes, is available from dev.w3.org, here: http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.html Happy times reviewing! Kind regards, -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
Re: [XMLHttpRequest] Last Call
OK, that's fine, I just didn't want it to be ignored indefinitely. Thanks, Kris - Original Message - From: Maciej Stachowiak [EMAIL PROTECTED] To: Kris Zyp [EMAIL PROTECTED] Cc: Web API WG (public) public-webapi@w3.org; Anne van Kesteren [EMAIL PROTECTED] Sent: Wednesday, April 16, 2008 9:35 PM Subject: Re: [XMLHttpRequest] Last Call On Apr 16, 2008, at 7:50 PM, Kris Zyp wrote: We still do not have anyway to advice user agents of long-lived responses in order to avoid the problem of indefinitely queued pipelined requests/responses. With both pipelining and long-lived responses becoming more common, this seems to be an envitable problem. It doesn't seem acceptable to simply ignore this issue. I am not asking for anything increasing the XHR implementation burden, simply a standard way of advising so that authors and user agents can communicate, so user agents have an approach for implementing proper delegation of pipelined requests when users start complaining of problems. I think this would be more appropriate to deal with in XHR2, since we don't have a settled design for this feature and XHR1 is already way behind schedule. - Maciej Thank you, Kris - Original Message - From: Anne van Kesteren [EMAIL PROTECTED] To: Web API WG (public) public-webapi@w3.org Sent: Tuesday, April 15, 2008 12:40 PM Subject: [XMLHttpRequest] Last Call Hi, The Web API WG resolved yesterday to publish a Last Call Working Draft of The XMLHttpRequest Object specification. Thanks to the webmasters at the W3C it was published earlier today. Last Call comments can be made until 2 June 2008. (Please don't comment about the befor typo there. My mistake.) The draft is located here: http://www.w3.org/TR/2008/WD-XMLHttpRequest-20080415/ This is a rough list of the changes since the last published draft: * Removed dependency on DOM Level 3 Events * Removed dependency on Window Object 1.0 (we already depended on HTML 5 so we could use that instead) * Clearly marked which HTTP methods are to raise SECURITY_ERR * Mention HttpOnly in the security section * Accept-Language is set conditionally * Requests using a GET method will not have an entity body * responseText can no longer return null * responseText and responseXML no longer throw * Headers starting with Sec- will raise * getResponseHeader() will return null for invalid headers * getAllResponseHeaderS() returns empty string in case of failure A more detailed log of these changes, including links to the actual text changes, is available from dev.w3.org, here: http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.html Happy times reviewing! Kind regards, -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
Re: XDR *API* Security Impact
I don't know whether most modern devs are using JSON, Yeah, I really don't have any data to back that up, I shouldn't make such claims :). But I think it is safe to say that many devs use it. More generically, I don't think that new cross-origin APIs that just return a string (but are clearly intended for structured data) should be standardized (or shipped) given the experience that's available. So you would be in favor of a responseJSON property? Kris
Re: What is Microsoft's intent with XDR vis-à-vis W3C? [Was: Re: IE Team's Proposal for Cross Site Requests]
From web developer's perspective, the idea of having a completely divergent API and technique for doing cross-site requests between browsers does not seem beneficial. As much as possible I would hope that MS and CS-XHR can converge. It seems like there are a number of differences that can be analyzed individually. Are there specific parts of the CS-XHR that IE finds objectionable, that is more specific than just broad sweeping statements about security? I appreciate Tyler's attempts at specifics, but they have been proven incorrect. Is it possible that IE could at least converge and get behind parts of CS-XHR? 1. Why do we need a new API (XDomainRequest) to do cross-site requests? Why can't I make cross-site requests with my existing code that uses XMLHttpRequest? This seems completely orthogonal to security, and as a developer I am certainly not eager to write a bunch of branch logic for a new API. The proposal mentioned not wanting to bolt on security which has zero value for web developers, maybe that is some internal IE team motivation. 2. Why does XDomainRequest need to use a different header names than CS-XHR? Once again, the actual names of the headers seems completely orthogonal to security. Why can't XDR use Referer-Root and Access-Control headers? Even if IE refused to implement method check request and granular host names, there is no reason XDR can't at least use the same syntax and semantics. Servers that want to support cross-browser cross-site request will need to handle two different sets of headers. 3. Why are POST and DELETE not allowed in XDR? I haven't seen any evidence to suggest these represent security breaches. They certainly not harmful with CS-XHR since they must servers must opt-in and authorize them. In my opinion this is devastating omission. REST is critical part of the future of web applications. 4. Why are developers restricted from setting the Accept-* and Content-Type headers in XDR? Once again the restriction and the inevitable content sniffing will likely harm security more than help it. 5. Why can't I send a request synchronously? This is essentially tied to the API. XDomainRequest does not provide async/sync choice. With FF3 (and any browsers that follow suite) new continuation-style handling of synchronous requests, this could possibly become a more important request mechanism than async in the future. There certainly isn't any security reason to deny such capability. 6. Why does XDR not use the method check technique from CS-XHR? This has been demonstrated to be more secure than XDR. XDR is prone to sending POST requests that otherwise not possible with traditional form posts (like a POST with JSON data in it, which is impossible to do with a form). 6b. Why not allow cookies? IMO, the idea that a server can make a better decision about security by having less information is bizarre. With requests that have side effects, ultimately it is the server that has to make the final decision about whether to accept the request or not. The whole idea that servers can make better decisions if they know less (don't know the cookie information) seems very strange to me. 7. Why does XDR prevent secure connections (HTTPS)? I am hoping this is just a bug, and somone didn't actually make such a design choice. Are there any specific issues in the CS-XHR that MS finds objectionable? I certainly could commiserate with objecting to XML processing instructions. I would love to see that be omitted. Or does MS object to the complexity of CS-XHR? Even a partial compromise would be better than the wild divergence that it seems we are headed towards. I can't understand how that can be seen as beneficial for web developers. That is the big problem with XDR’s restrictions. Well, aside from its breaking of REST by disallowing PUT and DELETE and setting the Content-Type and Accept-* headers, while favouring SOAP (which DOES have the ability to delete() and authenticate) and encouraging content sniffing. I hope you can see why I don’t share your enthusiasm for Microsoft’s proposal :). From a web developer perspective, I completely agree, these excessive restrictions mean that we have a lot of hacks, interoperability problems, and resulting insecurities ahead of us if we use XDR as it is. In order to acquire the desired functionality (for which it needs the user’s credentials), with XDR the resource host will most likely end up passing the authentication information along in the GET query string (bad), probably requiring the user to fill in his credentials on the origin site (bad), and sending the user’s password plain over the wire (bad). I agree, and want to add that being able to implement OAuth from the client side should be extremely important use case for a cross-site request mechanism. Thanks, Kris
Re: IE Team's Proposal for Cross Site Requests
How does this compare to the Cross-Site Extensions for XMLHttpRequest
standard
that is being developed by Web API and Web App Formats (and
as implemented in Firefox betas)?
The major ones, IMO:
1. XDomainRequest uses a different constructor/API to make requests
(XDomainRequest instead of XMLHttpRequest).
2. XDomainRequest does not allow any headers. The W3C/AC for XSite requests
allows headers like Accept and Accept-Language which are crucial for the
fundamental HTTP capability of content and language negotiation.
3. Completely different server side response required to validate cross-site
requests are acceptable. XDomainRequest looks for a response header of
XDomainRequestAllowed: 1, while W3C/AC, looks for an Access-Control header
which has more fine-grained control.
4. Cookies and credentials are not sent with XDomainRequest. I believe this
is still being discussed with W3C/AC.
5. Standard HTTP REST methods (PUT and DELETE) are not supported with
XDomainRequest, but are supported with W3C/AC.
From Apple's point of view we would like to have a
unified standard in this area.
From web developer/framework developer's point of view, we would also
like a unified standard. Creating a new divergent API for cross-site
requests with XDR is certainly not the most beneficial approach for web
developers. Cross-site requests that work in all browsers (those that
implement W3C/AC and XDR) would look like:
if (window.XDomainRequest) {
var xhr = new XDomainRequest(); // IE's cross-domain request technique
}
else {
var xhr = new XMLHttpRequest(); // W3C and standards compliant browsers
cross-domain request technique
}
xhr.open(POST,http://otherdomain.com,true);
xhr.send();
And on the server, the server needs to at least set headers
XDomainRequestAllowed: 1 and Access-Control: allow * for the responses
to be accessible in all browsers.
Does this seem like a clean API for devs? I think it would be much cleaner
and easier for developers if Microsoft adopted the W3C cross-site request
mechanism and worked to improve it, rather than creating a different API.
snipping from Chris's email:
The short version is, as Sunava discusses in the summary of this mail,
that x-domain XHR (and Flash’s approach, et al) is subject to specific
x-domain injection attacks because of its persistent-allow design.
I understand that MS has security concerns, but rewriting the API is not
necessary to address these concerns. If there are aspects of the W3C/AC
proposal that MS doesn't like, I am sure discussion and feedback would be
welcome. Even minor deviations from W3C/AC proposal to satisfy the Microsoft
security concerns (such as stripping cookies, and requiring the
Access-Control header on every response) would be vastly better than using a
completely different API. I think the web development community would
appreciate standards convergence much more than multiple APIs. Having
multiple APIs increases complexity which increases the vector of possible
attacks.
Specifically, any request sent by XDR could also be emitted by a
properly coded HTML FORM object. Hence, any “non-participating”
web server put at risk by XDR is also at risk from simple HTML.
This is not true. I could send a POST request with XDR with a body of any
string. This is not possible in a form. A form strictly encodes everything
in URL encoded format. A server that expects that a body with pure JSON
would only come from a safe client (since this is not possible with a form),
would be in for a surprise with XDR. W3C/AC protects against this
vulnerability. IMO, this is a vulnerability and XDR may actually be
less secure than W3C/AC (and certainly less funtional).
Thanks,
Kris
Re: IE Team's Proposal for Cross Site Requests
If XDR only supports GET and POST, it encourages sites to use POST to implement delete functionality and abuse the HTTP protocol because that is the only way they can get the functionality they desire to work. Basically, you’re boycotting REST in favour of SOAP. I completely agree, it shouldn't be the place of the browser to dictate that developers they must use SOAP style instead of REST, and cripple the features of HTTP. The full features of REST (PUT and DELETE) weren't available in traditional web apps, but with Ajax, I belive REST techniques are rapidly growing in popularity, and will be particularly useful in cross-site scenarios because authors can leverage HTTP semantics to indicate meaning with different consumers/providers. I believe the same potential is possible with Accept headers, especially in the cross-domain world. Accept headers are not heavily used right now because in the dominantly same-origin request world because developers usually know a priori what content type is needed. On the otherhand, with cross-site web services, different consumers may desire different formats, and the Accept parameters could have awesome potential for allowing developers to request data in a desired format in well-understood manner. How foolish it would be to disable these features in the arena where it would have the most opportunity for benefit. I believe the full potential of HTTP has a great opportunity to be fully realized in the cross-site world where leveraging well-defined broadly understood HTTP features like REST semantics, content-negotiation, partial content transfer, etc. could vastly improve interoperability, by allowing consumers and providers to communicate with a rich set of well-defined capabilities. Crippling HTTP will force developers to use work-arounds which will only decrease interoperability and will ultimately cause increased use of hacks which will envitably subvert security rather than benefiting it. Trying to create a simpler more restrictive form of security does not mean things will be more secure. Developers use of cross-site scripts with callbacks to get cross-site data is great example of how an excessively simple security model has resulted in developers being forced to use dangerous methods to accomplish their goals. Thanks, Kris
Re: XHR setRequestHeader(connection, close) is bogusly rejected
for UAs is best approach, and that it should be a number-based advice, not a boolean, so that it could be used more effectively and flexibly in heuristic algorithms for making informed pipelining decisions. Can you be more specific in what you mean about number-based advice? (Apologies if you explained this in an earlier message, I tried skimming them and did not find a description). I couldn't find my previous message in the archives, so I included below again (sorry about posting it again). Basically I was suggesting that a property (called durability) could take a number (rather than a boolean), that would represent advice about how long the author expected the response to take. The user agent could utilize this advice in any way it sees fit. A number provides more information (a greater range of values) than a boolean, and it could effectively be used as weighting in user agent heuristics. The advice could be used by user agents to determine if the connection should be counted against the connection limit (if it is larger than a threshold), and it could used as advice for pipelining. The user agent is not required to respond in any certain way, but this is author provided advice, that the user agent may use in any way it desires. The advantage of a number may be more clearly seen in a pipeline ordering example. Suppose I issue 4 requests, and I expect request A to take a few milliseconds, request B to take a half a second, request C to take a couple seconds, and request D could last indefinitely. I could set different durability values for each XHR request, and then the user agent could potentially use these values (possibly in combination with other information, like connection speed, proxy information, and so forth) to determine the order of pipelining: xhrA.durability = 0.01; xhrB.durability = 0.5; xhrC.durability = 2; xhrD.durability = 100; In this situation, a browser may decide to send A and B on different connections, and send C as pipelined by request A. It may also choose to send D on completely separate connection and not count that against the connection limit. However, the advice does not require the user agent to take any specific action, the browser can weigh the advice in combination with other factors, depending on the situation. Anyway, here is the more detailed proposal I had sent out, I apologize if you have already received it/read it (it had a subject Re: Pipelining Control Proposal): I wanted to propose another way to deal with the usability issues involved with pipelining in combination with long-lived responses for the purposes of server initiated messages. This is not an opposition to HTTP improvements for declaring extra connection limits. I hope that happens for the broader benefit that it provides and that HTTP improvements can solve the two- connection limit probem, however as I articulate in the reasons at the bottom, HTTP improvements are simply not sufficient for avoiding pipeling issues, and a simple mechanism for providing connection advise is critical as well. I believe this is a very minimalistic approach only adds a trivial amount of complexity to the XMLHttpRequest spec and has almost zero required implementation burden. The addition property defined here can be used by user agents to deal with the problem of responses becoming indefinitely queued due to being pipelined on a connection where a previous response is never finished or simply takes a long time. I propose that we have define a property called durability to the XMLHttpRequest object, that can be assigned a positive number (x = 0) by authors that want to advise the user agent on how long they expect the given request to last. A low value indicates that the request is expected to be very quick and a high value indicates that the request is expected to take longer. The durability property should default to 1, and so a value of 1 indicated a request with an average length of time. The durability property can set and modified at any time before, during, or after a request. A user agent is not required to behave in any particular way in response to the durability property, it is simply a standard way for authors to provide information about how long the response will probably take so that user agents can utilize this information in making informed optimization decisions, and provide a means for intelligently avoiding indefinitely queued requests/responses. There are no requirements for how the user agent responds to this property, but it is recommended that user agents use this information to help optimize pipelining (and potentially connection limiting). The lower the durability value, the faster the request is likely to receive a response, and therefore the corresponding connection is a better candidate for pipelining if that is enabled in the user agent. The higher the durability value, the slower the request is likely to receive a response, and therefore a
Re: XHR setRequestHeader(connection, close) is bogusly rejected
As a browser implementor, I don't think I would find this more useful than a boolean property indicating a connection is likely to be long- lived. I can't imagine doing anything more complicated than picking a threshold value to decide whether to count a request towards the connection limit and whether to consider pipelining requests on its connection. If all implementors did this, we'd likely end up with somewhat different threshold values, which would be unfortunate. We could certainly agree upon and define the threshold values, that is trivial. However, would you want the threshold for connection limit counting and pipelining to be the same (as a boolean would effectively be)? I would think that are some responses that may take a long enough to avoid pipelining, but still count in against the connection limit. More importantly, pipelining is not simply a yes or no question, it is a question of which connection to pipeline on. A faster response is attained by pipelining on the connection that will be able handle the request soonest, and therefore pipelining becomes a comparison calculation. If browsers have numeric values for response time estimates, they can make appropriate comparisons and do optimal pipelining. Perhaps the next iteration of browsers would only be based on thresholds, but numeric values allow great transfer of information, which could be used for more creative optimizations like pipelining comparisons and perhaps optimizations that we haven't even thought of yet. Just because we haven't foreseen all the possible optimizations, doesn't mean we shouldn't leave the door open for such optimizations. By allowing numeric values, authors can begin providing advice, and browsers can use it at their convenience. Also, as I had mentioned in the proposal, there are situations where authors actually want to pipeline requests behind long-lived responses, so they are intentially not responded to. Once again, boolean values don't provide an adequate mechanism for communicated this desire. Also, I'm skeptical that authors can estimate request latency with anything near that kind of precision. Thus, it is likely to be an unreliable source of information, meaning that the details of the value couldn't be trusted anyway, and even a high-low threshold might be unreliable. But most authors know if their XHR connection is intended to be used persistently, so the one bit of info would be more trustworthy. Sure, it is impossible to create perfectly accurate estimates, and it is in fact an estimate and should be treated that way, but it seems very presumptious to assume that authors can't provide meaningful data. If authors don't know, that can simply not set this property or use a very high value to indicate a long-lived response. If they provide errant values, that is their problem, but I don't think we should understimate the potential of what authors can understand and do. Thanks, Kris
Re: XHR setRequestHeader(connection, close) is bogusly rejected
However, there are web apps in existence (e.g., Gmail) that set the connection: close header to inform the user-agent that the HTTP transaction is going to take a long time. (This is also informative for the server.) This allows a user-agent to not count this connection against the RFC 2616 recommended maximum of 2 persistent connections per host. As far as I can tell, Firefox is the only browser that regards connection: close as an indication that the connection should not be counted against the connection limit, and this behavior does _not_ follow the HTTP recommendation regarding connections. Section 8.1.4 states that: Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. (All browsers use persistent connections, so all browsers are subject to the recommendation stated below) A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. There is no mention of only maintaining two persistent connections, all connections (persistent or otherwise) are supposed to be counted against the connection limit. In addition, there is no reason why long-lived responses (those that shouldn't be counted against the connection limit) should have to have their TCP connection closed after the response is fully received. Therefore, I don't see Connection: close as an appropriate means for advising that connections should not be counted against connection limits. Kris
Re: XHR setRequestHeader(connection, close) is bogusly rejected
If the problem we are trying to solve is preventing potentially long- lived requests from blowing out the connection limit I think it would be better to either: 1) Add an explicit XHR property that indicates this request may be long-lived - this would not only bypass the connection limit but would also indicate to the UA that it should not pipeline other requests on the same connection, if it supports pipelining. 2) Never count XHR-initiated http requests towards the per-server connection limit. not seem necessary for the goal of bypassing the connection limit on the UA side. And it seems that an explicit XHR property for this would be more clear. I agree, I think #1 is the way to go. I don't like #2, because connection limits really are valuable for minimizing server load, and even can help prevent DOS attacks. As I just mentioned in the other email, Connection: close is not an appropriate form of advice, IMO. I think that an explicit property that provides advice for UAs is best approach, and that it should be a number-based advice, not a boolean, so that it could be used more effectively and flexibly in heuristic algorithms for making informed pipelining decisions. Kris
Re: XHR setRequestHeader(connection, close) is bogusly rejected
As far as I can tell, Firefox is the only browser that regards connection: close as an indication that the connection should not be counted against the connection limit What gives you that idea? I'm not actually sure since I don't know the inner workings of the network code, but I have never heard of such behavior. I have never heard of such behavior either until this thread. I just tried it out against Safari 3, IE8, Opera 9, and FF2. Only FF2 seemed to be affected by Connection: close. FF2 seems to let me open an indefinite number of connections to a server if use Connection: close, but only two connections are allowed if don't use it. I was surprised as well. FF3 is not affected by Connection: close, only FF2. Kris
Re: Extra Connection Support Proposal
Thanks for the heads up, Mark, I will be watching the activity there, that would be great if the problem can be dealt with in that group. However, the issues of responses being indefinitely queued in pipelining situations and unbounded memory growth on continuous streaming responses are still present, and I don't think there is anything the HTTP WG can do about that. Thanks, Kris - Original Message - From: Mark Baker [EMAIL PROTECTED] To: Kris Zyp [EMAIL PROTECTED] Cc: public-webapi@w3.org Sent: Thursday, March 06, 2008 6:16 AM Subject: Re: Extra Connection Support Proposal FYI, you might be interested in this recent discussion in the HTTP WG, which could make this proposal unnecessary; http://lists.w3.org/Archives/Public/ietf-http-wg/2008JanMar/0423.html Mark.
Re: Extra Connection Support Proposal
Doing this on an HTTP level seems like the right solution to me. Though i'm not sure what working group would then be appropriate for standardizing it... I don't mind trying this avenue, I just fear that this is even more likely to be a dead-end. HTTP is already very complicated (it would seem more so than XHR to me) and is much more solidified and it seems unlikely that there would a be solution down that path. And I don't see how it is any more right for this to be done in HTTP, The user agent needs to be informed by something, and I don't see a clear distinction in why it is superior to be informed by the HTTP response rather than API/author since this doesn't really affect the actual HTTP communication protocol, but when a user agent chooses to use an (additional) HTTP communication channel. I am concerned that if all parties continually come up with really good reasons to keep deferring this to someone else, the user with hanging requests still lose. I am going to send out another proposal that will hopefully be more palatable/feasible. Thanks, Kris
Re: Extra Connection Support Proposal
We are still faced with the fundamental problem that if a browser that observes the two connection limit and two long-lived connections are currently open and the user does something that triggers another request (such as opening another tab), the browser is stuck and essentially hangs waiting for a connection to become available. This is a serious usability issue. Is there something that I need to do to improve my proposal, so can effectively tackle this issue, and provide a means for authors to inform the user agents when a response is long-lived? Thanks, Kris
Re: Extra Connection Support Proposal
The problem has always existed, though. The problem has always existed, but having long-lived responses for the purpose of the receiving messages from the server massively exacerbates the problem. Most responses are intended to be as short-lived as possible. Also, in the scenario you mentioned, if you click a link on a page with downloads happening, the browser can abort responses for images that are being downloaded, which immediately frees up connections. On the otherhand, if a couple connections are indefinitely open, and you open another tab that needs a connection, the browsers hangs... indefinitely. Kris
Pipelining Control Proposal
Pipelining Control HTTP Pipelining is when more than one outstanding request is sent over a single TCP connection, and it was introduced in HTTP 1.1. This proposal defines that XHR objects should be able to control whether or not they are pipelined. A pipeline property would be added to the XHR object. If pipeline property is set to true, when send is called, the XHR request SHOULD be pipelined over one of the currently active connection, even if all connections to the target server are currently waiting for a response. That is, the request should be pipelined if necessary to send it immediately. If there is an available connection is alive, but no responses are waiting, the request should be sent on this connection (just as a non-pipelined request would be). If the pipeline property is set to false, the XHR request SHOULD NOT be pipelined even if the user agent supports and would otherwise pipeline the request. The pipeline property may also be set to another XHR object with an open connection, in which case the request should be pipelined on that specific TCP connection. For example: var xhr1 = new XMLHttpRequest(); xhr1.open(GET,/resource1,true); xhr1.send(null); var xhr2 = new XMLHttpRequest(); xhr2.open(GET,/resource2,true); xhr2.pipeline = xhr1; xhr2.send(null); In this example, both requests should be sent over the same TCP connection. The GET for resource1 should be sent and the GET for resource2 should be pipelined behind the first request on the same connection. If a connection has been marked as an extra connection with the extraConnection property on the XHR object (see the extra connection proposal), that connection should not be used for pipelined requests unless another XHR request explicitly specifies that connection, in which case that XHR request SHOULD be pipelined on that connection. (This is because the extra connection request is generally to be used for long-lived responses that are kept open for server-sent messages. Requests that are sent behind such a request may never receive a response, which should not be the default behavior, but may be explicitly chosen to achieve full asynchronous duplex communication on a single TCP connection, a highly valuable capability channels with server-sent messages.) If a network error occurs while servicing a request, any pipelined requests that are queued behind the first request SHOULD NOT automatically be retried by the user agent. A network error in response to the first request should cause an error condition for both the first XHR object and all subsequent pipelined XHR objects per normal XHR behavior. It is the responsible of the application author to retry the request if desired. Note that RFC 2616 states that non-idempotent methods should not be pipelined. However, this SHOULD NOT be enforced by the user agent. Authors should follow the recommendation of RFC 2616, but if they do not and choose to pipeline a non-idempotent request, the user agent should oblige.
Extra Connection Support Proposal
Extra Connection Support The XMLHttpRequest should define a property called extraConnection. When extraConnection is set to true, it indicates that this XHR object's connection SHOULD NOT be counted against the user agent's connection limit. That is if the user agent adheres to the two-connection limit as recommended by RFC 2616 section 8.1.4, it SHOULD allow two connections not counting this XHR object's connection. If the user agent has other or additional connection limits, this connection should not be counted in the accounting for these connection limits. Only one XHR object should be allowed to gain extra connection status per document per server. Each document in the browser may be permitted one extra connection per server. Within a document if an XHR object has an extra connection status for domain1.com, no other XHR objects may have extra connection status for that server until the first XHR object has terminated it's connection. However, another XHR object may have extra connection status to domain2.com. Each document in the browser should have it's own set of extra connections for XHR objects. This limitation is intended to prevent a vector of denial of service attacks. To gain extra connection status, the extraConnection property on the XHR object can be set to true. When it is set to true, if the XHR object is allowed to gain extra connection status (no other XHR objects currently have extra connection status in this document for the specified target server), the XHR object will be given extra connection status and subsequent access to the property will return true. If the XHR object is not allowed to gain extra connection status, subsequent access to the property should return false. The following is an example of valid usage: var xhr = new XMLHttpRequest(); xhr.open(GET,/resource,true); xhr.extraConnection = true; var succesful = xhr.extraConnection; Setting the extraConnection property should throw an exception if it is called before a URL is provided to the XHR object (through the open method or the constructor). The extra connection status remains in effect until the connection is closed (by network error on normal termination), at which point the property should return false. Since the extraConnection property usually indicates that the response will be a long-lived streaming response, user agents SHOULD NOT pipeline requests on this connection unless the author explicitly specifies such pipelines using pipelining control (see the Pipelining Control Proposal).
Multipart Support Proposal
Multipart Support
Multipart support specifies that when an XHR request is made with multipart
enabled, the XHR response should be parsed as a multipart message, where
each part triggers the XHR response sequence. Multipart handling is enabled
by setting the multipart property on the XHR object to true. When the
response begins to stream into the user agent, the XHR response sequence should
be triggered for each part as it is received, and the user agent should not
wait for the entire HTTP response to be received. When multipart is enabled,
the user agent should perform multipart handling for responses with a MIME type
of any multipart type. That is, when a response includes a Content-Type
header with value of multipart/* where the subtype can be any value, then
multipart handling should be performed.
When handling a multipart message, the XHR object should expose an
additional property multipartState. The multipartState property should
provide the status of the XHR request exactly as the readyState property
would when multipart handling is not enabled. As each part is received is
being received, the multipartState should be 3 and when the response is
finished the multipartState should be 4.
The XHR object should also provide two additional methods,
getPartHeader(headerName) and getAllPartHeaders(). These methods should
provide access to the headers of each part. When a part is encountered, and
the headers have been received (readyState of 2 or higher), the
getPartHeader can be called to with a name of a header, and the method
should return the value for that header within the currently processed part.
The getAllPartHeaders can be called after the headers have been received and
it should return the entirety of the header block text for that part.
When the leading boundary prior to each part is received, the
onreadystatechange handler should be called with the readyState property set
to 1, and responseText should be an empty string. When the headers have been
received for the part (that is, two line breaks are encountered), the
onreadystatechange handler should be called with the readyState property set
to 2 and the getPartHeader and getAllPartHeaders should be able to return
header information. When any of the content of the part is received, the
onreadystatechange handler should be called with the readyState property set
to 3 and the responseText property with the available content. This may be
called multiple times as content is streamed in. When the ending boundary is
encountered, the onreadystatechange handler should be called with the
readyState property set to 4, and the responseText property with the content
of the part and the onload handler should be called. Note that since the
ending boundary is the starting boundary for the next part the beginning
onreadystatechange handler should be called for the next part immediately
following the last call for the previous part.
The following is an example of usage:
var xhr = new XMLHttpRequest;
xhr.multipart = true;
xhr.open(GET,/resource, true);
xhr.onreadystatechange=function() {}
xhr.send(null);
message/http Support
In addition to multpart/* handling, the message/http MIME
should also be handled. The message/http type consists of 1 or more
HTTP response messages. This format only differs from multipart/message
in that there are no boundaries. Boundaries are not necessary with HTTP
messages since each message defines it's own ending (via the
Content-Length header or chunked encoding termination) so boundaries
are unnecessary overhead. Each HTTP message would be treated as a
part, with the responseText being set to the response entity and
getPartHeader and getAllPartHeaders accessing the headers. In addition,
the XHR object's status and statusText properties would reflect the
status values in each of the contained HTTP messages.
Differences from Mozilla Implementation
Multipart support has been implemented in Mozilla with some notable
usability issues that are fixed in this proposal. First, there is no way to
detect when a multipart response is finished since each part triggers a
readyState of 4. Therefore this proposal adds a multipartState property
provides overall response status. Second, Mozilla provides no access to the
headers in each part. This is corrected by adding the getPartHeaders and
getAllPartHeaders methods (which behave very similar to getResponseHeader
and getAllResponseHeaders, except on the part instead of the whole
response). Third, when multipart is enabled, Mozilla only handles responses
with the MIME type of multipart/x-mixed-replace. This greatly limits the use
of more meaningful and accurate MIME types such as multipart/mixed,
multipart/message, and message/http. This proposal removes this limitation
and specifies that multipart handling can be used for any multipart response.
Re: Pipelining Control Proposal
Just about every feature available to web developers is non-uniformly available! Pipelining is vastly more broadly available than just about anything we have available to us. And arguments based on the unreliability of pipelining have still lacked any hard evidence. Dropping a proposal shouldn't be based on FUD. The bottom line is even if some proxies don't handle pipelining, application authors should have the choice. Authors are aware of the risks and can make informed decisions when other browser features aren't uniformly available, and they should have the choice in this matter as well. We shouldn't just assume that all developers are stupid and just throw are hands up in despair that we can never have pipelining. That won't lead to progress. This is our best opportunity to have an inroad to pipelining, via consenting authors. Kris - Original Message - From: Jonas Sicking [EMAIL PROTECTED] To: Kris Zyp [EMAIL PROTECTED] Cc: public-webapi@w3.org; Mark Baker [EMAIL PROTECTED] Sent: Tuesday, February 19, 2008 11:26 AM Subject: Re: Pipelining Control Proposal I would be very worried about implementing this feature in a browser since it runs a very big risk of creating websites that only work for some users. I.e. for users with a direct connection to the server the website would work fine, but for users sitting behind a proxy or a firewall the site would break. I doubt we can expect web authors to appropriately write fallback code when in their testing pipelineing will work fine. / Jonas Kris Zyp wrote: Pipelining Control HTTP Pipelining is when more than one outstanding request is sent over a single TCP connection, and it was introduced in HTTP 1.1. This proposal defines that XHR objects should be able to control whether or not they are pipelined. A pipeline property would be added to the XHR object. If pipeline property is set to true, when send is called, the XHR request SHOULD be pipelined over one of the currently active connection, even if all connections to the target server are currently waiting for a response. That is, the request should be pipelined if necessary to send it immediately. If there is an available connection is alive, but no responses are waiting, the request should be sent on this connection (just as a non-pipelined request would be). If the pipeline property is set to false, the XHR request SHOULD NOT be pipelined even if the user agent supports and would otherwise pipeline the request. The pipeline property may also be set to another XHR object with an open connection, in which case the request should be pipelined on that specific TCP connection. For example: var xhr1 = new XMLHttpRequest(); xhr1.open(GET,/resource1,true); xhr1.send(null); var xhr2 = new XMLHttpRequest(); xhr2.open(GET,/resource2,true); xhr2.pipeline = xhr1; xhr2.send(null); In this example, both requests should be sent over the same TCP connection. The GET for resource1 should be sent and the GET for resource2 should be pipelined behind the first request on the same connection. If a connection has been marked as an extra connection with the extraConnection property on the XHR object (see the extra connection proposal), that connection should not be used for pipelined requests unless another XHR request explicitly specifies that connection, in which case that XHR request SHOULD be pipelined on that connection. (This is because the extra connection request is generally to be used for long-lived responses that are kept open for server-sent messages. Requests that are sent behind such a request may never receive a response, which should not be the default behavior, but may be explicitly chosen to achieve full asynchronous duplex communication on a single TCP connection, a highly valuable capability channels with server-sent messages.) If a network error occurs while servicing a request, any pipelined requests that are queued behind the first request SHOULD NOT automatically be retried by the user agent. A network error in response to the first request should cause an error condition for both the first XHR object and all subsequent pipelined XHR objects per normal XHR behavior. It is the responsible of the application author to retry the request if desired. Note that RFC 2616 states that non-idempotent methods should not be pipelined. However, this SHOULD NOT be enforced by the user agent. Authors should follow the recommendation of RFC 2616, but if they do not and choose to pipeline a non-idempotent request, the user agent should oblige.
Re: Extra Connection Support Proposal
As with pipelining, I think this would be better handled at the HTTP level than the XHR API level. We could define response headers for a server to indicate that it allows more than two connections per client, or alternately that a specific connection should not count towards the limit. I don't have a big problem with this being defined by headers, that seems reasonable. It is worth noting that doesn't achieve quite the same effect as doing it in XHR since there is time elapsed between the time the request is made and the response headers are received. If two connections are currently servicing responses, and third connection wishes to declare itself as an extra connection, from the client side this could be delivered immediately, but with header based permission, it would have to wait it's turn. That being said, I don't think these are terrible issues, and a fixing it at the HTTP level is fine with me. Kris
Re: multipart, server-sent events, and
state of these issues. I will do my best to find whatever information I can. I appreciate that, I will try to research it as well. Five years is a long time in on the web, a lot could have changed. Some evidence I have turned up with a little research: Opera supports pipelining by default. Firefox allows users to enable pipelining and according to Robert, it will be enabled by default. This indicates that pipelining has been tested by browsers in the wild, it is not without some validity. They both use heuristics to determine if pipelining is appropriate. This most notable heuristic is disabling pipelining for IIS servers, which is another case where origin server knowledge properly facilitates informed consent. where it will fail, it cannot detect the failure up front or even after the fact. The result of proceeding in the face of failure will not be a clear failure but rather data corruption. While less than 1% of users may experience these kinds of problems, an Once again, why not let authors make this decision? There are other browser quirks that lead to unpredicable results. If developers think %1 is worth risk, they should be able to do that. There may be clever ways to work around these issues that we haven't even thought of yet. And once again some developers are developing for intranets where the routes are controlled. Why penalize them due to uncertainties on the open internet? XMLHttpRequest is much different than having some menu option to enable pipelining. Many web developers use frameworks that wrap XHR, but those that actually directly interact with XHR are usually doing so because they want precise control of their requests. Why not let them have it? They can read the literature on how dangerous it is and choose if it is worth it. The fact that Opera and now Firefox are doing so broadly certainly would suggest that it is a valid enough choice that devs could at least opt-in. I would ask further why HTTP pipelining is especially more important for XHR than for normal fetching of HTTP resources. Fundamentally, it is nothing more than a performance optimization, and even the most AJAXey of sites makes many more ordinary resource requests than XHR requests. I am not opposed to pipelining of all resources, but XHR provides an API for opt-in, so devs can explicitly choose to pipeline. Second, sometimes pipelining can actually be detrimental, if a slow resource ends up having several requests that become queued behind it. With explicit pipelining, developers can have control and ensure the correct delegation of pipeline requests for optimum efficiency. Finally, in particular with comet-style long-lived responses, being able to control pipelining is even more important. One can pipeline requests over the long-lived response connection to create a duplex communication over a single TCP connection. One can pipeline requests to another connection to ensure that responses are received properly. There is simply no way to do this efficiently otherwise, and comet-style communication requires two connections for duplex conversation even though it is fully with the means of HTTP do it over a single connection. I think we need some research before making a decision either way. I just wanted to make clear the nature of the risks, and in particular that content author opt-in was not sufficient to fully mitigate them. Freedom is not intended to be a risk mitigation. Opt-in is a choice to accept risk, not mitigate it. appropriate for this purpose. So I would propose that we define an HTTP response header field of Connection: pipeline to indicate that the server supports pipelining and encourages its use for this particular connection. I don't think this is a bad idea. But for the reasons stated above, I still think it is important to have an XHR API to suggest pipelining. Thanks for your feedback. Kris
Re: multipart, server-sent events, and
Hmm, I wouldn't think so because that would be tunneling. The difference between message separation with MIME multi-parts and HTTP messages is that multi-parts are separated by boundaries and HTTP messages are generally encapsulated by Content-Length (other ways too). The difference in efficiency is actually probably a wash. I just believe the semantics of HTTP messages are highly meaningful on the web. multipart/message is another another multipart format that indicates each part may have more headers than simply Content-Type (and transfer encoding), and therefore is more flexible than multipart/mixed, IMO... Anyhow, it doesn't really matter, as I wasn't necessarily trying to define something new. I was - in a round about way - trying to ask whether it was worth living with the limitations of x-mixed-replace if there was only one implementation of it around. If other browsers support it, then I'd guess it is worth it. Alex says it almost works in Safari - presumbly via Webkit functionality - so that's a bunch of browsers there. Even if it is based on a current implementation, the standard will be new, and it seems like we might as well as allow for other content types that are more reasonable than x-mixed-replace. It is not high-cost and it seems like good progress towards a more sane future. doing per-document connection accounting all the time would certainly not be opposed by us. I wonder if this is really necessary, since due to the same-domain constraint, publishers already have a large degree of control over how many connections their servers see since they're authoring the pages that create those connections. I haven't given this as much thought as you guys though, so perhaps I'm missing something. Publishers don't have control over how many tabs the user opens for the same site though. Four tabs to the same site means that the each tag/page may want it's own persistent connection to the server, but it is still one browser/agent. Another functionality that I believe would be extremely valuable to expose for XHR would be HTTP pipelining control. Most browsers do not provide HTTP pipelining because of compatibility concerns and performance implications of improperly order requests. I thought it was just that most proxies don't support it on the outbound connection.And I've never seen any ordering problems from the support, or lack thereof, of pipelining. Yes, this is true. But browsers (at least firefox) do use heuristics to determine whether or not to pipeline requests (even when it is enabled). Admittedly, I don't what those heuristics are, but I presumed they were guesses about the extent of support. Also pipelining can possibly slow things down if a certain request takes a long time and it has requests stacked behind (that otherwise would have been diverted to another connection). All of these things web devs could have knowledge and control over though, and so opt-in pipelining could avoid these issues. FWIW, I suspect pipelining control could be targeted for XHR2. Would you be able to write this up as a concrete proposal for additions to one or both of the existing XHR spec or the (TBD) XHR2 spec? Yes. Are you referring specifically to pipelining, or other matters discussed? Either I will do that. Thank you, Kris
