RE: Lazy Blob
Hi Glenn,
var xhr = new XMLHttpRequest();
xhr.open(GET, resource.jpg);
var urlObject = xhr.getURLObject();
var newURL = URL.getObjectURL(urlObject);
img.src = newURL;
It's neat and I like the idea, too. However, there is a reason that I prefer
our blob approaches:
=== Another XHR Approach
partial interface XMLHttpRequest {
Blob makeLazyBlob ();
};
Usage:
[Service]
var xhr = new XMLHttpRequest();
xhr.open(GET, /kitten.png, true);
xhr.setRequestHeader(Authorization, Basic DEADBEEF);
var blob =xhr.makeLazyBlob();
window.intent.postResult(blob);
From totally client developer's point of view, who perhaps do not care the
underlying details at all, it is absolutely transparent to use the obtained
blob as they used to deal with. (no matter which type of data the blob
contains)
[Client]
navigator.startActivity(intent, resultOK);
function resultOK (data) {
// *data* is a blob delivered by service end.
it can be a few MB of image file (kitten.png in the above example)
or a few MB of mp3 music file
or a few MB of raw text file
or a huge binary string
etc.
var blob = data;
var reader = new FileReader();
// developers have all the existing file reader options with no
additional concern such that they do any further operations with the result
reader.readAsDataURL(blob);
// or
reader.readAsBinaryString(blob);
// or
reader.readAsText(blob);
// or
reader.readAsArrayBuffer(blob);
}
Regards,
Jungkee
Jungkee Song
Samsung Electronics
Re: Lazy Blob
On Tue, Aug 7, 2012 at 4:07 AM, Jungkee Song jungkee.s...@samsung.comwrote: window.intent.postResult(blob); From totally client developer's point of view, who perhaps do not care the underlying details at all, it is absolutely transparent to use the obtained blob as they used to deal with. (no matter which type of data the blob contains) If you go down the nullable size route, the client developers do have to care: every piece of code that receives a Blob would have to test to make sure that the code still works if size is null. That said, if you want to go that way, you can make URLObject and Blob more interchangeable to users: make FileReader accept Blob or URLObject, so you can use it to read from URLObject just like Blob. Users will still have to be wary of the lack of .size and .type, but there's no avoiding that. This has the major benefit of keeping Blob's semantics very clear, as it doesn't make Blob.size itself nullable. A different option, equivalent to users, is to make URLObject a base class of Blob. URLObject would replace Blob in methods like FileReader.readAsDataURL, createObjectURL and all other places where methods can work without knowing the size in advance. It would have no methods or attributes (at least at the start). In other words, - URLObject represents a resource that can be fetched, FileReader'd, createObjectURL'd, and cloned, but without any knowledge of the contents (no size attribute, no type attribute) and no slice() as URLObjects may not be seekable. - Blob extends URLObject, adding size, type, slice(), and the notion of representing an immutable piece of data (URLObject might return different data on different reads; Blob can not). Aside from that, the previous URLObject approach is unchanged: creating URLObject requires no network fetches. This wouldn't change the behavior of Blobs themselves in any way. (I don't like listing multiple options. The latter option above seems like the better design. I only listed the simpler option as its lower surface area may make it easier to get traction with.) -- Glenn Maynard
Re: Lazy Blob
On Aug 7, 2012, at 17:06 , Glenn Maynard wrote: A different option, equivalent to users, is to make URLObject a base class of Blob. URLObject would replace Blob in methods like FileReader.readAsDataURL, createObjectURL and all other places where methods can work without knowing the size in advance. It would have no methods or attributes (at least at the start). In other words, - URLObject represents a resource that can be fetched, FileReader'd, createObjectURL'd, and cloned, but without any knowledge of the contents (no size attribute, no type attribute) and no slice() as URLObjects may not be seekable. - Blob extends URLObject, adding size, type, slice(), and the notion of representing an immutable piece of data (URLObject might return different data on different reads; Blob can not). +1 from me on this one. I get a sense that this could possibly be a consensus position (or at least I'm going to claim that it is so as to get disagreement to manifest). Assuming it is, the next steps are: • Having agreed on a solution, do we agree on the problem? (i.e. would this get implemented?) • If so, we can bake this as a standalone delta spec but it would make more sense to me to make the changes directly to the relevant specs, namely FileAPI and XHR. I've copied Anne, Arun, and Jonas — any thought? In either case, I'm happy to provide the content. -- Robin Berjon - http://berjon.com/ - @robinberjon
RE: Lazy Blob
- URLObject represents a resource that can be fetched, FileReader'd, createObjectURL'd, and cloned, but without any knowledge of the contents (no size attribute, no type attribute) and no slice() as URLObjects may not be seekable. - Blob extends URLObject, adding size, type, slice(), and the notion of representing an immutable piece of data (URLObject might return different data on different reads; Blob can not). +1 from me on this one. +1. I get a sense that this could possibly be a consensus position (or at least I'm going to claim that it is so as to get disagreement to manifest). Assuming it is, the next steps are: . Having agreed on a solution, do we agree on the problem? (i.e. would this get implemented?) . If so, we can bake this as a standalone delta spec but it would make more sense to me to make the changes directly to the relevant specs, namely FileAPI and XHR. I've copied Anne, Arun, and Jonas - any thought? In either case, I'm happy to provide the content. Having hammered out a consensus, I would like to contribute to providing the content. Jungkee Jungkee Song Samsung Electronics
Re: Lazy Blob
On Tue, Aug 7, 2012 at 6:53 PM, Jungkee Song jungkee.s...@samsung.comwrote: - URLObject represents a resource that can be fetched, FileReader'd, createObjectURL'd, and cloned, but without any knowledge of the contents (no size attribute, no type attribute) and no slice() as URLObjects may not be seekable. - Blob extends URLObject, adding size, type, slice(), and the notion of representing an immutable piece of data (URLObject might return different data on different reads; Blob can not). +1 from me on this one. +1. I get a sense that this could possibly be a consensus position (or at least I'm going to claim that it is so as to get disagreement to manifest). Assuming it is, the next steps are: . Having agreed on a solution, do we agree on the problem? (i.e. would this get implemented?) . If so, we can bake this as a standalone delta spec but it would make more sense to me to make the changes directly to the relevant specs, namely FileAPI and XHR. I've copied Anne, Arun, and Jonas - any thought? In either case, I'm happy to provide the content. Having hammered out a consensus, I would like to contribute to providing the content. I would suggest using a different name than URLObject. I think that name will cause a lot of head scratching.
Re: Lazy Blob
On Tue, Aug 7, 2012 at 8:14 PM, Glenn Adams gl...@skynav.com wrote: I would suggest using a different name than URLObject. I think that name will cause a lot of head scratching. No disagreement there; that was just a placeholder. I'd suggest waiting for further input from Anne, Jonas and Arun (the editors of the specs in question) before spending much time coming up with a name, though. -- Glenn Maynard
Re: Lazy Blob
On Tue, Aug 7, 2012 at 7:38 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Aug 7, 2012 at 8:14 PM, Glenn Adams gl...@skynav.com wrote: I would suggest using a different name than URLObject. I think that name will cause a lot of head scratching. No disagreement there; that was just a placeholder. I'd suggest waiting for further input from Anne, Jonas and Arun (the editors of the specs in question) before spending much time coming up with a name, though. sure... i don't have a suggested replacement, but i know a bad name when i see one; i'll defer to the editors to come up with something reasonable
Re: Lazy Blob
On Aug 2, 2012, at 14:51 , Tobie Langel wrote: On 8/2/12 2:29 PM, Robin Berjon ro...@berjon.com wrote: On Aug 2, 2012, at 10:45 , Tobie Langel wrote: On 8/1/12 10:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. Yes. Remote blob is more accurate and should help think about this problem in a more meaningful way. Actually, you need both to be accurate. With the current stack you can have lazy blobs, and you can have remote blobs, but you can't have both at the same time. If we're going to be strict about naming this, we're talking about remote lazy blobs. What's a remote blob in the current stack? Setting responseType to blob on an XHR request. -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: Lazy Blob
On Aug 2, 2012, at 17:45 , Glenn Adams wrote: Are you saying I am objecting for the fun of it? Where did I say I don't like the idea? You'd best reread my messages. For the fun of it is an expression. You don't like the idea that the solutions proposed in this thread are restricted to what is supported by XHR, but have shown no indication of having encountered problems with this restriction (if restriction it is, which I don't believe). Put differently, based on your messages, which I have read, you appear to be arguing from technical purity rather than from technical need. If you want a real world use case it is this: my architectural constraints as an author for some particular usage requires that I use WS rather than XHR. I wish to have support for the construct being discussed with WS. How is that not a real world requirement? Maybe there's a real world requirement underlying this that you're not stating — but it's not stated and so I can't just guess it. If you go back to my initial message, you will see that the issue I opened is based on a genuine problem that Jungkee bumped into while developing a Web application, for which we could find no proper workaround. Shortly thereafter, I also found the same problem, and solved it by simply dropping the feature (in this case, no pictures in an address book) — which is obviously far from ideal. When discussing it with others, several folks mentioned bumping into that limitation as well. I don't think we're just a bunch of crazy people and that we've hit this issue completely from left-field — indeed with usage of postMessage() increasing (and made all the more powerful with Intents) it seems highly likely to be a wall that other Web hackers will hit. In contrast, what you cite above as a use case seems rather abstract and — at least to me — contrived. I have some difficulty conceiving a situation, certainly not one common enough, in which one may be able to use WS but not XHR. Reading the above sounds to me like objecting to XHR not supporting telnet/RFC15 because if I stretch my imagination in just the right way I can conceive of a situation in which only telnet is available and HTTP isn't. So if you do have a use case, by all means please share it. If not, I maintain that you simply have no grounds for objection. -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: Lazy Blob
Hi Glenn,
On Aug 3, 2012, at 01:23 , Glenn Maynard wrote:
I'd suggest the following.
- Introduce an interface URLObject (bikeshedding can come later), with no
methods. This object is supported by structured clone.
- Add XMLHttpRequest.getURLObject(optional data), which returns a new
URLObject. This can only be called while XMLHttpRequest is in the OPENED
state. The returned URLObject represents the resource that would have been
fetched if xhr.send() had been called. No XHR callbacks are performed, and
no network activity takes place. The data argument acts like send(data),
to specify the request entity body that will be sent.
- Adjust URL.createObjectURL to take (Blob or URLObject), which returns a URL
that works the same way as Blob URLs.
Example:
var xhr = new XMLHttpRequest();
xhr.open(GET, resource.jpg);
var urlObject = xhr.getURLObject();
var newURL = URL.getObjectURL(urlObject);
img.src = newURL;
I like this idea, but I'm not certain if it differs that much from one of the
options I listed (albeit a fair bit less clearly, and in the middle of a
shopping list, which could explain why you overlooked it). Quoting:
=== Another XHR Approach
partial interface XMLHttpRequest {
Blob makeLazyBlob ();
};
Usage:
var xhr = new XMLHttpRequest();
xhr.open(GET, /kitten.png, true);
xhr.setRequestHeader(Authorization, Basic DEADBEEF);
var blob =xhr.makeLazyBlob();
Unless I missed something, the only differences are the name (which we can
bikeshed on later indeed — I certainly am not married to lazy blobs though it
would make for a finer insult than URLObject) and that you mint a new object
while I reuse Blob. Having mulled this over the weekend, the tradeoffs
(assuming that both work the same, i.e. like send() with no eventing, etc.) are:
Using Blob
• It doesn't introduce a new interface;
• Blobs can already be scloned;
• Blobs already work with getObjectURL which means that part needn't change
either.
But
• The Blob's |size| attribute cannot be set (without resorting to HEADs, which
blows for a bunch of reasons),
• It's a little weird that it seems to duplicate responseType = blob, the
primary difference (that developers are likely to ever care about) is that the
network request is deferred (or… lazy ;).
I wonder if we could circumvent the |size| issue by allowing Blobs to return
null for that (making the type unsigned long long?). I understand how Jonas
sees this as making it closer to Stream, but I think it's the primary way in
which it is. It seems more logical to me to occasionally have to deal with
files the size of which you don't know, than to e.g. assign a stream to an
img.
Frankly I can easily live with either and will be more than happy to bow to
implementer preference between the two; what I mostly need is a way of
exchanging pointers to remote resources in the way described in the original
use cases.
For URLObject, you mention the case of passing it to XHR:
var urlO = getURLObject(); // this comes from a service or something
var newURL = URL.getObjectURL(urlO);
var xhr = new XMLHttpRequest();
xhr.open(GET, newURL, false);
xhr.responseType = blob;
xhr.send();
var blob = xhr.response;
The ability to get a Blob (rather than just a blob URL) is vitally useful if
you wish to store the information, e.g. in an IndexedDB. So long as I can get
one, even if it's a bit more convoluted, I'm happy.
Passing one of these URLs back to XHR would need extra consideration (eg.
should you be able to specify more headers?).
I would assume that the request to the blob URL would work just like any
request to a blob URL (you can only use GET, setting headers does nothing
useful, etc.). None of this would have any effect whatsoever on the XHR from
which the URLObject was created (anything else would be a likely attack vector
and does not seem useful).
(Note that I've spent some time thinking about this because I think it's
technically interesting, but I haven't looked over the use cases closely
enough to say whether I think it'd be worthwhile or not.)
Well, it's an issue that a few of us have bumped into — so I think it's useful
:)
--
Robin Berjon - http://berjon.com/ - @robinberjon
Re: Lazy Blob
On Mon, Aug 6, 2012 at 6:53 AM, Robin Berjon ro...@berjon.com wrote: So if you do have a use case, by all means please share it. If not, I maintain that you simply have no grounds for objection. I did share a couple of use cases in my response to Ian: On Thu, Aug 2, 2012 at 11:39 AM, Glenn Adams gl...@skynav.com wrote: On Thu, Aug 2, 2012 at 11:26 AM, Ian Hickson i...@hixie.ch wrote: On Thu, 2 Aug 2012, Glenn Adams wrote: Are you asking for use cases for a remote/lazy blob in general? i.e., as would apply to the proposed XHR usage and any other underlying supported data source? or are you asking about high level use cases that are particular to a WS binding but not an XHR binding? Both would be useful, but my primary concern is Web Sockets, since I edit that spec. Before I can consider proposals that affect Web Sockets, I need to know what use case it is we're trying to address. I will let Robin and Jungkee reply to the more general use case requirements. As far as WS is concerned, I don't see any impact of this thread on the WS API or WSP specs, its really simply an application of WS/WSP to remote/lazy blobs. Clearly, there are many high level use cases that involve a repetitive send/response message paradigm, which can certainly be implemented with XHR, but some application authors would prefer using WS for various efficiency reasons. My suggestion is essentially: if we are going to define a remote blob bound to an XHR source for a one-shot send-response, then perhaps we should define a remote blob bound to a WS source for multiple send-response pairs. For example, a symmetric presence protocol or IM protocol would typically fall into this usage category. Using remote blobs for either the send or response data (or both) would be useful for certain architectures and provide more deployment flexibility and perhaps greater efficiencies.
Re: Lazy Blob
On Mon, 6 Aug 2012, Glenn Adams wrote: I did share a couple of use cases in my response to Ian: I will let Robin and Jungkee reply to the more general use case requirements. As far as WS is concerned, I don't see any impact of this thread on the WS API or WSP specs, its really simply an application of WS/WSP to remote/lazy blobs. Clearly, there are many high level use cases that involve a repetitive send/response message paradigm, which can certainly be implemented with XHR, but some application authors would prefer using WS for various efficiency reasons. My suggestion is essentially: if we are going to define a remote blob bound to an XHR source for a one-shot send-response, then perhaps we should define a remote blob bound to a WS source for multiple send-response pairs. For example, a symmetric presence protocol or IM protocol would typically fall into this usage category. Using remote blobs for either the send or response data (or both) would be useful for certain architectures and provide more deployment flexibility and perhaps greater efficiencies. Those are still not use cases, for the record. I tried explaining what a use case was here: http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0302.html http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0288.html -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Lazy Blob
On Mon, Aug 6, 2012 at 11:27 AM, Ian Hickson i...@hixie.ch wrote: On Mon, 6 Aug 2012, Glenn Adams wrote: I did share a couple of use cases in my response to Ian: I will let Robin and Jungkee reply to the more general use case requirements. As far as WS is concerned, I don't see any impact of this thread on the WS API or WSP specs, its really simply an application of WS/WSP to remote/lazy blobs. Clearly, there are many high level use cases that involve a repetitive send/response message paradigm, which can certainly be implemented with XHR, but some application authors would prefer using WS for various efficiency reasons. My suggestion is essentially: if we are going to define a remote blob bound to an XHR source for a one-shot send-response, then perhaps we should define a remote blob bound to a WS source for multiple send-response pairs. For example, a symmetric presence protocol or IM protocol would typically fall into this usage category. Using remote blobs for either the send or response data (or both) would be useful for certain architectures and provide more deployment flexibility and perhaps greater efficiencies. Those are still not use cases, for the record. I tried explaining what a use case was here: http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0302.html http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0288.html I'll leave the translation of IM protocol to user facing use case as homework for the reader. It is trivial. My intent is to show a use case where one would use a persistent connection and a series of send/response messages that easily maps to WS. Instant Messaging is such a use case.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 7:59 AM, Robin Berjon ro...@berjon.com wrote:
Hi all,
with the likes of postMessage and Web Intents that we are getting access to
now, it is increasingly common that data may flow from a server to an
in-browser page, that may then pass that data on to another in-browser page
(typically running at a different origin). In a many cases, such data will be
captured as Blobs. There can be several reasons why one may wish to toss a
Blob over postMessage rather than a URL:
• Origin restrictions and lack of CORS support (not necessarily fixable
by the hacker working on the site);
• CSP on the receiving end;
• Cookies or authorisation issues;
• Hotlinking prevention;
• Secrecy of the actual URL to the data (possibly for privacy reasons);
• Probably a bunch of others I haven't thought of.
The problem is that the way that things currently work, the sending end needs
to load the data in full before transmitting it. That's okay when it's small
and there are only few resources, or if you know that the receiving end will
read it all right away anyway. But there are cases in which there is a lot of
data (e.g. an image gallery) or where the receiving end might filter the
data, only load it item per item, etc.
This is something that we noted while working on implementing some Web
Intents based services but that applies more broadly. The most acute example
was Jungkee Song's Pick Image Intent that could easily find itself in a
position to load several hundreds of megabytes that might then just be thrown
away by the receiving end.
The proposals below were hashed out with Jungkee Song and Richard Tibbett,
based on the above use case as well as several people in the Intents and DAP
groups stating that they bumped into situations where they would have needed
this (feel free to send additional use cases folks).
POTENTIAL SOLUTIONS
===
The overall idea is simple irrespective of which approach is chosen: create a
Blob that can lazily fetch the content of a URL. Blobs are designed to be
transparently lazy if they need to, so nothing needs to change at the Blob
level, or for that matter at the level of anything that may end up reading
from one. In fact, a good implementation that reads file-based Blobs is
probably already lazy. A Lazy Blob is just a Blob.
Things start to get messier (unless we've missed an option) when you plug
that into URLs (don't they always). That's where the multiple approaches
outlined below kick in.
In all approaches it is assumed that if there is information to be inferred
from context (e.g. access to cookies) then the relevant context is the one in
which the Blob is created. Reading of the Blob performed in another context
changes nothing to that.
User agents are always allowed to access the actual data at any arbitrary
moment, between immediately and when code requests it. Of course, quality of
implementation may dictate a strategy that doesn't make it just as bad as
loading everything immediately.
Also, none of the options below are detailed anywhere near where they should
be — but if we agree on the need and on a general direction I'll be happy to
take care of that. If there's agreement on this, I volunteer as editor
(Jungkee and perhaps Richard may be interested as well).
=== The Simple Approach
partial interface BlobBuilder {
Blob getBlobFromURL (DOMString url);
};
Usage:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;);
This is the simplest possible approach. When called, essentially the
equivalent of this happens:
var xhr = new XMLHttpRequest();
xhr.open(GET, url, true);
xhr.responseType = blob;
And upon reading the blob, send() is triggered and everything else happens as
if the XHR blob from xhr.response were accessed directly.
Pro: Extremely simple.
Con: Does not allow much control over the request that gets made, notably
some uses will likely require setting headers.
=== The Somewhat Less Simple Approach
partial interface BlobBuilder {
Blob getBlobFromURL (DOMString url, optional DOMString method, optional
Object headers);
};
Usage:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET, {
Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
This is currently my preferred approach. If you like it too, you may consider
the following ones as merely curiosities.
=== Using XHR For Options
partial interface BlobBuilder {
Blob getBlobFromURL (XMLHttpRequest xhr);
};
Usage:
var bb = new BlobBuilder()
, xhr = new XMLHttpRequest();
xhr.open(GET, /kitten.png, true);
xhr.setRequestHeader(Authorization, Basic DEADBEEF);
var blob =
Re: Lazy Blob
On Mon, Aug 6, 2012 at 8:39 PM, Glenn Adams gl...@skynav.com wrote: I'll leave the translation of IM protocol to user facing use case as homework for the reader. It is trivial. My intent is to show a use case where one would use a persistent connection and a series of send/response messages that easily maps to WS. Instant Messaging is such a use case. What is it exactly that requires you to use a remote blob with type blob in the browser over a WS you cannot achieve with a WS and array buffers?
Re: Lazy Blob
On Mon, Aug 6, 2012 at 1:31 PM, Florian Bösch pya...@gmail.com wrote: On Mon, Aug 6, 2012 at 8:39 PM, Glenn Adams gl...@skynav.com wrote: I'll leave the translation of IM protocol to user facing use case as homework for the reader. It is trivial. My intent is to show a use case where one would use a persistent connection and a series of send/response messages that easily maps to WS. Instant Messaging is such a use case. What is it exactly that requires you to use a remote blob with type blob in the browser over a WS you cannot achieve with a WS and array buffers? The same reason that a remote blob would be useful with XHR.
Re: Lazy Blob
On Mon, Aug 6, 2012 at 2:16 PM, Jonas Sicking jo...@sicking.cc wrote: Since it appears my original email went unnoticed, here's another try. I think what you are looking for is Streams. The resulting code would be something like: var xhr = new XMLHttpRequest(); xhr.responseType = stream; xhr.open(...); xhr.send(); var stream = xhr.result; otherGuy.postMessage(stream); I don't recall that proposal, and the name is too generic to search for. Can you give us a link? -- Glenn Maynard
Re: Lazy Blob
On Mon, Aug 6, 2012 at 9:33 PM, Glenn Adams gl...@skynav.com wrote: The same reason that a remote blob would be useful with XHR. Since you're steadfastly refusing to detail your use case, that'll just mean none to me.
Re: Lazy Blob
On Mon, Aug 6, 2012 at 12:33 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Aug 6, 2012 at 2:16 PM, Jonas Sicking jo...@sicking.cc wrote: Since it appears my original email went unnoticed, here's another try. I think what you are looking for is Streams. The resulting code would be something like: var xhr = new XMLHttpRequest(); xhr.responseType = stream; xhr.open(...); xhr.send(); var stream = xhr.result; otherGuy.postMessage(stream); I don't recall that proposal, and the name is too generic to search for. Can you give us a link? Arthur was awesome enough to dig it up: http://lists.w3.org/Archives/Public/public-webapps/2011OctDec/1494.html http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm Though we would need to modify that proposal such that .response returns a stream also in the OPENED state as long as the send() flag is set. / Jonas
Re: Lazy Blob
On Mon, Aug 6, 2012 at 2:06 PM, Florian Bösch pya...@gmail.com wrote: On Mon, Aug 6, 2012 at 9:33 PM, Glenn Adams gl...@skynav.com wrote: The same reason that a remote blob would be useful with XHR. Since you're steadfastly refusing to detail your use case, that'll just mean none to me. I feel I don't have any obligation to justify the use of WS any more than that necessary for XHR. It is simply short-sighted to define a remote blob only for XHR. If you can't see that, then let's not waste our time continuing this thread.
Re: Lazy Blob
On Mon, Aug 6, 2012 at 3:28 PM, Jonas Sicking jo...@sicking.cc wrote: Arthur was awesome enough to dig it up: Thanks. http://lists.w3.org/Archives/Public/public-webapps/2011OctDec/1494.html http://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm Though we would need to modify that proposal such that .response returns a stream also in the OPENED state as long as the send() flag is set. This is actually very different, I think. Looking at the Stream proposal, it looks like calling client.send() actually kicks off a fetch, just as with any other XHR responseType, and then the Stream object you get from client.response represents the data coming in from that fetch. (I may be misinterpreting this; section 10 is written as a delta spec, which makes it a bit hard to follow.) URLObject doesn't represent a fetch; it represents the data required to perform a fetch. This means that you don't need to perform a fetch at all in order to create the URLObject. (For example, if you're creating a playlist of 1 songs in a Song Manager app and you want to hand those off to a Song Player app via an intent, you don't want to perform 1 fetches to create those objects, not even HEAD requests.) A fetch only occurs if the URLObject is actually used, and also importantly, you can reuse the URLObject in as many fetches as you want and get the same data (eg. play a song multiple times). In other words, a URLObject works just like a URL, except with some metadata attached (headers, credentials) and without exposing the actual URL to the receiver. It's not exactly clear to me what happens, in the Stream proposal, if (in the second example in the introduction) you assign a URL pointing to theStream to multiple objects. Do the receivers of the URL just start receiving data at whatever point the stream is at at the time (which would break most protocols)? With URLObject, this simply results in two fetches. Also, Streams aren't seekable. (That would require a major overhaul of StreamBuilder to support. I think I remember discussing this at one point: you'd need a pull interface--give me 1000 bytes at position 10--rather than a push interface. I think I discarded that idea since the particular sequence of fetch callbacks would be impractical to specify.) URLObject would allow media seeking, if they point at a video or audio stream, since they represent a resource rather than a fetch. -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 7:59 AM, Robin Berjon ro...@berjon.com wrote:
Hi all,
with the likes of postMessage and Web Intents that we are getting access to
now, it is increasingly common that data may flow from a server to an
in-browser page, that may then pass that data on to another in-browser page
(typically running at a different origin). In a many cases, such data will be
captured as Blobs. There can be several reasons why one may wish to toss a
Blob over postMessage rather than a URL:
• Origin restrictions and lack of CORS support (not necessarily fixable
by the hacker working on the site);
• CSP on the receiving end;
• Cookies or authorisation issues;
• Hotlinking prevention;
• Secrecy of the actual URL to the data (possibly for privacy reasons);
• Probably a bunch of others I haven't thought of.
The problem is that the way that things currently work, the sending end needs
to load the data in full before transmitting it. That's okay when it's small
and there are only few resources, or if you know that the receiving end will
read it all right away anyway. But there are cases in which there is a lot of
data (e.g. an image gallery) or where the receiving end might filter the
data, only load it item per item, etc.
This is something that we noted while working on implementing some Web
Intents based services but that applies more broadly. The most acute example
was Jungkee Song's Pick Image Intent that could easily find itself in a
position to load several hundreds of megabytes that might then just be thrown
away by the receiving end.
The proposals below were hashed out with Jungkee Song and Richard Tibbett,
based on the above use case as well as several people in the Intents and DAP
groups stating that they bumped into situations where they would have needed
this (feel free to send additional use cases folks).
POTENTIAL SOLUTIONS
===
The overall idea is simple irrespective of which approach is chosen: create a
Blob that can lazily fetch the content of a URL. Blobs are designed to be
transparently lazy if they need to, so nothing needs to change at the Blob
level, or for that matter at the level of anything that may end up reading
from one. In fact, a good implementation that reads file-based Blobs is
probably already lazy. A Lazy Blob is just a Blob.
Things start to get messier (unless we've missed an option) when you plug
that into URLs (don't they always). That's where the multiple approaches
outlined below kick in.
In all approaches it is assumed that if there is information to be inferred
from context (e.g. access to cookies) then the relevant context is the one in
which the Blob is created. Reading of the Blob performed in another context
changes nothing to that.
User agents are always allowed to access the actual data at any arbitrary
moment, between immediately and when code requests it. Of course, quality of
implementation may dictate a strategy that doesn't make it just as bad as
loading everything immediately.
Also, none of the options below are detailed anywhere near where they should
be — but if we agree on the need and on a general direction I'll be happy to
take care of that. If there's agreement on this, I volunteer as editor
(Jungkee and perhaps Richard may be interested as well).
=== The Simple Approach
partial interface BlobBuilder {
Blob getBlobFromURL (DOMString url);
};
Usage:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;);
This is the simplest possible approach. When called, essentially the
equivalent of this happens:
var xhr = new XMLHttpRequest();
xhr.open(GET, url, true);
xhr.responseType = blob;
And upon reading the blob, send() is triggered and everything else happens as
if the XHR blob from xhr.response were accessed directly.
Pro: Extremely simple.
Con: Does not allow much control over the request that gets made, notably
some uses will likely require setting headers.
=== The Somewhat Less Simple Approach
partial interface BlobBuilder {
Blob getBlobFromURL (DOMString url, optional DOMString method, optional
Object headers);
};
Usage:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET, {
Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
This is currently my preferred approach. If you like it too, you may consider
the following ones as merely curiosities.
=== Using XHR For Options
partial interface BlobBuilder {
Blob getBlobFromURL (XMLHttpRequest xhr);
};
Usage:
var bb = new BlobBuilder()
, xhr = new XMLHttpRequest();
xhr.open(GET, /kitten.png, true);
xhr.setRequestHeader(Authorization, Basic DEADBEEF);
var blob =
Re: Lazy Blob
On Wed, Aug 1, 2012 at 10:44 PM, Ian Hickson i...@hixie.ch wrote:
On Wed, 1 Aug 2012, Glenn Adams wrote:
Of course, implementers are free to ignore whatever they want, but last
time I checked, the W3C was a consensus based standards organization
which means agreement needs to be reached on what specs go out the door
and what are in those specs.
Doesn't really matter what's in the specs going out the door if it's not
what's implemented...
I don't really care about the XHR side of this (happy to let Anne figure
that out), but since WebSockets was mentioned: what's the use case that
involves Web Socket? I don't really understand what problem we're trying
to solve here.
based on the pattern proposed by
partial interface BlobBuilder {
Blob getBlobFromURL (XMLHttpRequest xhr);
};
i would like to support two use cases:
(1) simple - single blob send, single blob response
(2) multiple - multiple instances of simple, i.e., send/response pairs
these could be handled with the following:
partial interface BlobBuilder {
// simple
Blob getBlobFromSource (WebSocket ws, Blob send);
// multiple
Blob getBlobFromSource (WebSocket ws, EventHandler sendHandler);
};
in the simple case, the creator of the lazy blob provides the initial send
blob, which is sent by the underlying lazy blob implementation upon a read
on the lazy blob, then the next (and only) response blob is returned as a
result from the read
in the multiple case, the creator of the lazy blob provides an event
handler to invoke to send a blob corresponding to a read on the lazy blob,
thus providing for multiple send/receive blob message pairs, with one lazy
blob for each pair
of course, the simple case could be folded into the multiple case, leaving
only one method to define:
partial interface BlobBuilder {
Blob getBlobFromSource (WebSocket ws, EventHandler sendHandler);
};
a use of this might be as follows:
var bb = new BlobBuilder();
var ws = new WebSocket(ws://example.com:/test);
var lb = [];
ws.onopen = function() {
lb.push ( bb.getBlobFromSource(ws, function() {
ws.send(getSendMessageAsBlob(1)); }) );
lb.push ( bb.getBlobFromSource(ws, function() {
ws.send(getSendMessageAsBlob(2)); }) );
lb.push ( bb.getBlobFromSource(ws, function() {
ws.send(getSendMessageAsBlob(3)); }) );
setTimeout(sendAndReceive);
}
function getSendMessageAsBlob(msgNum) {
return new Blob ( [ String(msgNum) ] );
}
function sendAndReceive() {
var numMsgs = 0;
var numBytes = 0;
// trigger read on queued lazy blobs
while ( lb.length 0 ) {
b = lb.shift();
// read on size triggers stored send 'promise'
numBytes += b.size;
numMsgs += 1;
}
alert('Received ' + numMsgs + ' messages, containing ' + numBytes + '
bytes.');
ws.close();
}
of course, this example make use of a particular message paradigm
(send/recv pairs); while this may capture only a subset of interchange
patterns, one could easily generalize the above to provide more flexibility;
Re: Lazy Blob
On Thu, 2 Aug 2012, Glenn Adams wrote: I don't really care about the XHR side of this (happy to let Anne figure that out), but since WebSockets was mentioned: what's the use case that involves Web Socket? I don't really understand what problem we're trying to solve here. i would like to support two use cases: (1) simple - single blob send, single blob response (2) multiple - multiple instances of simple, i.e., send/response pairs Sorry, I was vague. What I mean is what user-facing problem is it that we're trying to solve? For example, I have a real-time game that needs to send commands to the server from the client and needs to receive updates from the server to the client, where latency is not a big issue (it's turn-based or tick-based with widely-spaced ticks) but where reliability is paramount (dropping a message would mean the state was inconsistent) would be a use case for Web Sockets. That use case could be more simply described as I want to write a chess game as a Web app. Another use case for Web Sockets could be I want to write an instant messaging application in my Web page, or I want the view of the data that my application shows to update in place when the server notices that it has changed. Notice the lack of any mention of specific solutions in the use case descriptions -- it's just a high-level problem statement, typically one that even an end-user could actually understand (or at least that a programmer could understand, even if they were not familiar with the Web). What's the use case that is driving the ideas discussed in this thread in the context of Web Sockets? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Lazy Blob
On Thu, Aug 2, 2012 at 1:04 AM, Ian Hickson i...@hixie.ch wrote: On Thu, 2 Aug 2012, Glenn Adams wrote: I don't really care about the XHR side of this (happy to let Anne figure that out), but since WebSockets was mentioned: what's the use case that involves Web Socket? I don't really understand what problem we're trying to solve here. i would like to support two use cases: (1) simple - single blob send, single blob response (2) multiple - multiple instances of simple, i.e., send/response pairs Sorry, I was vague. What I mean is what user-facing problem is it that we're trying to solve? see DAR's initial message in this thread; bringing WS into scope doesn't change the problem statement, it merely enlarges the solution space, or keeps it from being unnecessarily narrow
Re: Lazy Blob
On Aug 1, 2012, at 21:26 , Glenn Adams wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? No one said that they would have to be HTTP specific, but I agree with Florian that I don't see how it could apply to WS. It could, for instance, be made to work fine with FTP or similar resource retrieval protocols. -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: Lazy Blob
On Aug 1, 2012, at 22:13 , Glenn Adams wrote: The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later. Objections need to be built on something — just objecting for the fun of it does not carry some weight. Up to this point, you have provided no real world use case that requires the feature you propose and your sole justification for the whole subthread is that you don't like the idea. As far as I'm concerned, barring the introduction of better arguments the objection is dealt with hic et nunc. -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: Lazy Blob
On 8/1/12 10:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. Yes. Remote blob is more accurate and should help think about this problem in a more meaningful way. --tobie
Re: Lazy Blob
On Thu, 02 Aug 2012 10:45:03 +0200, Tobie Langel to...@fb.com wrote: On 8/1/12 10:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. Yes. Remote blob is more accurate and should help think about this problem in a more meaningful way. Good name, makes it easier to understand why it best applies to transports that represent their resource with a URL pointing directly at it. All this talk about lazy blob was at least confusing me. :-) -- Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com
Re: Lazy Blob
On Aug 2, 2012, at 10:45 , Tobie Langel wrote: On 8/1/12 10:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. Yes. Remote blob is more accurate and should help think about this problem in a more meaningful way. Actually, you need both to be accurate. With the current stack you can have lazy blobs, and you can have remote blobs, but you can't have both at the same time. If we're going to be strict about naming this, we're talking about remote lazy blobs. -- Robin Berjon - http://berjon.com/ - @robinberjon
RE: Lazy Blob
Hi Glenn and all,
From: Glenn Maynard [mailto:gl...@zewt.org]
Sent: Thursday, August 02, 2012 12:45 AM
To: Robin Berjon
Cc: WebApps WG; Jungkee Song
Subject: Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:59 AM, Robin Berjon ro...@berjon.com wrote:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET, {
Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
We already have an API to allow scripts to make network requests: XHR.
Please don't create a new API that will end up duplicating all of that.
However this might be done, it should hang off of XHR.
* Let me use the word *Lazy* until we get a consensus upon the term.
As stated at the proposal, we came up with certain use cases in Web Intents
postResult() callback. In this use case, there is no certain moment at which
the client can explicitly trigger XHR request. Rather, it has to wait huge
Blobs of data (e.g., an array of Media objects in the Pick Media Intent [1]) on
its intents callback.
Here is an example:
[Client]
navigator.startActivity(intent, mediaOK, mediaFail);
function mediaOK (mediaObjectArray) {
// iterate over the array of media objects to do something useful with them
console.log(mediaObjectArray.length + media objects) // In console,
250 media objects
for (var i = 0; i mediaObjectArray.length; i++) {
// read mediaObjectArray[i].content.blob triggers actual resource
transfer
}
}
At the point the mediaOK() callback is called, 250 *Lazy* Blobs from an image
gallery is ready.
[Service]
var mediaObjectArray = [];
// e.g., the service builds *mediaObjectArray* containing hundreds of png files
from an image gallery
{ // loop
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specification.com/kitten00.png;,
GET, { Authorization: Basic DEADBEEF });
var content = {};
content.blob = blob;
mediaObject.content = content;
mediaObjectArray.push(mediaObject);
}
window.intent.postResult(mediaObjectArray);
Personally, I prefer option B (used in this email) as it is fairly simple to
developers while providing useful options.
[1] http://w3c-test.org/dap/gallery/
Regards,
Jungkee
Jungkee Song
Samsung Electronics
Re: Lazy Blob
On 8/2/12 2:29 PM, Robin Berjon ro...@berjon.com wrote: On Aug 2, 2012, at 10:45 , Tobie Langel wrote: On 8/1/12 10:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. Yes. Remote blob is more accurate and should help think about this problem in a more meaningful way. Actually, you need both to be accurate. With the current stack you can have lazy blobs, and you can have remote blobs, but you can't have both at the same time. If we're going to be strict about naming this, we're talking about remote lazy blobs. What's a remote blob in the current stack? --tobie
Re: Lazy Blob
On Thu, Aug 2, 2012 at 2:36 AM, Robin Berjon ro...@berjon.com wrote: On Aug 1, 2012, at 22:13 , Glenn Adams wrote: The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later. Objections need to be built on something — just objecting for the fun of it does not carry some weight. Up to this point, you have provided no real world use case that requires the feature you propose and your sole justification for the whole subthread is that you don't like the idea. Are you saying I am objecting for the fun of it? Where did I say I don't like the idea? You'd best reread my messages. As far as I'm concerned, barring the introduction of better arguments the objection is dealt with hic et nunc. No it hasn't. If you want a real world use case it is this: my architectural constraints as an author for some particular usage requires that I use WS rather than XHR. I wish to have support for the construct being discussed with WS. How is that not a real world requirement?
Re: Lazy Blob
On Thu, Aug 2, 2012 at 9:51 AM, Florian Bösch pya...@gmail.com wrote: On Thu, Aug 2, 2012 at 5:45 PM, Glenn Adams gl...@skynav.com wrote: No it hasn't. If you want a real world use case it is this: my architectural constraints as an author for some particular usage requires that I use WS rather than XHR. I wish to have support for the construct being discussed with WS. How is that not a real world requirement? Your particular use-case of content/range aquisition over WS requires a particular implementation on the server in order to understand the WS application layer protocol. This particular implementation on the server of yours is not implemented by any other common hosting infrastructure based on any kind of existing standard. You should specify this particular protocol standard to be used on top of WS first before you can even discuss how your custom implementation of this protocol justifies enshrining it in a browser standard. All WS usage requires a particular (application specific) implementation on the server, does it not? Notwithstanding that fact, such usage will fall into certain messaging patterns. I happened to give an example of two possible message patterns and showed how the proposal under discussion could address those patterns. It is not necessary to marry my proposal to a specific sub-protocol on WS in order to provide useful functionality that can be exploited by applications that use those functions.
Re: Lazy Blob
On Thu, Aug 2, 2012 at 10:04 AM, Florian Bösch pya...@gmail.com wrote: On Thu, Aug 2, 2012 at 5:58 PM, Glenn Adams gl...@skynav.com wrote: All WS usage requires a particular (application specific) implementation on the server, does it not? Notwithstanding that fact, such usage will fall into certain messaging patterns. I happened to give an example of two possible message patterns and showed how the proposal under discussion could address those patterns. It is not necessary to marry my proposal to a specific sub-protocol on WS in order to provide useful functionality that can be exploited by applications that use those functions. If you wish to introduce a particular browser supported semantic for which a specific implementation on the server is required, then people should be able to consult a standard that tells them how they have to provide this implementation. Therefore it is quite necessary to marry your desire to extend remote blobs to WS to a protocol, otherwise you'll have a browser implemented protocol that nobody knows how to implement. I am not proposing a particular browser supported semantic for a specific implementation on the server. I have suggested, by way of example, two particular patterns be supported independently of any such implementation. I did not restrict the results to just those patterns in case someone wishes to generalize. That is little different from the proposed or implied XHR patterns being discussed.
Re: Lazy Blob
On Thu, 2 Aug 2012, Glenn Adams wrote: Sorry, I was vague. What I mean is what user-facing problem is it that we're trying to solve? see DAR's initial message in this thread; bringing WS into scope doesn't change the problem statement, it merely enlarges the solution space, or keeps it from being unnecessarily narrow Do you have a link to a specific message? I went through the archives and couldn't find any e-mails in this thread that came close to describing a use case for anything, let alone anything that would be related to persistent bi-directional full-duplex communication with a remote server. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Lazy Blob
On Thu, Aug 2, 2012 at 11:01 AM, Ian Hickson i...@hixie.ch wrote: On Thu, 2 Aug 2012, Glenn Adams wrote: Sorry, I was vague. What I mean is what user-facing problem is it that we're trying to solve? see DAR's initial message in this thread; bringing WS into scope doesn't change the problem statement, it merely enlarges the solution space, or keeps it from being unnecessarily narrow Do you have a link to a specific message? I went through the archives and couldn't find any e-mails in this thread that came close to describing a use case for anything, let alone anything that would be related to persistent bi-directional full-duplex communication with a remote server. I was referring to [1]. [1] http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0264.html While that message does not specifically refer to a full-duplex comm path, it states the general problem in terms of It is increasingly common that data may flow from a server to an in-browser page, that may then pass that data on to another in-browser page (typically running at a different origin). In a many cases, such data will be captured as Blobs. It goes on to describe a solution space oriented towards XHR as the comm path. It occurred to me that the same problem could apply to WS comm path patterns, which is why I suggested enlarging the solution space to include WS.
Re: Lazy Blob
On Thu, 2 Aug 2012, Glenn Adams wrote: I was referring to http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0264.html While that message does not specifically refer to a full-duplex comm path, it states the general problem in terms of It is increasingly common that data may flow from a server to an in-browser page, that may then pass that data on to another in-browser page (typically running at a different origin). In a many cases, such data will be captured as Blobs. Ah, sorry, I probably still haven't explained the term use case properly. :-( The above isn't a use case, it's a description of an architectural design, the first step towards the description of a solution. What I'm trying to understand is the underlying _problem_ that the technology is trying to solve. Something like I want to be able to sell plane tickets for people to go on holiday, say. Or I want to provide a service to users that lets them merge data from a medical drugs database and a patient database, without giving me their credentials to those databases. Or some such. I don't know exactly what the use case here would be, hence my questions. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Lazy Blob
On Thu, Aug 2, 2012 at 11:09 AM, Florian Bösch pya...@gmail.com wrote: On Thu, Aug 2, 2012 at 6:37 PM, Glenn Adams gl...@skynav.com wrote: I am not proposing a particular browser supported semantic for a specific implementation on the server. I have suggested, by way of example, two particular patterns be supported independently of any such implementation. I did not restrict the results to just those patterns in case someone wishes to generalize. That is little different from the proposed or implied XHR patterns being discussed. So I'll take a stab, the remote blob resource/range protocol over WS 1.0: 1) A websocket to a URL is opened by the browser, the path and query of the URL is interpreted to specify a resource. 2) During the lifetime of a websocket session onto a wsblob resource, the resource is guaranteed to be reflected unchanged to the session, it cannot be changed, appended or removed. 3) The client has to send these bytes handshake as a first message 4) The server has to respond with a handshakelength message to indicate that he understands this protocol and indicate the byte length of the resource. 5) after successful setup the client may request ranges from the server by sending this message: rangestartend, start and end have to be in range of the byte resource. 6) The server will respond to each range request in the form of rangestartendbytes in case that a range request is valid, the length of bytes has to be start - end. In case a range is not valid the server will respond with invalidstartend. These are the protocol field definitions: handshake := wsblob length := unsigned int 4 bytes start := unsigned int 4 bytes end := unsigned int 4 bytes bytes := string of bytes range := 0x01 invalid := 0x02 ok, that is fine, but I never suggested limiting the semantics of interchange to a resource/range protocol; as is clear, the above application specific protocol does in fact use the multiple pattern I described, i.e., each interchange consists of a pair of send-response messages, each of which can be encapsulated in a blob, and each response blob could be implemented as a remotable 'promise' encapsulating a send blob and its resultant response blob;
Re: Lazy Blob
On Thu, Aug 2, 2012 at 11:19 AM, Ian Hickson i...@hixie.ch wrote: On Thu, 2 Aug 2012, Glenn Adams wrote: I was referring to http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0264.html While that message does not specifically refer to a full-duplex comm path, it states the general problem in terms of It is increasingly common that data may flow from a server to an in-browser page, that may then pass that data on to another in-browser page (typically running at a different origin). In a many cases, such data will be captured as Blobs. The above isn't a use case, it's a description of an architectural design, the first step towards the description of a solution. What I'm trying to understand is the underlying _problem_ that the technology is trying to solve. Something like I want to be able to sell plane tickets for people to go on holiday, say. Or I want to provide a service to users that lets them merge data from a medical drugs database and a patient database, without giving me their credentials to those databases. Or some such. I don't know exactly what the use case here would be, hence my questions. Are you asking for use cases for a remote/lazy blob in general? i.e., as would apply to the proposed XHR usage and any other underlying supported data source? or are you asking about high level use cases that are particular to a WS binding but not an XHR binding?
Re: Lazy Blob
On Thu, 2 Aug 2012, Glenn Adams wrote: Are you asking for use cases for a remote/lazy blob in general? i.e., as would apply to the proposed XHR usage and any other underlying supported data source? or are you asking about high level use cases that are particular to a WS binding but not an XHR binding? Both would be useful, but my primary concern is Web Sockets, since I edit that spec. Before I can consider proposals that affect Web Sockets, I need to know what use case it is we're trying to address. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Lazy Blob
On Thu, Aug 2, 2012 at 11:26 AM, Ian Hickson i...@hixie.ch wrote: On Thu, 2 Aug 2012, Glenn Adams wrote: Are you asking for use cases for a remote/lazy blob in general? i.e., as would apply to the proposed XHR usage and any other underlying supported data source? or are you asking about high level use cases that are particular to a WS binding but not an XHR binding? Both would be useful, but my primary concern is Web Sockets, since I edit that spec. Before I can consider proposals that affect Web Sockets, I need to know what use case it is we're trying to address. I will let Robin and Jungkee reply to the more general use case requirements. As far as WS is concerned, I don't see any impact of this thread on the WS API or WSP specs, its really simply an application of WS/WSP to remote/lazy blobs. Clearly, there are many high level use cases that involve a repetitive send/response message paradigm, which can certainly be implemented with XHR, but some application authors would prefer using WS for various efficiency reasons. My suggestion is essentially: if we are going to define a remote blob bound to an XHR source for a one-shot send-response, then perhaps we should define a remote blob bound to a WS source for multiple send-response pairs. For example, a symmetric presence protocol or IM protocol would typically fall into this usage category. Using remote blobs for either the send or response data (or both) would be useful for certain architectures and provide more deployment flexibility and perhaps greater efficiencies.
Re: Lazy Blob
I'd suggest the following. - Introduce an interface URLObject (bikeshedding can come later), with no methods. This object is supported by structured clone. - Add XMLHttpRequest.getURLObject(optional data), which returns a new URLObject. This can only be called while XMLHttpRequest is in the OPENED state. The returned URLObject represents the resource that would have been fetched if xhr.send() had been called. No XHR callbacks are performed, and no network activity takes place. The data argument acts like send(data), to specify the request entity body that will be sent. - Adjust URL.createObjectURL to take (Blob or URLObject), which returns a URL that works the same way as Blob URLs. Example: var xhr = new XMLHttpRequest(); xhr.open(GET, resource.jpg); var urlObject = xhr.getURLObject(); var newURL = URL.getObjectURL(urlObject); img.src = newURL; Some benefits: - We inherit XHR's ability to manipulate the request (eg. setRequestHeader), and all the security considerations that have gone into it. - The result is very much like Blob: you can transfer it around where you want it (workers or other pages), create object URLs, and then use those URLs with every other API (including Web Sockets, incidentally), since they're just like blob URLs. - It avoids a huge pitfall of getBlobFromURL, by not implementing a whole new mechanism for specifying a request. We already have XHR. - It avoids the pitfalls of returning a Blob: we don't make any network request at all at creation time to learn the size (avoiding spamming hundreds of HEAD requests), and it won't run into problems with servers with broken HEAD, transfer encodings, and so on. Any fetches performed as a result of this would be done under the origin of the page that created it, even if you transfer a URLObject somewhere else. This seems safe, since the caller can do nothing with it except trigger a fetch as-is, but it would need careful security review (as would any API like this). Passing one of these URLs back to XHR would need extra consideration (eg. should you be able to specify more headers?). (Note that I've spent some time thinking about this because I think it's technically interesting, but I haven't looked over the use cases closely enough to say whether I think it'd be worthwhile or not.) -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:50 PM, Glenn Adams gl...@skynav.com wrote: Further, a default behavior in the absence of such an injection might be defined simply to read data from the WS and stuff into the blob. Which kind of defeats the purpose because you wanted to read ranges, so not a whole resource has to be transferred, and you can already read binary data from websockets if you wish to do that, without having to invent another blob.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 6:51 PM, Glenn Adams gl...@skynav.com wrote: I'm questioning defining a LazyBlob that is solely usable with XHR. It would be better to have a more generic version IMO. Websockets have no content semantics, therefore any lazy content negotiating reader cannot deal with websockets unless an additional application layer protocol and implementation on the server side is introduced, something that does not exist for websockets otherwise. You could for instance implement HTTP over websockets to get the content semantics, and if your server gets a websocket request, it could be proxied to a domain socket which happend to have a webserver listening which would understand the HTTP request and deliver the resource/range. Now instead of implementing HTTP over websockets over HTTP over sockets, you could just use XHRs which implement HTTP over sockets. Which is why generalising lazy readers to websockets does not make sense.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 6:40 PM, Glenn Adams gl...@skynav.com wrote: Why restrict to XHR? How about WebSocket as data source? Websockets support array buffers and therefore by extension any blob/file object. However as a stream oriented API websockets have no content aquisition, negotation, range and transfer semantics unless you prop those up by yourself as an application layer protocol.
Re: Lazy Blob
On Thu, Aug 2, 2012 at 5:45 PM, Glenn Adams gl...@skynav.com wrote: No it hasn't. If you want a real world use case it is this: my architectural constraints as an author for some particular usage requires that I use WS rather than XHR. I wish to have support for the construct being discussed with WS. How is that not a real world requirement? Your particular use-case of content/range aquisition over WS requires a particular implementation on the server in order to understand the WS application layer protocol. This particular implementation on the server of yours is not implemented by any other common hosting infrastructure based on any kind of existing standard. You should specify this particular protocol standard to be used on top of WS first before you can even discuss how your custom implementation of this protocol justifies enshrining it in a browser standard.
Re: Lazy Blob
On Thu, Aug 2, 2012 at 6:37 PM, Glenn Adams gl...@skynav.com wrote: I am not proposing a particular browser supported semantic for a specific implementation on the server. I have suggested, by way of example, two particular patterns be supported independently of any such implementation. I did not restrict the results to just those patterns in case someone wishes to generalize. That is little different from the proposed or implied XHR patterns being discussed. So I'll take a stab, the remote blob resource/range protocol over WS 1.0: 1) A websocket to a URL is opened by the browser, the path and query of the URL is interpreted to specify a resource. 2) During the lifetime of a websocket session onto a wsblob resource, the resource is guaranteed to be reflected unchanged to the session, it cannot be changed, appended or removed. 3) The client has to send these bytes handshake as a first message 4) The server has to respond with a handshakelength message to indicate that he understands this protocol and indicate the byte length of the resource. 5) after successful setup the client may request ranges from the server by sending this message: rangestartend, start and end have to be in range of the byte resource. 6) The server will respond to each range request in the form of rangestartendbytes in case that a range request is valid, the length of bytes has to be start - end. In case a range is not valid the server will respond with invalidstartend. These are the protocol field definitions: handshake := wsblob length := unsigned int 4 bytes start := unsigned int 4 bytes end := unsigned int 4 bytes bytes := string of bytes range := 0x01 invalid := 0x02
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? Reading a resource at arbitrary locations requires two things: 1) That a resource is understood as a container of bytes having a start and end 2) That the protocol used to access it supports specifying a range into that resource The HTTP protocol satisfies these two criteria (disregarding things like chunked transfer etc. for now): 1) a HTTP GET is understood as an access to a resource having a start and end, the URL maps to the resource location or handler 2) HTTP ranges let you specify which part of that resource you'd like to get The WebSoicket protocol does not satisfy those conditions 1) A resource is understood as a realtime duplex channel of communication between two communication partners, the URL indicates which channel you're using. 2) Websockets do not understand or handle HTTP content ranges, there is no part of the websocket protocol that would dictate accessing a byte-container at the other at specified offsets into a resource. So if you want to have a lazy reader on Websockets you have either: 1) respecify the websocket protocol to include content semantics for accessing resources defined by an URL and having a specified size OR 2) define an additional protocol on top of websockets, which websockets know nothing about, that allows a custom implementation at the server side to respond in a meaningful fashion to resource range requests. Both these things are completely outside of the scope of generalizing lazy blobs.
Re: Lazy Blob
On Thu, Aug 2, 2012 at 5:58 PM, Glenn Adams gl...@skynav.com wrote: All WS usage requires a particular (application specific) implementation on the server, does it not? Notwithstanding that fact, such usage will fall into certain messaging patterns. I happened to give an example of two possible message patterns and showed how the proposal under discussion could address those patterns. It is not necessary to marry my proposal to a specific sub-protocol on WS in order to provide useful functionality that can be exploited by applications that use those functions. If you wish to introduce a particular browser supported semantic for which a specific implementation on the server is required, then people should be able to consult a standard that tells them how they have to provide this implementation. Therefore it is quite necessary to marry your desire to extend remote blobs to WS to a protocol, otherwise you'll have a browser implemented protocol that nobody knows how to implement.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 10:13 PM, Glenn Adams gl...@skynav.com wrote: The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later. Then you better get onto specifying a resource/range transfer protocol on top of websockets alongside with web-server modules/extensions to be able to understand that protocol, because other than that there is no way that you'll get what you want.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 7:57 PM, Glenn Adams gl...@skynav.com wrote: blob = bb.getBlobFromURL(ws://specifiction.com/image/kitten.pnghttp://specifiction.com/kitten.png ) There is no application layer transfer protocol inherent in websockets. Requesting a resource does not have any inherent meaning other than that you are opening a channel onto /image/kitten.png. Whoever receives that request is free to respond to that however he likes. You would need to introduce an application layer content protocol on top of websockets, and introduce a default websocket server framework capable of understanding such content requests. You're not just extending lazy reading to websockets. You're putting the burden on yourself to also specify a completely new standard application layer protocol for transfer and range and acquisition of resources over websocket channels.
Re: Lazy Blob
The URLObject proposal is a pretty slick way of cooking up a request in contextA for later (and all manner of) retrieval in contextB. On Thu, Aug 2, 2012 at 4:23 PM, Glenn Maynard gl...@zewt.org wrote: I'd suggest the following. - Introduce an interface URLObject (bikeshedding can come later), with no methods. This object is supported by structured clone. - Add XMLHttpRequest.getURLObject(optional data), which returns a new URLObject. This can only be called while XMLHttpRequest is in the OPENED state. The returned URLObject represents the resource that would have been fetched if xhr.send() had been called. No XHR callbacks are performed, and no network activity takes place. The data argument acts like send(data), to specify the request entity body that will be sent. - Adjust URL.createObjectURL to take (Blob or URLObject), which returns a URL that works the same way as Blob URLs. Example: var xhr = new XMLHttpRequest(); xhr.open(GET, resource.jpg); var urlObject = xhr.getURLObject(); var newURL = URL.getObjectURL(urlObject); img.src = newURL; Some benefits: - We inherit XHR's ability to manipulate the request (eg. setRequestHeader), and all the security considerations that have gone into it. - The result is very much like Blob: you can transfer it around where you want it (workers or other pages), create object URLs, and then use those URLs with every other API (including Web Sockets, incidentally), since they're just like blob URLs. - It avoids a huge pitfall of getBlobFromURL, by not implementing a whole new mechanism for specifying a request. We already have XHR. - It avoids the pitfalls of returning a Blob: we don't make any network request at all at creation time to learn the size (avoiding spamming hundreds of HEAD requests), and it won't run into problems with servers with broken HEAD, transfer encodings, and so on. Any fetches performed as a result of this would be done under the origin of the page that created it, even if you transfer a URLObject somewhere else. This seems safe, since the caller can do nothing with it except trigger a fetch as-is, but it would need careful security review (as would any API like this). Passing one of these URLs back to XHR would need extra consideration (eg. should you be able to specify more headers?). (Note that I've spent some time thinking about this because I think it's technically interesting, but I haven't looked over the use cases closely enough to say whether I think it'd be worthwhile or not.) -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:59 AM, Robin Berjon ro...@berjon.com wrote:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET, {
Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
We already have an API to allow scripts to make network requests: XHR.
Please don't create a new API that will end up duplicating all of that.
However this might be done, it should hang off of XHR.
Ideally, a new responseType could be added to XHR which operates like
blob, except instead of reading the whole resource, it just performs a
HEAD to retrieve the response length and immediately returns the Blob,
which can be read to perform further Content-Range reads. This sits
relatively cleanly within the XHR API. It also has nice security
properties, as if you send that Blob somewhere else--possibly to a
different origin--it can do nothing with the Blob but read it as it was
given.
The trouble is chunked responses, where the length of the resource isn't
known in advance, since Blobs have a static length and aren't designed for
streaming.
--
Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:44 AM, Glenn Maynard gl...@zewt.org wrote:
On Wed, Aug 1, 2012 at 9:59 AM, Robin Berjon ro...@berjon.com wrote:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET,
{ Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
We already have an API to allow scripts to make network requests: XHR.
Please don't create a new API that will end up duplicating all of that.
However this might be done, it should hang off of XHR.
Why restrict to XHR? How about WebSocket as data source?
Re: Lazy Blob
On Wed, Aug 1, 2012 at 10:46 AM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 6:40 PM, Glenn Adams gl...@skynav.com wrote: Why restrict to XHR? How about WebSocket as data source? Websockets support array buffers and therefore by extension any blob/file object. However as a stream oriented API websockets have no content aquisition, negotation, range and transfer semantics unless you prop those up by yourself as an application layer protocol. I'm questioning defining a LazyBlob that is solely usable with XHR. It would be better to have a more generic version IMO.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 11:13 AM, Florian Bösch pya...@gmail.com wrote:
On Wed, Aug 1, 2012 at 6:51 PM, Glenn Adams gl...@skynav.com wrote:
I'm questioning defining a LazyBlob that is solely usable with XHR. It
would be better to have a more generic version IMO.
Websockets have no content semantics, therefore any lazy content
negotiating reader cannot deal with websockets unless an additional
application layer protocol and implementation on the server side is
introduced, something that does not exist for websockets otherwise. You
could for instance implement HTTP over websockets to get the content
semantics, and if your server gets a websocket request, it could be proxied
to a domain socket which happend to have a webserver listening which would
understand the HTTP request and deliver the resource/range.
Now instead of implementing HTTP over websockets over HTTP over sockets,
you could just use XHRs which implement HTTP over sockets. Which is why
generalising lazy readers to websockets does not make sense.
Given the Simple approach suggested by DAR:
partial interface BlobBuilder {
Blob getBlobFromURL (DOMString url);
};
Usage:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;);
I don't see why the following isn't feasible:
blob =
bb.getBlobFromURL(ws://specifiction.com/image/kitten.pnghttp://specifiction.com/kitten.png
)
Or, given the Using XHR for Options approach:
partial interface BlobBuilder {
Blob getBlobFromURL (XMLHttpRequest xhr);
};
Usage:
var bb = new BlobBuilder()
, xhr = new XMLHttpRequest();
xhr.open(GET, /kitten.png, true);
xhr.setRequestHeader(Authorization, Basic DEADBEEF);
var blob = bb.getBlobFromURL(xhr);
why one couldn't have:
partial interface BlobBuilder {
Blob getBlobFromURL (WebSocket ws);
};
var bb = new BlobBuilder()
, ws = new
WebSocket(ws://specifiction.com/imagehttp://specifiction.com/kitten.png
);
ws.onopen = function(){ws.send(kitten.png);}
var blob = bb.getBlobFromURL(ws);
Re: Lazy Blob
On Aug 1, 2012, at 8:44 AM, Glenn Maynard gl...@zewt.org wrote:
On Wed, Aug 1, 2012 at 9:59 AM, Robin Berjon ro...@berjon.com wrote:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET, {
Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
We already have an API to allow scripts to make network requests: XHR.
Please don't create a new API that will end up duplicating all of that.
However this might be done, it should hang off of XHR.
Ideally, a new responseType could be added to XHR which operates like blob,
except instead of reading the whole resource, it just performs a HEAD to
retrieve the response length and immediately returns the Blob, which can be
read to perform further Content-Range reads. This sits relatively cleanly
within the XHR API. It also has nice security properties, as if you send
that Blob somewhere else--possibly to a different origin--it can do nothing
with the Blob but read it as it was given.
The trouble is chunked responses, where the length of the resource isn't
known in advance, since Blobs have a static length and aren't designed for
streaming.
The HEAD request seems a little like a FileEntry for use with FileReader.
Something similar (very similar) is done for http+fuse on several platforms.
It's quite useful for working with large archives such as PDF or ZIP.
-Charles
Re: Lazy Blob
On Wed, Aug 1, 2012 at 12:03 PM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 7:57 PM, Glenn Adams gl...@skynav.com wrote: blob = bb.getBlobFromURL(ws://specifiction.com/image/kitten.pnghttp://specifiction.com/kitten.png ) There is no application layer transfer protocol inherent in websockets. Requesting a resource does not have any inherent meaning other than that you are opening a channel onto /image/kitten.png. Whoever receives that request is free to respond to that however he likes. You would need to introduce an application layer content protocol on top of websockets, and introduce a default websocket server framework capable of understanding such content requests. You're not just extending lazy reading to websockets. You're putting the burden on yourself to also specify a completely new standard application layer protocol for transfer and range and acquisition of resources over websocket channels. So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP?
Re: Lazy Blob
Maybe another XHR based way to phrase this: define a new response type for
XHR that results in the construction of a lazyBlob. I guess that's similar
to the more explicit xhr.makeLazyBlob() method, but allows its construction
to be async, and for greater reuse of the same signaling for progress and
errors, and to capture the results of a POST as a lazyBlob as well.
Knowing the length and to some degree the content-type upfront is trouble.
An xhr.responseType of 'stream' would have some trouble with the
content-type too.
On Wed, Aug 1, 2012 at 10:57 AM, Charles Pritchard ch...@jumis.com wrote:
On Aug 1, 2012, at 8:44 AM, Glenn Maynard gl...@zewt.org wrote:
On Wed, Aug 1, 2012 at 9:59 AM, Robin Berjon ro...@berjon.com wrote:
var bb = new BlobBuilder()
, blob = bb.getBlobFromURL(http://specifiction.com/kitten.png;, GET,
{ Authorization: Basic DEADBEEF });
Everything is the same as the previous version but the method and some
headers can be set by enumerating the Object. I *think* that those are all
that would ever be needed.
We already have an API to allow scripts to make network requests: XHR.
Please don't create a new API that will end up duplicating all of that.
However this might be done, it should hang off of XHR.
Ideally, a new responseType could be added to XHR which operates like
blob, except instead of reading the whole resource, it just performs a
HEAD to retrieve the response length and immediately returns the Blob,
which can be read to perform further Content-Range reads. This sits
relatively cleanly within the XHR API. It also has nice security
properties, as if you send that Blob somewhere else--possibly to a
different origin--it can do nothing with the Blob but read it as it was
given.
The trouble is chunked responses, where the length of the resource isn't
known in advance, since Blobs have a static length and aren't designed for
streaming.
The HEAD request seems a little like a FileEntry for use with FileReader.
Something similar (very similar) is done for http+fuse on several
platforms.
It's quite useful for working with large archives such as PDF or ZIP.
-Charles
Re: Lazy Blob
On Wed, Aug 1, 2012 at 1:36 PM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 9:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? So if you want to have a lazy reader on Websockets you have either: 1) respecify the websocket protocol to include content semantics for accessing resources defined by an URL and having a specified size OR 2) define an additional protocol on top of websockets, which websockets know nothing about, that allows a custom implementation at the server side to respond in a meaningful fashion to resource range requests. OR define a mechanism for LazyBlob that permits the injection of app specific code into the underlying LazyBlob reader loop.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 1:47 PM, Glenn Adams gl...@skynav.com wrote: On Wed, Aug 1, 2012 at 1:36 PM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 9:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? So if you want to have a lazy reader on Websockets you have either: 1) respecify the websocket protocol to include content semantics for accessing resources defined by an URL and having a specified size OR 2) define an additional protocol on top of websockets, which websockets know nothing about, that allows a custom implementation at the server side to respond in a meaningful fashion to resource range requests. OR define a mechanism for LazyBlob that permits the injection of app specific code into the underlying LazyBlob reader loop. Further, a default behavior in the absence of such an injection might be defined simply to read data from the WS and stuff into the blob.
Re: Lazy Blob
Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. On Wed, Aug 1, 2012 at 2:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? XHR is no more specific to HTTP than it is to XML. It serves as the primary JavaScript API for performing generic network fetches. WebSockets has an entirely different API from blobs, and bringing them up is only derailing the thread. On Wed, Aug 1, 2012 at 2:32 PM, Michael Nordman micha...@google.com wrote: Maybe another XHR based way to phrase this: define a new response type for XHR that results in the construction of a lazyBlob. I guess that's similar to the more explicit xhr.makeLazyBlob() method, but allows its construction to be async, and for greater reuse of the same signaling for progress and errors, and to capture the results of a POST as a lazyBlob as well. Knowing the length and to some degree the content-type upfront is trouble. I like the idea of being able to retrieve a Blob for a URL via XHR, but I'm uneasy about exposing the transfer-encoding (eg. whether Content-Length is known in advance or not). It's very confusing to move your code between servers and to have things mysteriously break because the server uses a different Transfer-Encoding. I think web APIs should avoid exposing this at all. A middle-ground, if not an ideal one, would be for the Blob to return after the HEAD request if Content-Length is present. However, if it's absent, perform the complete fetch before returning the Blob. That would at least abstract away the HTTP specifics, so code wouldn't break. That said, this approach (with or without greedy downloading for chunked) may not quite fit some of the OP's use cases. In particular, if you have 1000 URLs and you want to hand Blobs off to another page in this way, then while this wouldn't retrieve them all, it would still result in 1000 HEAD requests. Sending XHR itself between contexts seems like it would be prickly from a security standpoint. A possible solution would be a higher-level wrapper, eg. a thin abstraction of XHR that can only send a request and receive the result (eg. exposing a very small subset of XMLHttpRequest's interface). That's much more heavyweight than the Blob approach, though, and I don't have an opinion on whether the use cases justify it. -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 1:54 PM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 9:50 PM, Glenn Adams gl...@skynav.com wrote: Further, a default behavior in the absence of such an injection might be defined simply to read data from the WS and stuff into the blob. Which kind of defeats the purpose because you wanted to read ranges, so not a whole resource has to be transferred, and you can already read binary data from websockets if you wish to do that, without having to invent another blob. A default behavior does not have to handle all uses cases. App specific code injection could handle this if the author wished it, provided a mechanism supported it.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 2:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. On Wed, Aug 1, 2012 at 2:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? XHR is no more specific to HTTP than it is to XML. It serves as the primary JavaScript API for performing generic network fetches. WebSockets has an entirely different API from blobs, and bringing them up is only derailing the thread. The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later.
Re: Lazy Blob
On 8/1/12 11:44 AM, Glenn Maynard wrote: Ideally, a new responseType could be added to XHR which operates like blob, except instead of reading the whole resource, it just performs a HEAD to retrieve the response length and immediately returns the Blob, which can be read to perform further Content-Range reads. Unless the server is one of the common ones with broken HEAD handling. Or unless the resource is served with Content-Encoding:gzip, in which case your Content-Range is all sorta broken. :( The trouble is chunked responses, where the length of the resource isn't known in advance, since Blobs have a static length and aren't designed for streaming. That's another trouble, yes. -Boris
Re: Lazy Blob
On Wed, Aug 1, 2012 at 8:16 PM, Boris Zbarsky bzbar...@mit.edu wrote: Unless the server is one of the common ones with broken HEAD handling. Or unless the resource is served with Content-Encoding:gzip, in which case your Content-Range is all sorta broken. :( Those could fall back on fetching the whole resource, too, so these issues would still be transparent to scripts (if the broken HEAD cases are detectable, anyway). gzip/deflated content could still work (without fetching the entire resource) in the common case of reading a Blob sequentially. It'd only need to fall back on retrieving the entire resource if you're doing nonsequential reads. It's not ideal, but it doesn't seem like such a big problem that it'd make the feature not worth it. It still doesn't help the creating lots of resources without making lots of network requests use case, though. -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 2:35 PM, Florian Bösch pya...@gmail.com wrote: On Wed, Aug 1, 2012 at 10:13 PM, Glenn Adams gl...@skynav.com wrote: The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later. Then you better get onto specifying a resource/range transfer protocol on top of websockets alongside with web-server modules/extensions to be able to understand that protocol, because other than that there is no way that you'll get what you want. I don't think so. There is nothing about Blob that would require a data source to implement range access. Blob.slice() does not require the underlying source to provide range access. The source could be read in entirety and buffered by a Blob instance. If a reasonable WS enabled mechanism were defined for a Lazy Blob that permitted an application injected range access, then that could be used to perform actual range access. There is no need for WS/WSP to support those semantics directly. I don't particularly care if a default behavior for WS is provided that buffers the entire read stream. It's fine to mandate that an application defined function implement those semantics on a WS instance. My concern is that use of WS be recognized as a legitimate source for filling a lazy blob, and that an author should have an option to use WS, depending on app injected code as needed, instead of mandating XHR for this purpose. I'll leave the details of defining this to the proposers of lazy blob.
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:54 PM, Glenn Adams gl...@skynav.com wrote: I don't particularly care if a default behavior for WS is provided that buffers the entire read stream. Sorry, but that doesn't make sense. You don't access a message-based protocol (Web Sockets) using a character-based API (Blob). They're utterly different APIs. I'll leave the details of defining this to the proposers of lazy blob. You're free to come up with your own proposal, of course, and editors and vendors will choose among them (or come up with something else, or reject the idea entirely) as they always do, but others are not obligated to twist their proposals to your demands. -- Glenn Maynard
Re: Lazy Blob
On Wed, Aug 1, 2012 at 9:35 PM, Glenn Maynard gl...@zewt.org wrote: On Wed, Aug 1, 2012 at 9:54 PM, Glenn Adams gl...@skynav.com wrote: I don't particularly care if a default behavior for WS is provided that buffers the entire read stream. Sorry, but that doesn't make sense. You don't access a message-based protocol (Web Sockets) using a character-based API (Blob). They're utterly different APIs. Have you read the Blob interface spec? To quote: This interface represents *immutable* raw data. It provides a method to slice http://dev.w3.org/2006/webapi/FileAPI/#dfn-slice data objects between ranges of bytes into further chunks of raw data. The last time I checked, bytes are bytes, not characters. The fact that the interface provides access to those bytes via a particular string encoding is irrelevant. I'll leave the details of defining this to the proposers of lazy blob. You're free to come up with your own proposal, of course, and editors and vendors will choose among them (or come up with something else, or reject the idea entirely) as they always do, but others are not obligated to twist their proposals to your demands. Of course, implementers are free to ignore whatever they want, but last time I checked, the W3C was a consensus based standards organization which means agreement needs to be reached on what specs go out the door and what are in those specs. Since this is a W3C ML and not an implementers' forum, then I will continue to assume that the W3C process applies. There is a fixed obligation for editors and WG to address comments. They can't simply be rejected because they require work on the part of the editors or proposers.
Re: Lazy Blob
On 8/1/12 9:50 PM, Glenn Maynard wrote: (if the broken HEAD cases are detectable, anyway) I wish. Some of them are, but a lot are not. We used to use HEAD for things like save link as in Firefox (for prompting the user for a filename and such), but had to stop because it was broken so often in totally undetectable ways. :( -Boris
Re: Lazy Blob
On Wed, Aug 1, 2012 at 2:13 PM, Glenn Adams gl...@skynav.com wrote: On Wed, Aug 1, 2012 at 2:04 PM, Glenn Maynard gl...@zewt.org wrote: Can we please stop saying lazy blob? It's a confused and confusing phrase. Blobs are lazy by design. On Wed, Aug 1, 2012 at 2:26 PM, Glenn Adams gl...@skynav.com wrote: So? Why should lazy blob be specific to HTTP specific semantics when an arbitrary URL is not specific to HTTP? XHR is no more specific to HTTP than it is to XML. It serves as the primary JavaScript API for performing generic network fetches. WebSockets has an entirely different API from blobs, and bringing them up is only derailing the thread. The subject line says Lazy Blob, not Lazy Blob and XHR. For the record, I will object to a LazyBlob solution that is tied solely to XHR, so deal with it now rather than later. Just to make it clear, I support the idea of defining a lazy blob mechanism. However, I am not satisfied that a solution that is tied solely to XHR is sufficient. I would like to see a mechanism that supports both XHR and WS [and others?]. Despite the repeated claims of Florian and GlennM that it doesn't make sense, etc., I think it does make sense and can be reasonably (and simply) defined to handle such use cases. If necessary I can volunteer a strawman to that end. However, I would prefer that DAR or other proposers take the time to consider this use case and factor it into their proposals.
Re: Lazy Blob
On Wed, 1 Aug 2012, Glenn Adams wrote: Of course, implementers are free to ignore whatever they want, but last time I checked, the W3C was a consensus based standards organization which means agreement needs to be reached on what specs go out the door and what are in those specs. Doesn't really matter what's in the specs going out the door if it's not what's implemented... I don't really care about the XHR side of this (happy to let Anne figure that out), but since WebSockets was mentioned: what's the use case that involves Web Socket? I don't really understand what problem we're trying to solve here. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
