Re: File API: reading a Blob
+Aaron On Thu, Sep 11, 2014 at 6:48 PM, Aymeric Vitte vitteayme...@gmail.com wrote: But I suppose that should be one of the first use case for Google to introduce streams with MSE, no? We're (and will) sharing updates about Streams with MSE folk (Aaron). MSE side would be updated once Streams API becomes stable enough for them to start revising. To be more clear about what I mean by back pressure for things coming from outside of the browser: - XHR: the Streams API should define how xhr gets chunks using Range according to the flow and adapt accordingly transparently for the users We're not thinking of involving multiple HTTP transactions for one send(). That's beyond current XHR semantics. I think even if we implement that kind of feature, we should add it as a new mode or API, not as a new responseType. What we plan to do is connecting the Streams API's back pressure with TCP congestion control. Since socket API doesn't expose much details to user, we'll just control frequency and the size of buffer to pass to read(2). - WebSockets: use something like bufferedAmount but that can be notified to the users, so the users can adapt the flow, currently bufferedAmount is not extremely usefull since you might need to do some polling to check it. For WebSockets, there're two approaches of Streams integration. ReadableStream of messages or ReadableStream of ArrayBuffer representing chunks of the payload of a message. We need to discuss that first. Le 11/09/2014 08:36, Takeshi Yoshino a écrit : On Thu, Sep 11, 2014 at 8:47 AM, Aymeric Vitte vitteayme...@gmail.com wrote: Does your experimentation pipe the XHR stream to MSE? Obviously that should be the target for yt, this would be a first real application of the Streams API. It's not yet updated to use the new Streams. Here's our layout test for MSE. responseType = 'legacystream' makes the XHR return the old version of the stream. https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-append-legacystream.htmlq=createMediaXHRsq=package:chromiumtype=csl=12 You can find the following call in the file. sourceBuffer.appendStream(xhr.response); Because the Streams API does not define how to apply back pressure to XHR, but does define how to apply back pressure between XHR and MSE. Probably the spec should handle on a per case basis what should be the behavior in term of back pressure for things coming from outside of the browser (xhr, websockets, webrtc, etc , not specified as far as I know) and for things going on inside the browser (already specified) Le 08/09/2014 06:58, Takeshi Yoshino a écrit : On Thu, Sep 4, 2014 at 7:02 AM, Aymeric Vitte vitteayme...@gmail.com wrote: The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. Chrome has an experimental implementation [1] of the new Streams API [2] integrated with XHR [3] behind a flag. We receive data from the browser process over IPC (both network and blob case). The size of data chunks and arrival timing depend on various factors. The received chunks are passed to the XMLHttpRequest class on the same thread as JavaScript runs. We create a new instance of ReadableStream [4] on arrival of the first chunk. On every chunk arrival, we create an ArrayBuffer from the chunk and then call [[enqueue]](chunk) [5] equivalent C++ function to put it into the ReadableStream. The ReadableStream is available from the response attribute in the LOADING and DONE state (if no error). The chunks pushed to the ReadableStream become available for read immediately. Any problem occurs while loading data from network/blob, we call [[error]](e) [6] equivalent C++ function with an exception as defined in the XHR spec for sync XHR. Currently, XMLHttpRequest doesn't exert any back pressure. We plan to do something not to read too much data from disk/network. It might be worth specifying something about the flow control in the abstract read from blob/network operation at standard level. [1] https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/xmlhttprequest/response-stream.html [2] https://github.com/whatwg/streams
Re: File API: reading a Blob
On Thu, Sep 11, 2014 at 8:47 AM, Aymeric Vitte vitteayme...@gmail.com wrote: Does your experimentation pipe the XHR stream to MSE? Obviously that should be the target for yt, this would be a first real application of the Streams API. It's not yet updated to use the new Streams. Here's our layout test for MSE. responseType = 'legacystream' makes the XHR return the old version of the stream. https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-append-legacystream.htmlq=createMediaXHRsq=package:chromiumtype=csl=12 You can find the following call in the file. sourceBuffer.appendStream(xhr.response); Because the Streams API does not define how to apply back pressure to XHR, but does define how to apply back pressure between XHR and MSE. Probably the spec should handle on a per case basis what should be the behavior in term of back pressure for things coming from outside of the browser (xhr, websockets, webrtc, etc , not specified as far as I know) and for things going on inside the browser (already specified) Le 08/09/2014 06:58, Takeshi Yoshino a écrit : On Thu, Sep 4, 2014 at 7:02 AM, Aymeric Vitte vitteayme...@gmail.com wrote: The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. Chrome has an experimental implementation [1] of the new Streams API [2] integrated with XHR [3] behind a flag. We receive data from the browser process over IPC (both network and blob case). The size of data chunks and arrival timing depend on various factors. The received chunks are passed to the XMLHttpRequest class on the same thread as JavaScript runs. We create a new instance of ReadableStream [4] on arrival of the first chunk. On every chunk arrival, we create an ArrayBuffer from the chunk and then call [[enqueue]](chunk) [5] equivalent C++ function to put it into the ReadableStream. The ReadableStream is available from the response attribute in the LOADING and DONE state (if no error). The chunks pushed to the ReadableStream become available for read immediately. Any problem occurs while loading data from network/blob, we call [[error]](e) [6] equivalent C++ function with an exception as defined in the XHR spec for sync XHR. Currently, XMLHttpRequest doesn't exert any back pressure. We plan to do something not to read too much data from disk/network. It might be worth specifying something about the flow control in the abstract read from blob/network operation at standard level. [1] https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/xmlhttprequest/response-stream.html [2] https://github.com/whatwg/streams https://github.com/whatwg/streams#readablestream [3] https://github.com/tyoshino/streams_integration/ [4] https://github.com/whatwg/streams#readablestream [5] https://github.com/whatwg/streams#enqueuechunk [6] https://github.com/whatwg/streams#errore -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
But I suppose that should be one of the first use case for Google to introduce streams with MSE, no? To be more clear about what I mean by back pressure for things coming from outside of the browser: - XHR: the Streams API should define how xhr gets chunks using Range according to the flow and adapt accordingly transparently for the users - WebSockets: use something like bufferedAmount but that can be notified to the users, so the users can adapt the flow, currently bufferedAmount is not extremely usefull since you might need to do some polling to check it. Le 11/09/2014 08:36, Takeshi Yoshino a écrit : On Thu, Sep 11, 2014 at 8:47 AM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: Does your experimentation pipe the XHR stream to MSE? Obviously that should be the target for yt, this would be a first real application of the Streams API. It's not yet updated to use the new Streams. Here's our layout test for MSE. responseType = 'legacystream' makes the XHR return the old version of the stream. https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-append-legacystream.htmlq=createMediaXHRsq=package:chromiumtype=csl=12 You can find the following call in the file. sourceBuffer.appendStream(xhr.response); Because the Streams API does not define how to apply back pressure to XHR, but does define how to apply back pressure between XHR and MSE. Probably the spec should handle on a per case basis what should be the behavior in term of back pressure for things coming from outside of the browser (xhr, websockets, webrtc, etc , not specified as far as I know) and for things going on inside the browser (already specified) Le 08/09/2014 06:58, Takeshi Yoshino a écrit : On Thu, Sep 4, 2014 at 7:02 AM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. Chrome has an experimental implementation [1] of the new Streams API [2] integrated with XHR [3] behind a flag. We receive data from the browser process over IPC (both network and blob case). The size of data chunks and arrival timing depend on various factors. The received chunks are passed to the XMLHttpRequest class on the same thread as JavaScript runs. We create a new instance of ReadableStream [4] on arrival of the first chunk. On every chunk arrival, we create an ArrayBuffer from the chunk and then call [[enqueue]](chunk) [5] equivalent C++ function to put it into the ReadableStream. The ReadableStream is available from the response attribute in the LOADING and DONE state (if no error). The chunks pushed to the ReadableStream become available for read immediately. Any problem occurs while loading data from network/blob, we call [[error]](e) [6] equivalent C++ function with an exception as defined in the XHR spec for sync XHR. Currently, XMLHttpRequest doesn't exert any back pressure. We plan to do something not to read too much data from disk/network. It might be worth specifying something about the flow control in the abstract read from blob/network operation at standard level. [1] https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/xmlhttprequest/response-stream.html [2] https://github.com/whatwg/streams https://github.com/whatwg/streams#readablestream [3] https://github.com/tyoshino/streams_integration/ [4] https://github.com/whatwg/streams#readablestream [5] https://github.com/whatwg/streams#enqueuechunk [6] https://github.com/whatwg/streams#errore -- Peersm :http://www.peersm.com torrent-live:https://github.com/Ayms/torrent-live node-Tor :https://www.github.com/Ayms/node-Tor GitHub :https://www.github.com/Ayms -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
Arun, I know you (File API) care about it, I was more refering to other groups that seem not to care a lot, leading to absurd situations where we are streaming things without streams and have to implement some strange inadapted mechanisms for flow control/backpressure for example. The examples I gave in this thread are just a small subset of what I (everybody) need, which involves all the groups listed below. Despite of the fact that the spec is probably mature enough, I am not sure it can really be finalized without parallel field experimentation, which to work well needs to involve several groups and browsers from now, despite of the efforts of people involved the process is really too long. If we take the File API to address your concern, probably the question is not whether the earlier (or whatever) version should be modified (because the answer would be obviously yes for me, use cases are legion) but to make it work on the field with streams and finalize the spec accordingly, same thing for the other APIs. Regards, Aymeric Le 04/09/2014 02:39, Arun Ranganathan a écrit : On Sep 3, 2014, at 6:02 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I care about it. Till the API is totally baked, I’m amenable to getting the model right. File API now refers to chunks read, which is more correct. But I understand that your use cases aren’t catered to just yet; FileReader/FileReaderSync don’t do easily extractable partials. I’d like to see if there’s interest in the earlier proposal, to extract a stream straight from Blob. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. What issue is being eluded? Seems like another of your main use cases is to have URL.createObjectURL or URL.createFor return a streamable resource. I agree that’s a good problem to solve. — A* -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
On Thu, Sep 4, 2014 at 12:02 AM, Aymeric Vitte vitteayme...@gmail.com wrote: Sorry to interfer then but your discussion with Arun seems to have no point if streams are there. I don't follow. The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. Well, we need to sort out the fundamental abstraction first and that is taking its time. Only then can we integrate it throughout the system. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. Well, we are looking into it from an API perspective. Perhaps not all, but we have had quite a bit of discussion (elsewhere, not here) how streams should work with the new fetch() method. But again, I think as with promises, we need the primitive first. -- http://annevankesteren.nl/
Re: File API: reading a Blob
Sorry to interfer then but your discussion with Arun seems to have no point if streams are there. But some points I mentioned have, whether it's really linked to the initial subject or not. The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. Le 01/09/2014 11:14, Anne van Kesteren a écrit : On Thu, Aug 28, 2014 at 2:15 AM, Aymeric Vitte vitteayme...@gmail.com wrote: I have contributed to streams already, of course it will solve most of what is being discussed here but when? And then what's the point of this discussion? This discussion between me and Arun is about how to describe various algorithms in File API and Fetch, ideally enabling APIs on top of those low-level descriptions later on. But it is about describing the behavior of existing APIs in more detail. -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
On Aug 11, 2014, at 7:24 AM, Anne van Kesteren ann...@annevk.nl wrote: Other than “chunks of bytes” which needs some normative backbone, is the basic abstract model what you had in mind? If so, that might be worth getting into File APi and calling it done, because that’s a reusable abstract model for Fetch and for FileSystem. Yeah that looks good. https://whatwg.github.io/streams/ defines chunks and such, but is not quite there yet. But it is what we want to build this upon eventually. I’ve fixed the model according to these discussions: 1. We’ve separated the read operation and the “annotated task read operation.” The former uses Fetch’s notion of a body, which is a byte stream, and so is convenient to work with. Hopefully, it should allow plugging right back into Fetch. 2. The methods now invoke an annotated task read operation. http://dev.w3.org/2006/webapi/FileAPI/#reading-data-section — A*
Re: File API: reading a Blob
On Sep 3, 2014, at 6:02 PM, Aymeric Vitte vitteayme...@gmail.com wrote: The fact is that most of the W3C groups impacted by streams (File, indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream, etc, I must forget a lot here) seem not to care a lot about it and maybe just expect streams to land in the right place in the APIs when they are available, by some unknown magic. I care about it. Till the API is totally baked, I’m amenable to getting the model right. File API now refers to chunks read, which is more correct. But I understand that your use cases aren’t catered to just yet; FileReader/FileReaderSync don’t do easily extractable partials. I’d like to see if there’s interest in the earlier proposal, to extract a stream straight from Blob. I still think that the effort should start from now for all the APIs (as well as the implementation inside browsers, which apparently has started for Chrome, but Chrome was supposed to have started some implementation of the previous Streams APIs, so it's not very clear), and that it should be very clearly synchronized, disregarding vague assumptions from the groups about low/high level and Vx releases, eluding the issue. What issue is being eluded? Seems like another of your main use cases is to have URL.createObjectURL or URL.createFor return a streamable resource. I agree that’s a good problem to solve. — A*
Re: File API: reading a Blob
On Thu, Aug 28, 2014 at 2:15 AM, Aymeric Vitte vitteayme...@gmail.com wrote: I have contributed to streams already, of course it will solve most of what is being discussed here but when? And then what's the point of this discussion? This discussion between me and Arun is about how to describe various algorithms in File API and Fetch, ideally enabling APIs on top of those low-level descriptions later on. But it is about describing the behavior of existing APIs in more detail. -- http://annevankesteren.nl/
Re: File API: reading a Blob
I have contributed to streams already, of course it will solve most of what is being discussed here but when? And then what's the point of this discussion? But maybe it will not solve everything, like my indexedDB example, which is not even considered in indexedDB V2 as far as I know Le 25/08/2014 11:46, Anne van Kesteren a écrit : On Sun, Aug 24, 2014 at 11:59 PM, Aymeric Vitte vitteayme...@gmail.com wrote: File, XHR and indexedDB should handle partial data, I thought I understood from messages last year that it was clearly identified as a mistake. This will be solved with streams. I recommend contributing to https://github.com/whatwg/streams because until we have such a primitive, there's nothing to expose. -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
On Sun, Aug 24, 2014 at 11:59 PM, Aymeric Vitte vitteayme...@gmail.com wrote: File, XHR and indexedDB should handle partial data, I thought I understood from messages last year that it was clearly identified as a mistake. This will be solved with streams. I recommend contributing to https://github.com/whatwg/streams because until we have such a primitive, there's nothing to expose. -- http://annevankesteren.nl/
Re: File API: reading a Blob
Back in this thread lately too... I still don't see how you intend to solve the use case I gave: - load 'something' progressively, get chunks (should be possible via xhr but it's not) - handle it via File augmenting the related Blob (not possible) and store it progressively in indexedDB (not possible too), or do the contrary, store it progressively and get it as an (uncomplete) Blob from indexedDB that will get augmented while it is stored (not possible again) - URL.createObjectURL(something) (will not work) where 'something' has a size of 1 GB (video for example) That's equivalent to do with your browser: file:/something or http://abcd.com/something where something is a file on you computer being downloaded or on a web site, so not complete. Which works very well since a long time. File, XHR and indexedDB should handle partial data, I thought I understood from messages last year that it was clearly identified as a mistake. Regards, Aymeric Le 11/08/2014 13:24, Anne van Kesteren a écrit : On Fri, Aug 8, 2014 at 5:56 PM, Arun Ranganathan a...@mozilla.com wrote: I strongly think we should leave FileReaderSync and FileReader alone. Also note that FileReaderSync and XHR (sync) are not different, in that both don’t do partial data. But we should have a stream api that evolves to read, and it might be something off Blob itself. Seems fair. Other than “chunks of bytes” which needs some normative backbone, is the basic abstract model what you had in mind? If so, that might be worth getting into File APi and calling it done, because that’s a reusable abstract model for Fetch and for FileSystem. Yeah that looks good. https://whatwg.github.io/streams/ defines chunks and such, but is not quite there yet. But it is what we want to build this upon eventually. -- Peersm : http://www.peersm.com torrent-live: https://github.com/Ayms/torrent-live node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
On Fri, Aug 8, 2014 at 5:56 PM, Arun Ranganathan a...@mozilla.com wrote: I strongly think we should leave FileReaderSync and FileReader alone. Also note that FileReaderSync and XHR (sync) are not different, in that both don’t do partial data. But we should have a stream api that evolves to read, and it might be something off Blob itself. Seems fair. Other than “chunks of bytes” which needs some normative backbone, is the basic abstract model what you had in mind? If so, that might be worth getting into File APi and calling it done, because that’s a reusable abstract model for Fetch and for FileSystem. Yeah that looks good. https://whatwg.github.io/streams/ defines chunks and such, but is not quite there yet. But it is what we want to build this upon eventually.
Re: File API: reading a Blob
Welcome back - we missed you :-) On Aug 5, 2014, at 9:43 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jul 17, 2014 at 2:58 PM, Arun Ranganathan a...@mozilla.com wrote: There are two questions: 1. How should FileReaderSync behave, to solve the majority of use cases? 2. What is a useful underlying abstraction for spec. authors that can be reused in present APIs like Fetch and future APIs? I'm not sure. I strongly think we should leave FileReaderSync and FileReader alone. Also note that FileReaderSync and XHR (sync) are not different, in that both don’t do partial data. But we should have a stream api that evolves to read, and it might be something off Blob itself. That leaves us the next problem, and what I think is the last problem in File API: Yeah, I now think that we want something even lower-level and build the task queuing primitive on top of that. (Basically by observing the stream that is being read and queuing tasks as data comes in, similar to Fetch. The synchronous case would just wait for the stream to complete. If I understand you correctly, you mean something that might be two-part (some hand waving below, but …): To read a Blob object /blob/, run these steps: 1. Let /s/ be a new buffer. 2. Return /s/, but continue running these steps asynchronously. 3. While /blob/'s underlying data stream is not closed, run these substeps: 1. Let /bytes/ be the result of reading a chunk from /blob/'s underlying data. 2. If /bytes/ is not failure, push /bytes/ to /s/ and set /s/'s transmitted to /bytes/'s length. 3. Otherwise, signal some kind of error to /s/ and terminate these steps. AND To read a Blob object with tasks: 1. Run the read a Blob algorithm above. 2. When reading the first /bytes/ queue a task called process read. 3. When pushing /bytes/ to /s/, queue a task called process read data. 4. When all /bytes/ are pushed to /s/ queue a task called process read EOF. 5. If an error condition is signaled queue a task called process error with a failure reason. Is “chunk” implementation defined? Right now we assume 1 byte or 50ms. “Chunk” seems a bit hand-wavy and hard to enforce, but… it might be the right approach. Would have to discuss with Domenic, but something like chunks seems to be much closer to how these things actually work. Other than “chunks of bytes” which needs some normative backbone, is the basic abstract model what you had in mind? If so, that might be worth getting into File APi and calling it done, because that’s a reusable abstract model for Fetch and for FileSystem. — A*
Re: File API: reading a Blob
Sorry for the late response Arun. I blame vacation and not being quite sure how we should solve this taking into account the legacy consumers. On Thu, Jul 17, 2014 at 2:58 PM, Arun Ranganathan a...@mozilla.com wrote: There are two questions: 1. How should FileReaderSync behave, to solve the majority of use cases? 2. What is a useful underlying abstraction for spec. authors that can be reused in present APIs like Fetch and future APIs? I'm not sure. We agreed some time ago to not have partial data. Pointer? I also don't really see how that makes sense given how asynchronous read would perform. Well, the bug that removed them is: https://www.w3.org/Bugs/Public/show_bug.cgi?id=23158 and dates to last year. Problems really include decoding strings according to the encoding determination for incomplete Blobs: http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0063.html Another thread covered deltas in progress events: http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/0069.html I don’t have pointers to IRC conversations, but: 1. Decoding was an issue with *readAsText*. I suppose we could make that method alone be all or nothing. Well a synchronous readAsText would presumably operate on the bytes returned. What that would do is clearly defined. Yeah, I now think that we want something even lower-level and build the task queuing primitive on top of that. (Basically by observing the stream that is being read and queuing tasks as data comes in, similar to Fetch. The synchronous case would just wait for the stream to complete. If I understand you correctly, you mean something that might be two-part (some hand waving below, but …): To read a Blob object /blob/, run these steps: 1. Let /s/ be a new buffer. 2. Return /s/, but continue running these steps asynchronously. 3. While /blob/'s underlying data stream is not closed, run these substeps: 1. Let /bytes/ be the result of reading a chunk from /blob/'s underlying data. 2. If /bytes/ is not failure, push /bytes/ to /s/ and set /s/'s transmitted to /bytes/'s length. 3. Otherwise, signal some kind of error to /s/ and terminate these steps. AND To read a Blob object with tasks: 1. Run the read a Blob algorithm above. 2. When reading the first /bytes/ queue a task called process read. 3. When pushing /bytes/ to /s/, queue a task called process read data. 4. When all /bytes/ are pushed to /s/ queue a task called process read EOF. 5. If an error condition is signaled queue a task called process error with a failure reason. Is “chunk” implementation defined? Right now we assume 1 byte or 50ms. “Chunk” seems a bit hand-wavy and hard to enforce, but… it might be the right approach. Would have to discuss with Domenic, but something like chunks seems to be much closer to how these things actually work. -- http://annevankesteren.nl/
Re: File API: reading a Blob
Aymeric, (Sorry for the delay; I was traveling for the last part of July.) On Jul 17, 2014, at 11:25 AM, Aymeric Vitte vitteayme...@gmail.com wrote: I don't get this, most reads inside browsers are about fetching, and fetching does increment a resource while it is loaded/displayed (image, video, crypto operations, etc) And it's question of the fetch API in this thread. XHR has the same issue, fortunately you can use XHR with Range, now I think this issue should stop propagating What I meant is that while I don’t think FileReader *or* FileReaderSync should be modified *at this stage* to include partial data, there is a workaround: 1. After obtaining the Blob or File, use .slice() to generate something like a mySlices[ ] array of e.g. 2K each slice. 2. Then use fileReader.readAsArrayBuffer(mySlices[0]…). This should be relatively efficient, and in fact can be cued to work at the completion of the last read, based on the final event being fired (or something similar — you get the idea). This might be similar to XHR range. This at least allows chunked reads. I agree that we need a stream-based API, but I don’t agree that it should be FileReader or FileReaderSync. That’s simply doing too much with too little. However, I think the next step would be to generate stream-based reads directly off the Blob, e.g. have something like myBlob.read(…) which returns a stream, which can be cued with API calls. 1. Decoding was an issue with *readAsText*. I suppose we could make that method alone be all or nothing. I don't see where the issue can be, it should behave like textdecoder and reinject the undecoded bytes to the next chunk, personnaly I find really useless to offer the 'text' methods in files/fetching APIs. The issue is that it isn’t possible to know the total number of decoded bytes is till the whole file is decoded, but of course we could only represent the bytes read thus far in progress events. Again, I don’t think FileReader or FileReaderSync are the right APIs for that. — A*
Re: File API: reading a Blob
Aymeric, (Sorry for the delay; I was traveling for the last part of July.) On Jul 17, 2014, at 11:25 AM, Aymeric Vitte vitteayme...@gmail.com wrote: I don't get this, most reads inside browsers are about fetching, and fetching does increment a resource while it is loaded/displayed (image, video, crypto operations, etc) And it's question of the fetch API in this thread. XHR has the same issue, fortunately you can use XHR with Range, now I think this issue should stop propagating What I meant is that while I don’t think FileReader *or* FileReaderSync should be modified *at this stage* to include partial data, there is a workaround: 1. After obtaining the Blob or File, use .slice() to generate something like a mySlices[ ] array of e.g. 2K each slice. 2. Then use fileReader.readAsArrayBuffer(mySlices[0]…). This should be relatively efficient, and in fact can be cued to work at the completion of the last read, based on the final event being fired (or something similar — you get the idea). This might be similar to XHR range. This at least allows chunked reads. I agree that we need a stream-based API, but I don’t agree that it should be FileReader or FileReaderSync. That’s simply doing too much with too little. However, I think the next step would be to generate stream-based reads directly off the Blob, e.g. have something like myBlob.read(…) which returns a stream, which can be cued with API calls. 1. Decoding was an issue with *readAsText*. I suppose we could make that method alone be all or nothing. I don't see where the issue can be, it should behave like textdecoder and reinject the undecoded bytes to the next chunk, personnaly I find really useless to offer the 'text' methods in files/fetching APIs. The issue is that it isn’t possible to know the total number of decoded bytes is till the whole file is decoded, but of course we could only represent the bytes read thus far in progress events. Again, I don’t think FileReader or FileReaderSync are the right APIs for that. — A*
Re: File API: reading a Blob
On Jul 14, 2014, at 3:47 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jul 10, 2014 at 7:05 PM, Arun Ranganathan a...@mozilla.com wrote: On Jul 3, 2014, at 10:50 AM, Anne van Kesteren ann...@annevk.nl wrote: That would mean you would get different results between using FileReaderSync and XMLHttpRequest. That does not seem ideal The implementation train has already left the station on this. The movitation of an “ideal match-up with XMLHttpRequest doesn’t seem strong enough to revisit this by filing browser bugs across implementations (but Cc’ing K. Huey also). Well, surely if we support both, we'd like them to work in the same way so they can share the same underlying abstraction. There are two questions: 1. How should FileReaderSync behave, to solve the majority of use cases? 2. What is a useful underlying abstraction for spec. authors that can be reused in present APIs like Fetch and future APIs? I don’t think it is necessary to mix the two questions for APIs that are already shipping. Do you think that FileReaderSync AND FileReader should support partial Blob data in read results? Or that stream-based reads should do this with a different API? We agreed some time ago to not have partial data. Pointer? I also don't really see how that makes sense given how asynchronous read would perform. Well, the bug that removed them is: https://www.w3.org/Bugs/Public/show_bug.cgi?id=23158 and dates to last year. Problems really include decoding strings according to the encoding determination for incomplete Blobs: http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0063.html Another thread covered deltas in progress events: http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/0069.html I don’t have pointers to IRC conversations, but: 1. Decoding was an issue with *readAsText*. I suppose we could make that method alone be all or nothing. 2. Use cases. MOST reads are useful with all data, after which you could use Blob manipulation (e.g. *slice*). New result objects each time didn’t seem optimal. And, it was something punted to Streams since that seemed like a longer term direction. There was also the idea of a Promise-based File API that could be consumed by the FileSystem API. But it might be useful to have an abstract read that helps Fetch and other things like Streams also: Yeah, I now think that we want something even lower-level and build the task queuing primitive on top of that. (Basically by observing the stream that is being read and queuing tasks as data comes in, similar to Fetch. The synchronous case would just wait for the stream to complete. If I understand you correctly, you mean something that might be two-part (some hand waving below, but …): To read a Blob object /blob/, run these steps: 1. Let /s/ be a new buffer. 2. Return /s/, but continue running these steps asynchronously. 3. While /blob/'s underlying data stream is not closed, run these substeps: 1. Let /bytes/ be the result of reading a chunk from /blob/'s underlying data. 2. If /bytes/ is not failure, push /bytes/ to /s/ and set /s/'s transmitted to /bytes/'s length. 3. Otherwise, signal some kind of error to /s/ and terminate these steps. AND To read a Blob object with tasks: 1. Run the read a Blob algorithm above. 2. When reading the first /bytes/ queue a task called process read. 3. When pushing /bytes/ to /s/, queue a task called process read data. 4. When all /bytes/ are pushed to /s/ queue a task called process read EOF. 5. If an error condition is signaled queue a task called process error with a failure reason. Is “chunk” implementation defined? Right now we assume 1 byte or 50ms. “Chunk” seems a bit hand-wavy and hard to enforce, but… it might be the right approach. — A*
Re: File API: reading a Blob
Aymeric, On Jul 16, 2014, at 8:20 AM, Aymeric Vitte vitteayme...@gmail.com wrote: Example: var myfile=new Blob(); //chunks are coming in myfile=new Blob([myfile,chunk],...) //mylink A tag mylink.href=URL.createObjectURL(myfile) click on mylink -- does not work Expected behavior: the file (a video for example) should play as it is incremented. This is inconsistent with the standard files behavior (see [1] for example), this example should work without having to use the Media Source Extensions API. This is a great use case, but breaks File API currently (and the abstract model, too). Options to consider might be: 1. Including partial Blob data with the existing FIle API. But we already ship as is, so we could forego decoding to get this right, but that doesn’t seem optimal either. 2. Use a new API for this, such as Streams. Since we’ll use streams for non-file cases, it could be useful here (and might be a better fit for the use case, which features heavy video use). — A*
Re: File API: reading a Blob
Arun, Streams will solve this, assuming they are implemented by all related APIs (this is not obvious, most of WGs seem not to care at all now that they have implemented their own so-called streams) Quoting your other answer: Well, the bug that removed them is:https://www.w3.org/Bugs/Public/show_bug.cgi?id=23158 and dates to last year. Indeed, you removed this right away after I mentioned to the group last year that it was not implemented inside browsers and that, designed as it was (to display a progress bar), it seemed not very useful, but I insisted at that time that what was useful was to have the delta data, so I don't really see a consensus of not implementing partial data. 2. Use cases. MOST reads are useful with all data, after which you could use Blob manipulation (e.g.*slice*). New result objects each time didn’t seem optimal. And, it was something punted to Streams since that seemed like a longer term direction. There was also the idea of a Promise-based File API that could be consumed by the FileSystem API. I don't get this, most reads inside browsers are about fetching, and fetching does increment a resource while it is loaded/displayed (image, video, crypto operations, etc) And it's question of the fetch API in this thread. XHR has the same issue, fortunately you can use XHR with Range, now I think this issue should stop propagating. 1. Decoding was an issue with*readAsText*. I suppose we could make that method alone be all or nothing. I don't see where the issue can be, it should behave like textdecoder and reinject the undecoded bytes to the next chunk, personnaly I find really useless to offer the 'text' methods in files/fetching APIs. So, I would insist again that you go for 1, whether it's promises like, append like, filewriter like, we must get access to partial data and we must be able to increment a blob (as well as its storage in indexedDB, other remark sent to the group some time ago, maybe you can combine both for the implementation since I don't really know what gets broken in the abstract model you mention) Regards Aymeric Le 17/07/2014 15:10, Arun Ranganathan a écrit : Aymeric, On Jul 16, 2014, at 8:20 AM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: Example: var myfile=new Blob(); //chunks are coming in myfile=new Blob([myfile,chunk],...) //mylink A tag mylink.href=URL.createObjectURL(myfile) click on mylink -- does not work Expected behavior: the file (a video for example) should play as it is incremented. This is inconsistent with the standard files behavior (see [1] for example), this example should work without having to use the Media Source Extensions API. This is a great use case, but breaks File API currently (and the abstract model, too). Options to consider might be: 1. Including partial Blob data with the existing FIle API. But we already ship as is, so we could forego decoding to get this right, but that doesn’t seem optimal either. 2. Use a new API for this, such as Streams. Since we’ll use streams for non-file cases, it could be useful here (and might be a better fit for the use case, which features heavy video use). — A* -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
Le 10/07/2014 19:05, Arun Ranganathan a écrit : We agreed some time ago to not have partial data. I still think that's a big mistake. Even if the Streams API will solve this, this should be corrected in the File API. Unless I am misusing the API, you can not even increment a Blob, you have to create a new one each time, this should be corrected too. Example: var myfile=new Blob(); //chunks are coming in myfile=new Blob([myfile,chunk],...) //mylink A tag mylink.href=URL.createObjectURL(myfile) click on mylink -- does not work Expected behavior: the file (a video for example) should play as it is incremented. This is inconsistent with the standard files behavior (see [1] for example), this example should work without having to use the Media Source Extensions API. Regards Aymeric [1] https://github.com/Ayms/torrent-live -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: File API: reading a Blob
On Thu, Jul 10, 2014 at 7:05 PM, Arun Ranganathan a...@mozilla.com wrote: On Jul 3, 2014, at 10:50 AM, Anne van Kesteren ann...@annevk.nl wrote: That would mean you would get different results between using FileReaderSync and XMLHttpRequest. That does not seem ideal The implementation train has already left the station on this. The movitation of an “ideal match-up with XMLHttpRequest doesn’t seem strong enough to revisit this by filing browser bugs across implementations (but Cc’ing K. Huey also). Well, surely if we support both, we'd like them to work in the same way so they can share the same underlying abstraction. We agreed some time ago to not have partial data. Pointer? I also don't really see how that makes sense given how asynchronous read would perform. Then let’s have a different read operation that doesn’t use the FileReader Task Source. The primary API in File API uses the event loop (FileReader), so for that purpose, the existing read operation is important. Early discussions about this made me feel that a task queue based system to asynchronously read blobs could be reusable by Fetch, and by future Promise-based APIs. Yeah, I now think that we want something even lower-level and build the task queuing primitive on top of that. (Basically by observing the stream that is being read and queuing tasks as data comes in, similar to Fetch. The synchronous case would just wait for the stream to complete. -- http://annevankesteren.nl/
Re: File API: reading a Blob
On Jul 3, 2014, at 10:50 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jul 3, 2014 at 4:29 PM, Arun Ranganathan a...@mozilla.com wrote: OK, this is fixable. I’ll ensure that the read operation’s synchronous component does return the results thus far, but that FIleReaderSync itself ignores them in case of a midstream error, unless we collectively agree that it SHOULD return partial instead of “all or nothing” as an API. My understanding of the FileReaderSync requirement is all or nothing, but I’m open to being wrong via bug filing. That would mean you would get different results between using FileReaderSync and XMLHttpRequest. That does not seem ideal The implementation train has already left the station on this. The movitation of an “ideal match-up with XMLHttpRequest doesn’t seem strong enough to revisit this by filing browser bugs across implementations (but Cc’ing K. Huey also). We agreed some time ago to not have partial data. The tasks are still a bit of a concern as a normal implementation would not queue tasks. E.g. it's not even clear what task loop Fetch would retrieve this from as fetch is itself a different process. Then let’s have a different read operation that doesn’t use the FileReader Task Source. The primary API in File API uses the event loop (FileReader), so for that purpose, the existing read operation is important. Early discussions about this made me feel that a task queue based system to asynchronously read blobs could be reusable by Fetch, and by future Promise-based APIs. Since it’s not reusable for Fetch, let’s not try and force it. We’ll do a new one along the lines you described at the start of this email. — A*
Re: File API: reading a Blob
On Wed, Jul 2, 2014 at 7:06 PM, Arun Ranganathan a...@mozilla.com wrote: For instance, I thought the idea was that within Fetch to read /blob/ we’d do something like: 1. Let /s/ be a new body. Return /s/ and perform the rest of the steps async. 2. Perform a read operation [File API] on /blob/. 3. To process read… 4. To process read data, transfer each byte read to /s/ and set /s/’s transmitted to the number of bytes read. // Chunked byte transfer is possible within the 50ms delta for process read data. We could specify that here better.// 5. To process read EOF ... 6. Otherwise, to process read error with a failure reason on /s/ …. Why is something like that unworkable and why do we need another variant of the read operation exactly? It's unclear to me why we'd want to use the event loop for this, basically. Also, the current set of synchronous steps (which I could opt not to use, granted, but other APIs already might, and I'd like us to be consistent) simply return failure when something bad happens rather than returning the bytes read so far. It seems like that is a problem, perhaps following from not having this lower-level description. On Wed, Jul 2, 2014 at 8:51 PM, Domenic Denicola dome...@domenicdenicola.com wrote: Or are you trying to transfer from a vague conceptual model of /blob/'s underlying data stream to a different conceptual model of A body is a byte stream, only to later actually reify the byte stream as a stream object? That seems potentially problematic, but perhaps it could be made to work... Yes. E.g. for img there would not be an observable stream here. Just some of the effects (JPEG progression) might be observable. Fetch is the model of the networking layer, there are no concrete JavaScript objects there. -- http://annevankesteren.nl/
Re: File API: reading a Blob
On Jul 3, 2014, at 4:14 AM, Anne van Kesteren ann...@annevk.nl wrote: It's unclear to me why we'd want to use the event loop for this, basically. The FileReader object uses the event loop; your initial request for Fetch was to have a reusable “abstract” read operation which used tasks. You’ve since changed your mind, which is totally fine: we could have a different read operation that doesn’t use the event loop that’s put in place for Fetch, but FileReader needs the event loop. We’re talking about an abstract model in specification land on which to pin higher level concepts that culminate eventually in JS objects. It’s useful (I agreed long ago), but overhauling the current read operation for a change of mind/model is a lot of pain without elegant gain. Also, since there isn’t an observable stream or an object, but merely things like load progression (JPEG progression), tasks does seem useful for that. The one thing I suggested which we could do better is the “one byte vs. 50ms” model, and use the “chunk” concept of bytes that the streams folks use. The one thing I’m not clear on here is how to get to a pull-chunk size for files, but I think we could do this better. Also, the current set of synchronous steps (which I could opt not to use, granted, but other APIs already might, and I'd like us to be consistent) simply return failure when something bad happens rather than returning the bytes read so far. It seems like that is a problem, perhaps following from not having this lower-level description. OK, this could be a problem. But this is really immediately usable by the FileReaderSync object on threads, for which a use case for partial data didn’t materialize (and in general, the spec. shunned partial data — references to those threads date way back in time, but they’re there). It seems that for a synchronous block of file i/o, all or nothing catered to most use cases. But if it IS a problem — that is, if you think synchronous I/O has implications outside of FileReaderSync, OR that FileReaderSync’s return itself should be partial if a failure occurs, then let’s file bugs and solve them. I’d REALLY like to have solid abstract models in place for Fetch, since I buy into the cross-purposability of it. But I’d also like shipping implementations to be defined (99% done in File API), with the small delta remaining — Blob URL autorevoking and Blob closing — to be nailed down. — A*
Re: File API: reading a Blob
On Thu, Jul 3, 2014 at 3:58 PM, Arun Ranganathan a...@mozilla.com wrote: You’ve since changed your mind, which is totally fine I wish I could foresee all problems before starting to write things out, that'd be great! :-) but overhauling the current read operation for a change of mind/model is a lot of pain without elegant gain. Well, it does seem like a problem that synchronous and asynchronous operations have different error handling as far as what bytes they return is concerned. But if it IS a problem — that is, if you think synchronous I/O has implications outside of FileReaderSync, OR that FileReaderSync’s return itself should be partial if a failure occurs, then let’s file bugs and solve them. So most of Fetch is asynchronous. If it's invoked with the synchronous flag set it's just that it waits for the entire response before returning. That's why I'd like to use the asynchronous path of reading a blob. But I'd like that not to be observably different from using the synchronous path of reading a blob. That seems wrong. -- http://annevankesteren.nl/
Re: File API: reading a Blob
On Jul 3, 2014, at 10:17 AM, Anne van Kesteren ann...@annevk.nl wrote: So most of Fetch is asynchronous. If it's invoked with the synchronous flag set it's just that it waits for the entire response before returning. That's why I'd like to use the asynchronous path of reading a blob. But I'd like that not to be observably different from using the synchronous path of reading a blob. That seems wrong. OK, this is fixable. I’ll ensure that the read operation’s synchronous component does return the results thus far, but that FIleReaderSync itself ignores them in case of a midstream error, unless we collectively agree that it SHOULD return partial instead of “all or nothing” as an API. My understanding of the FileReaderSync requirement is all or nothing, but I’m open to being wrong via bug filing. Are you agreed (as far as it is possible for you to agree with me on anything) that the event loop async read might allow us to address cases like JPEG progression? It seems imminently usable here. — A*
Re: File API: reading a Blob
On Thu, Jul 3, 2014 at 4:29 PM, Arun Ranganathan a...@mozilla.com wrote: OK, this is fixable. I’ll ensure that the read operation’s synchronous component does return the results thus far, but that FIleReaderSync itself ignores them in case of a midstream error, unless we collectively agree that it SHOULD return partial instead of “all or nothing” as an API. My understanding of the FileReaderSync requirement is all or nothing, but I’m open to being wrong via bug filing. That would mean you would get different results between using FileReaderSync and XMLHttpRequest. That does not seem ideal. Are you agreed (as far as it is possible for you to agree with me on anything) that the event loop async read might allow us to address cases like JPEG progression? It seems imminently usable here. The tasks are still a bit of a concern as a normal implementation would not queue tasks. E.g. it's not even clear what task loop Fetch would retrieve this from as fetch is itself a different process. -- http://annevankesteren.nl/
Re: File API: reading a Blob
On Jul 2, 2014, at 10:28 AM, Anne van Kesteren ann...@annevk.nl wrote: So what I need is something like this: To read a Blob object /blob/, run these steps: 1. Let /s/ be a new body. [FETCH] 2. Return /s/, but continue running these steps asynchronously. 3. While /blob/'s underlying data stream is not closed, run these substeps: 1. Let /bytes/ be the result of reading a chunk from /blob/'s underlying data. 2. If /bytes/ is not failure, push /bytes/ to /s/ and set /s/'s transmitted to /bytes/'s length. 3. Otherwise, signal some kind of error to /s/ and terminate these steps. Are you sure this simply cannot be done with the existing read operation, which uses annotated tasks for asynchronous use? The existing read operation serves FileReader and FileReaderSync (and maybe other APIs in FileSystem) and hopefully Fetch too. For instance, I thought the idea was that within Fetch to read /blob/ we’d do something like: 1. Let /s/ be a new body. Return /s/ and perform the rest of the steps async. 2. Perform a read operation [File API] on /blob/. 3. To process read… 4. To process read data, transfer each byte read to /s/ and set /s/’s transmitted to the number of bytes read. // Chunked byte transfer is possible within the 50ms delta for process read data. We could specify that here better.// 5. To process read EOF ... 6. Otherwise, to process read error with a failure reason on /s/ …. Why is something like that unworkable and why do we need another variant of the read operation exactly? — A*
RE: File API: reading a Blob
I am a little unclear as to what you are trying to accomplish here. From my perspective streams are not really an abstract concept, but instead are specific JavaScript APIs that conform to a given contract. In contrast, an underlying source [1] is an abstract concept; a given (readable) stream object wraps such an abstract underlying source. Are you trying to specify the stream object itself? Your /s/ seems like it might be that. In which case, I think the most rigorous approach is that taken by Claes in [2]; see e.g. step 10. The terminology has changed a tiny bit since he wrote it (s/push/enqueue/g, and we try to use the word chunk now), but the basic idea is sound. I think in your case you are describing a pull underlying source, whereas his was a push, so most of your stuff would go in pull() instead of start(). Also I don't think failure signals are usually in the /bytes/, but instead out-of-band (e.g. return codes from syscalls). Or are you trying to transfer from a vague conceptual model of /blob/'s underlying data stream to a different conceptual model of A body is a byte stream, only to later actually reify the byte stream as a stream object? That seems potentially problematic, but perhaps it could be made to work... [1]: https://whatwg.github.io/streams/#rs-model [2]: http://www.w3.org/2012/sysapps/tcp-udp-sockets/#interface-tcpsocket From: annevankeste...@gmail.com annevankeste...@gmail.com on behalf of Anne van Kesteren ann...@annevk.nl Sent: Wednesday, July 02, 2014 10:28 To: WebApps WG; Arun Ranganathan Cc: Domenic Denicola Subject: File API: reading a Blob I tried to work out Fetch/Blob integration today. What I actually want is that reading a Blob is basically pushing the IO bytes that are read in chunks into a stream. Specifically, a body: http://fetch.spec.whatwg.org/#concept-body That should also make it clearer what happens if the underlying data is borked somehow (file is moved, disk corrupted). Presumably that signals some kind of error to the stream, but would not cause complete failure. Something similar to a network connection going down while in transmission. I also don't really need the synchronous flag for this purpose. I just need an algorithm (invoked async) that pushes data from disk on this stream. Making the fetch appear synchronous is handled elsewhere. The only problem with doing this is that we don't really have a good way to talk about streams in the abstract yet. Standards such as Encoding define some terminology, but it's not really a complete model yet, there's certainly no error handling. So what I need is something like this: To read a Blob object /blob/, run these steps: 1. Let /s/ be a new body. [FETCH] 2. Return /s/, but continue running these steps asynchronously. 3. While /blob/'s underlying data stream is not closed, run these substeps: 1. Let /bytes/ be the result of reading a chunk from /blob/'s underlying data. 2. If /bytes/ is not failure, push /bytes/ to /s/ and set /s/'s transmitted to /bytes/'s length. 3. Otherwise, signal some kind of error to /s/ and terminate these steps. What do you think? -- http://annevankesteren.nl/
