Re: Thoughts behind the Streams API ED
Takeshi, See discussion here too: https://github.com/whatwg/streams/issues/33 The problem with stop again is that I need to handle myself the clone operations, the advantage of stop-eof is: - clone the operation - close it - restart from the clone And as I mentioned before this would work for any operation that has unresolved bytes (TextDecoder, etc) without the need of modifying the operation API for explicit clones or options. Regards Aymeric Le 08/11/2013 14:31, Takeshi Yoshino a e'crit : On Fri, Nov 8, 2013 at 8:54 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: I would expect Poc (stop, keep 0xd1 for the next data) and сия It can be seen a bit different indeed, while with crypto you expect the finalization of the operation since the begining (but only by computing the latest bytes), here you can not expect the string since the begining of course. It just depends how the Operation (here TextDecoder) handles stop but I find it very similar, TextEncoder closes the operation with the bytes it has and clone its state (ie do nothing here except clearing resolved bytes and keeping unresolved ones for data to come). I'd say more generally that stop() is kinda in-band control signal that is inserted between elements of the stream and distinguishable from the elements. As you said, interpretation of the stop() symbol depends on what the destination is. One thing I'm still not sure is that I think you can just add stop() equivalent method to the destination, and - pipe() data until the point you were calling stop() - call the stop() equivalent on e.g. hash - restart pipe() At least our spec allows for this. Of course, it's convenient that Stream can carry such a signal. But there's trade-off between the convenience and API size. Similar to decision whether we include abort() to WritableByteStream or not. Extremely, abort(), close() and stop() can be merged into one method (unless abort() method has a functionality to abandon already written data). They're all signal inserting methods. close() - signal(FIN) stop(info) - signal(CONTROL, info) abort(error) - signal(ABORT, error) and the signal is packed and inserted into the stream's internal buffer. -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
No, see my previous reply, unless I am proven incorrect, I still think we should have: - pause/unpause - stop/(implicit resume) Regards, Aymeric Le 11/11/2013 22:06, Takeshi Yoshino a écrit : Aymeric, Re: pause()/resume(), I've moved flow control functionality for non-exact read() method to a separate attribute pullAmount [1] [2]. pullAmount limits the max size of data to be read by read() method. Currently the pipe() method is specified not to respect pullAmount but we can choose to have it to respect pullAmount, i.e. pausing pipe()'s transfer when pullAmount is set to 0. Does this work for your use case? [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=23790 [2] https://dvcs.w3.org/hg/streams-api/rev/8a7f99536516 -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
On Tue, Nov 12, 2013 at 5:23 PM, Aymeric Vitte vitteayme...@gmail.comwrote: No, see my previous reply, unless I am proven incorrect, I still think we should have: - pause/unpause - stop/(implicit resume) Regards, Aymeric Le 11/11/2013 22:06, Takeshi Yoshino a écrit : Aymeric, Re: pause()/resume(), Sorry. This is typo. I meant pause()/unpause(). I've moved flow control functionality for non-exact read() method to a separate attribute pullAmount [1] [2]. pullAmount limits the max size of data to be read by read() method. Currently the pipe() method is specified not to respect pullAmount but we can choose to have it to respect pullAmount, i.e. pausing pipe()'s transfer when pullAmount is set to 0. Does this work for your use case? [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=23790 [2] https://dvcs.w3.org/hg/streams-api/rev/8a7f99536516 -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
On Tue, Nov 12, 2013 at 5:20 PM, Aymeric Vitte vitteayme...@gmail.comwrote: Takeshi, See discussion here too: https://github.com/whatwg/streams/issues/33 The problem with stop again is that I need to handle myself the clone operations, the advantage of stop-eof is: - clone the operation - close it - restart from the clone And as I mentioned before this would work for any operation that has unresolved bytes (TextDecoder, etc) without the need of modifying the operation API for explicit clones or options. OK. I understood your needs. As explained in the previous mail, if we decide to take in your suggestion, I'd implement your stop-eof as a more generic in-band control signal. In other words, changing the draft to propose message stream than byte stream. I estimate that this feature sophisticates implementation much. So, I want to hear more support and justification. For now, I filed a bug to track discussion. https://www.w3.org/Bugs/Public/show_bug.cgi?id=23799
Re: Thoughts behind the Streams API ED
Aymeric, Re: pause()/resume(), I've moved flow control functionality for non-exact read() method to a separate attribute pullAmount [1] [2]. pullAmount limits the max size of data to be read by read() method. Currently the pipe() method is specified not to respect pullAmount but we can choose to have it to respect pullAmount, i.e. pausing pipe()'s transfer when pullAmount is set to 0. Does this work for your use case? [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=23790 [2] https://dvcs.w3.org/hg/streams-api/rev/8a7f99536516
Re: Thoughts behind the Streams API ED
Please see here https://github.com/whatwg/streams/issues/33, I realized that this would apply to operations like textDecoder too without the need of an explicit stream option, so that's no more WebCrypto only related. Regards Aymeric Le 07/11/2013 11:25, Aymeric Vitte a écrit : Le 07/11/2013 10:42, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:27 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: Le 07/11/2013 10:21, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I So, here, partial hash for ABCDEFGH is output No, you get the digest for ABCDEFGHI and you get a cloned operation which will restart from ABCDEFGH OK. resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is padding). Right? If you close the stream here you get the digest for ABCDEFGHIJKLF resume happens implicitly when new data comes in without explicit method call say resume()? Good question, I would say yes, so we don't need resume finally, but maybe others have different opinion, let's ask whatwg if they foresee this case. So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node. -- Peersm :http://www.peersm.com node-Tor :https://www.github.com/Ayms/node-Tor GitHub :https://www.github.com/Ayms -- Peersm :http://www.peersm.com node-Tor :https://www.github.com/Ayms/node-Tor GitHub :https://www.github.com/Ayms -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
On Fri, Nov 8, 2013 at 5:38 PM, Aymeric Vitte vitteayme...@gmail.comwrote: Please see here https://github.com/whatwg/streams/issues/33, I realized that this would apply to operations like textDecoder too without the need of an explicit stream option, so that's no more WebCrypto only related. Similar but a bit different? For clarification, could you review the following? textDecoderStream.write(araybuffer of 0xd0 0xa0 0xd0 0xbe 0xd1); textDecoderStream.stop(); textDecoderStream.write(arraybuffer of 0x81 0xd1 0x81 0xd0 0xb8 0xd1 0x8f) This generates DOMString stream [Рос, сия]. Right? Or you want to get [Рос, Россия]?
Re: Thoughts behind the Streams API ED
Sorry. I've cut the input at wrong position. textDecoderStream.write(arraybuffer of 0xd0 0xa0 0xd0 0xbe 0xd1 0x81 0xd1); textDecoderStream.stop(); textDecoderStream.write(arraybuffer of 0x81 0xd0 0xb8 0xd1 0x8f)
Re: Thoughts behind the Streams API ED
I would expect Poc (stop, keep 0xd1 for the next data) and сия It can be seen a bit different indeed, while with crypto you expect the finalization of the operation since the begining (but only by computing the latest bytes), here you can not expect the string since the begining of course. It just depends how the Operation (here TextDecoder) handles stop but I find it very similar, TextEncoder closes the operation with the bytes it has and clone its state (ie do nothing here except clearing resolved bytes and keeping unresolved ones for data to come). Regards Aymeric Le 08/11/2013 11:33, Takeshi Yoshino a écrit : Sorry. I've cut the input at wrong position. textDecoderStream.write(arraybuffer of 0xd0 0xa0 0xd0 0xbe 0xd1 0x81 0xd1); textDecoderStream.stop(); textDecoderStream.write(arraybuffer of 0x81 0xd0 0xb8 0xd1 0x8f) -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
On Fri, Nov 8, 2013 at 8:54 PM, Aymeric Vitte vitteayme...@gmail.comwrote: I would expect Poc (stop, keep 0xd1 for the next data) and сия It can be seen a bit different indeed, while with crypto you expect the finalization of the operation since the begining (but only by computing the latest bytes), here you can not expect the string since the begining of course. It just depends how the Operation (here TextDecoder) handles stop but I find it very similar, TextEncoder closes the operation with the bytes it has and clone its state (ie do nothing here except clearing resolved bytes and keeping unresolved ones for data to come). I'd say more generally that stop() is kinda in-band control signal that is inserted between elements of the stream and distinguishable from the elements. As you said, interpretation of the stop() symbol depends on what the destination is. One thing I'm still not sure is that I think you can just add stop() equivalent method to the destination, and - pipe() data until the point you were calling stop() - call the stop() equivalent on e.g. hash - restart pipe() At least our spec allows for this. Of course, it's convenient that Stream can carry such a signal. But there's trade-off between the convenience and API size. Similar to decision whether we include abort() to WritableByteStream or not. Extremely, abort(), close() and stop() can be merged into one method (unless abort() method has a functionality to abandon already written data). They're all signal inserting methods. close() - signal(FIN) stop(info) - signal(CONTROL, info) abort(error) - signal(ABORT, error) and the signal is packed and inserted into the stream's internal buffer.
Re: Thoughts behind the Streams API ED
stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node. pipe/fork: I think b is better. Regards Aymeric Le 06/11/2013 20:15, Takeshi Yoshino a écrit : On Wed, Nov 6, 2013 at 7:33 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: I have seen the different bugs too, some comments: - maybe I have missed some explaination or some obvious thing but I don't understand very well right now the difference/use between readable/writablebytestream and bytestream ReadableByteStream and WritableByteStream are defining interfaces not only for ByteStream but more generally for other APIs. For example, we discussed how WebCrypto's encryption method should work with Stream concept recently, and one idea you showed was making WebCrypto.subtle return an object (which I called filter) to which we can pipe data. By defining a protocol how to pass data to consumer as the WritableByteStream interface, we can reuse it later for defining IDL for those filters. Similarly, ReadableByteStream can provide uniform protocol how data producing APIs should communicate with consumers. ByteStream is now a class inheriting both ReadableByteStream and WritableByteStream (sorry, I forgot to include inheritance info in the IDL). - pause/unpause: as far as I understand the whatwg spec does not recommend it but I don't understand the reasons. As I previously mentionned the idea is to INSERT a pause signal in the stream, you can not control the stream and therefore know when you are pausing it. Maybe after decoupling the interface, pause/unpause are things to be added to ByteStream? IIUC, pause prevents data from being read from a ByteStream, and unpause removes the dam? - stop/resume: same, see my previous post, the difference is that the consuming API should clone the state of the operation and close the current operation as if eof was received, then restart from the clone on resume Sorry that I haven't replied to your one. Your post about those methods: http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0343.html WebCrypto ISSUE-22: http://www.w3.org/2012/webcrypto/track/issues/22 Maybe I still don't quite understand your ideas. Let me confirm. stop() tells the consumer API implementing WritableByteStream that it should behave as if it received EOF, but when resume() is called, restart processing the data written between stop() call and resume() call as if the API received data for the first time? How should stop() work for ByteStream? ByteStream's read() method will receive EOF at least once when all data written before stop() call has been read, and it keeps returning EOF until resume() tells the ByteStream to restart outputting? I've been feeling that your use case is very specific to WebCrypto. Saving state and restoring it sounds more like feature request for WebCrypto, not a Stream. But I'm a bit interested in what your stop()/resume() enables. With this feature, ByteStream becomes message stream which is convenient for handling WebSocket. - pipe []/fork: I don't see why the fast stream should wait for the slow one, so maybe the stream is forked and pause can be used for the slow one There could be apps that want to limit memory usage strictly. We think there're two strategies fork() can take. a) wait until the slowest substream consumes b) grow not to block the fastest substream while keeping data for the slowest a) is useful to limit memory usage. b) is more performance oriented. - flow control: could it be possible to advertise a maximum bandwidth rate for a stream? It's currently communicated as window similar to TCP. Consumer can adjust size argument of read() and frequency of read() call to match the
Re: Thoughts behind the Streams API ED
On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.comwrote: stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I So, here, partial hash for ABCDEFGH is output resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is padding). Right? So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node.
Re: Thoughts behind the Streams API ED
Le 07/11/2013 10:21, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I So, here, partial hash for ABCDEFGH is output No, you get the digest for ABCDEFGHI and you get a cloned operation which will restart from ABCDEFGH resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is padding). Right? If you close the stream here you get the digest for ABCDEFGHIJKLF So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node. -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
On Thu, Nov 7, 2013 at 6:27 PM, Aymeric Vitte vitteayme...@gmail.comwrote: Le 07/11/2013 10:21, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.comwrote: stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I So, here, partial hash for ABCDEFGH is output No, you get the digest for ABCDEFGHI and you get a cloned operation which will restart from ABCDEFGH OK. resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is padding). Right? If you close the stream here you get the digest for ABCDEFGHIJKLF resume happens implicitly when new data comes in without explicit method call say resume()? So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node. -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
Le 07/11/2013 10:42, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:27 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: Le 07/11/2013 10:21, Takeshi Yoshino a écrit : On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.com mailto:vitteayme...@gmail.com wrote: stop/resume: Indeed as I mentioned this is related to WebCrypto Issue22 but I don't think this is a unique case. Issue22 was closed because of lack of proposals to solve it, apparently I was the only one to care about it (but I saw recently some other messages that seem to be related), and finally this would involve a public clone method with associated security concerns. But with Streams it could be different, the application will internally clone the state of the operation probably eliminating the security issues, as simple as that. To describe simply the use case, let's take a progressive hash computing 4 bytes by 4 bytes: incoming stream: ABCDE bytes hash operation: process ABCD, keep E for the next computation incoming stream: FGHI bytes + STOP-EOF hash operation: process EFGH, process STOP-EOF: clone the state of the hash, close the operation: digest hash with I So, here, partial hash for ABCDEFGH is output No, you get the digest for ABCDEFGHI and you get a cloned operation which will restart from ABCDEFGH OK. resume: incoming stream: JKLF hash operation (clone): process IJKL, keep F for next computation etc... and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is padding). Right? If you close the stream here you get the digest for ABCDEFGHIJKLF resume happens implicitly when new data comes in without explicit method call say resume()? Good question, I would say yes, so we don't need resume finally, but maybe others have different opinion, let's ask whatwg if they foresee this case. So you do not restart the operation as if it was the first time it was receiving data, you just continue it from the state it was when stop was received. That's not so unusual to do this, it has been requested many times in node. -- Peersm :http://www.peersm.com node-Tor :https://www.github.com/Ayms/node-Tor GitHub :https://www.github.com/Ayms -- Peersm : http://www.peersm.com node-Tor : https://www.github.com/Ayms/node-Tor GitHub : https://www.github.com/Ayms
Re: Thoughts behind the Streams API ED
I have seen the different bugs too, some comments: - maybe I have missed some explaination or some obvious thing but I don't understand very well right now the difference/use between readable/writablebytestream and bytestream - pause/unpause: as far as I understand the whatwg spec does not recommend it but I don't understand the reasons. As I previously mentionned the idea is to INSERT a pause signal in the stream, you can not control the stream and therefore know when you are pausing it. - stop/resume: same, see my previous post, the difference is that the consuming API should clone the state of the operation and close the current operation as if eof was received, then restart from the clone on resume - pipe []/fork: I don't see why the fast stream should wait for the slow one, so maybe the stream is forked and pause can be used for the slow one - flow control: could it be possible to advertise a maximum bandwidth rate for a stream? Regards Aymeric Le 04/11/2013 18:11, Takeshi Yoshino a écrit : I'd like to summarize my ideas behind this API surface since the overlap thread is too long. We'll put these into bug entries soon. Feedback on Overlap thread, especially Issac's exhaustive list of considerations and conversation with Aymeric were very helpful. In reply to his mail, I drafted my initial proposal [2] in past which addresses almost all of them. Since the API surface was so big, I tried to compact it while incorporating Promises. Current ED [3] addresses not all but some of important requirements. I think it's a good (re)starting point. * Flow control read() and write() in the ED does provide flow control by controlling the timing of resolution of the returned Promise. A Stream would have a window to limit data to be buffered in it. If a big value is passed as size parameter of read(), it may extend the window if necessary. In reading data as a DOMString, the size param of read() doesn't specify exact raw size of data to be read out. It just works as throttle to prevent internal buffer from being drained too fast. StreamReadResult tells how many bytes were actually consumed. If more explicit and precise flow control is necessary, we could cherry pick some from my old big API proposal [1]. For example, making window size configurable. If it makes sense, size can be generalized to be cost of each element. It would be useful when trying to generalize Stream to various objects. To make window dynamically adjustable, we could introduce methods such as drainCapacity(), expandCapacity() to it. * Passive producer Thinking of producers like a random number generator, it's not always good to ask a producer to prepare data and push it to a Stream using write(). This was possible in [2], but not in the ED. This can be addressed for example by adding one overload to write(). - write() and write(size) doesn't write data but wait until the Stream can accept some amount or the specified size of data. * Conversion from existing active and unstoppable producer API E.g. WebSocket invokes onmessage immediately when new data is available. For this kind of API, finite size Stream cannot absorb the production. So, there'll be need to buffer read data manually. In [2], Stream always accepted write() even if buffer is full assuming that if necessary the producer should be using onwritable method. Currently, only one write() can be issued concurrently, but we can choose to have Stream queue write() requests in it. * Sync read if possible By adding sync flag to StreamReadResult and introducing StreamWriteResult to signal if read was done sync (data is the actual result) or async (data is a Promise) to save Promise post tasking cost. I estimated that post tasking overhead should be negligible for bulk reading, and when to read small fields, we often read some amount into ArrayBuffer and then parse it directly. So it's currently excluded. * Multiple consumers pipe() can take multiple destination Streams. This allows for mirroring Stream's output into two or more Streams. I also considered making Stream itself consumable, but I thought it complicates API and implementation. - Elements in internal buffer need to be reference counted. - It must be able to distinguish consumers. If one of consumers is fast and the other is slow, we need to wait for the slower one before starting processing the rest in the original Stream. We can choose to allow multiple consumers to address this by introducing a new concept Cursor that represents reading context. Cursor can be implemented as a new interface or Stream that refers to (and adds reference count to elements of) the original Stream's internal buffer. Needs some more study to figure out if context approach is really better than pipe()-ing to new Stream instance. * Splitting InputStream (ReadableStream) and OutputStream (WritableStream) Writing part and reading part of the API can be split into two
Re: Thoughts behind the Streams API ED
On Wed, Nov 6, 2013 at 7:33 PM, Aymeric Vitte vitteayme...@gmail.comwrote: I have seen the different bugs too, some comments: - maybe I have missed some explaination or some obvious thing but I don't understand very well right now the difference/use between readable/writablebytestream and bytestream ReadableByteStream and WritableByteStream are defining interfaces not only for ByteStream but more generally for other APIs. For example, we discussed how WebCrypto's encryption method should work with Stream concept recently, and one idea you showed was making WebCrypto.subtle return an object (which I called filter) to which we can pipe data. By defining a protocol how to pass data to consumer as the WritableByteStream interface, we can reuse it later for defining IDL for those filters. Similarly, ReadableByteStream can provide uniform protocol how data producing APIs should communicate with consumers. ByteStream is now a class inheriting both ReadableByteStream and WritableByteStream (sorry, I forgot to include inheritance info in the IDL). - pause/unpause: as far as I understand the whatwg spec does not recommend it but I don't understand the reasons. As I previously mentionned the idea is to INSERT a pause signal in the stream, you can not control the stream and therefore know when you are pausing it. Maybe after decoupling the interface, pause/unpause are things to be added to ByteStream? IIUC, pause prevents data from being read from a ByteStream, and unpause removes the dam? - stop/resume: same, see my previous post, the difference is that the consuming API should clone the state of the operation and close the current operation as if eof was received, then restart from the clone on resume Sorry that I haven't replied to your one. Your post about those methods: http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0343.html WebCrypto ISSUE-22: http://www.w3.org/2012/webcrypto/track/issues/22 Maybe I still don't quite understand your ideas. Let me confirm. stop() tells the consumer API implementing WritableByteStream that it should behave as if it received EOF, but when resume() is called, restart processing the data written between stop() call and resume() call as if the API received data for the first time? How should stop() work for ByteStream? ByteStream's read() method will receive EOF at least once when all data written before stop() call has been read, and it keeps returning EOF until resume() tells the ByteStream to restart outputting? I've been feeling that your use case is very specific to WebCrypto. Saving state and restoring it sounds more like feature request for WebCrypto, not a Stream. But I'm a bit interested in what your stop()/resume() enables. With this feature, ByteStream becomes message stream which is convenient for handling WebSocket. - pipe []/fork: I don't see why the fast stream should wait for the slow one, so maybe the stream is forked and pause can be used for the slow one There could be apps that want to limit memory usage strictly. We think there're two strategies fork() can take. a) wait until the slowest substream consumes b) grow not to block the fastest substream while keeping data for the slowest a) is useful to limit memory usage. b) is more performance oriented. - flow control: could it be possible to advertise a maximum bandwidth rate for a stream? It's currently communicated as window similar to TCP. Consumer can adjust size argument of read() and frequency of read() call to match the consumer's processing speed.
Re: Thoughts behind the Streams API ED
FYI, I added a branch named Preview version into which suggestions are incorporated aggressively to see how the API surface would be change. https://dvcs.w3.org/hg/streams-api/raw-file/tip/preview.html Please take a look if you're interested in. For stabler version edited after having discussion, check ED as usual.
Re: Thoughts behind the Streams API ED
On Tue, Nov 5, 2013 at 2:11 AM, Takeshi Yoshino tyosh...@google.com wrote: Feedback on Overlap thread, especially Issac's exhaustive list of considerations Deleted citation by mistake. [1] is the Issac's post. [1] http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0355.html
