where Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-05 Thread Charles McCathie Nevile
On Mon, 04 Nov 2013 09:52:20 +0100, Domenic Denicola  
dome...@domenicdenicola.com wrote:


As for *where* the work is done, I will be working within the context of  
the WHATWG to produce this specification. My understanding is that  
usually the W3C picks some point in time to fork WHATWG specifications  
into W3C ones, changes some minor details (such as removing authorship  
information and changing the genders used in examples), then advancing  
it through the usual ED/WD/LCWD/CR/PR/REC track in order to get patent  
disclosure.


Err, no.

Although this has happened, it is not the usual pattern and is not the way  
this or other Working Groups try to work.


I'm very interested in ensuring patent disclosure for the streams  
specification, so I hope someone takes on this work, but I do not think  
it would be a good use of my time to do so, as from what I understand  
there are people at the W3C who have this process down to an art.


You seem to have misunderstood the point of the Working Group. Although  
Art is pretty efficient at publishing documents where the editor doesn't  
do the work required, he is under no obligation to do so and it is not a  
goal of the Working Group to spend his time on that.


The idea is that the Working Group discusses the issues it has, reaches  
consensus (typically this happens because technical people come to an  
agreement by discussing things, although e.g. on bikeshed issues it can  
be more painful and need to be done by some formal process), and produces  
a specification. It is not an assumption that people in this Working Group  
are also following WHATWG, and there are likely to be important  
contributors who are not doing so.


Where specs have been taken to WHATWG (e.g. XHR) the development is  
essentially independent, converging to the extent that the two communities  
come to the same conclusions.


You're welcome to work where and (within requirements that may be made by  
each community) how you want. But the effective way to provide your  
specification with the patent commitments made by W3C members is to do the  
work within the W3C process. W3C members make no commitment to specs that  
do not go through that process, and you should not assume that a spec  
developed elsewhere will automatically be considered - the quid pro quo of  
the patent commitments made by W3C members is a deliberate choice to take  
up any given work item.


Cheers

Chaals

--
Charles McCathie Nevile - Consultant (web standards) CTO Office, Yandex
  cha...@yandex-team.ru Find more at http://yandex.com



Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-04 Thread Aymeric Vitte

Yes for the different questions, I first mentioned it in [1]

That's not an invention of mine, node is doing this:

var handle=function(req,res) {
var raw=fs.createReadStream(file);
res.writeHead(200,head);

//and the magic comes here:

raw.pipe(zlib.createGzip()).pipe(res);

};

http.createServer(handle).listen(80,function() {});

If the concept is correct and applicable (with promises), as previously 
mentioned, a good example could be WebCrypto:


crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey, sourceStream).createStream()


So the WebCrypto API does not have to be modified and just need to 
support the createStream method, so a two lines change in the spec.


Regards,

Aymeric

[1] http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0593.html

Le 04/11/2013 00:41, Feras Moussa a écrit :

Streams instantiations somewhere make me think to the structured clone
algorithm, as I proposed before there should be a method like a
createStream so you just need to say for a given API that it supports
this method and you don't have to modify the API except for specific
cases (xhr,ws,etc), like for the structured clone algo, and this is missing.

This is an interesting idea. But I'm not entirely clear on your proposal. Is 
[1] where you mentioned it, or is there another thread I've missed?

You're not proposing changing the stream constructor, but rather also defining 
a generic way an API can add support for stream by implementing a 
strongly-defined createStream method?

Is your thinking to have this in order to give users a consistent way to obtain 
a stream from various APIs?
On first thought I like the idea, but I think once we settle on a definition of 
'Stream', we can asses what is really required for other APIs to begin 
supporting it. If so, I can create a bug to track this concept.

[1] http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0246.html




Date: Sun, 3 Nov 2013 23:16:12 +0100
From: vitteayme...@gmail.com
To: art.bars...@nokia.com
CC: public-webapps@w3.org
Subject: Re: CfC: publish WD of Streams API; deadline Nov 3

Yes, with good results, groups are throwing the ball to others... I
don't know right now all the groups that might need to be involved,
that's the reason of my question.

4 days out without internet connection, usually one email every two
weeks on the subject and suddendly tons of emails, looks like a
conspiracy...

I will reread the threads (still perplex about some issues, a txt stream
is a binary stream that should be piped to textEncoder/Decoder from my
standpoint, making it a special case just complicates everything, maybe
it's too late to revert this) but it looks like the consensus is to wait
for Domenic's proposal, OK but as I mentioned he missed some points in
the current proposal and it's interesting to read carefully the Overlap
thread, and I find it important to have a simple way to handle
ArrayBuffer, View, Blob without converting all the time.

Streams instantiations somewhere make me think to the structured clone
algorithm, as I proposed before there should be a method like a
createStream so you just need to say for a given API that it supports
this method and you don't have to modify the API except for specific
cases (xhr,ws,etc), like for the structured clone algo, and this is missing.

Regards

Aymeric

Le 03/11/2013 19:02, Arthur Barstow a écrit :

Hi Aymeric,

On 10/29/13 7:22 AM, ext Aymeric Vitte wrote:

Who is coordinating each group that should get involved?

I thought you agreed to do that ;).


MediaStream for example should be based on the Stream interface and
all related streams proposals.

More seriously though, this is good to know, and if there is
additional coordination that needs to be done, please let us know.

-Thanks, ArtB



--
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms




--
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms




RE: CfC: publish WD of Streams API; deadline Nov 3

2013-11-04 Thread Domenic Denicola
From: Arthur Barstow [mailto:art.bars...@nokia.com]

 Domenic - Mike Smith mentioned you have worked on a related spec. What is the 
 URL?

We are working on a streams specification which addresses the appropriate 
requirements at https://github.com/whatwg/streams.

It is still a work in progress, but the most important differences in approach 
and API can be seen. In particular, the extensive Requirements section details 
the problems a streaming API should solve; very few of them are solved by the 
draft this CfC was targeted at.

I will be continuing to work on it throughout the week, as time permits, to 
flesh out more of the ideas that are currently sketches or one-sentence 
summaries, and instead making them complete APIs.

 Also, are you interested and willing to work with Feras and Takeshi on a 
 joint/converged spec in the context of WebApps?

I welcome any input and help from Feras, Takeshi, or any others who wish to be 
involved. I am already getting great feedback and input from many quarters, 
including the Node.js community, the web developer community, a couple 
implementers, and a few editors of related specifications (such as the serial 
port API, the raw socket API, the XHR standard, and the service worker spec). 
Pull requests or discussion in the issue tracker would definitely be welcome, 
as there is much work left to do!

As for *where* the work is done, I will be working within the context of the 
WHATWG to produce this specification. My understanding is that usually the W3C 
picks some point in time to fork WHATWG specifications into W3C ones, changes 
some minor details (such as removing authorship information and changing the 
genders used in examples), then advancing it through the usual 
ED/WD/LCWD/CR/PR/REC track in order to get patent disclosure. I'm very 
interested in ensuring patent disclosure for the streams specification, so I 
hope someone takes on this work, but I do not think it would be a good use of 
my time to do so, as from what I understand there are people at the W3C who 
have this process down to an art.


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-04 Thread Aymeric Vitte
The main difference is that you are explaining in details what you are 
doing and the current draft does not, I will look more closely and 
comment, I am not sure the ES style really helps everybody understanding it.


Regards,

Aymeric

Le 04/11/2013 09:52, Domenic Denicola a écrit :

From: Arthur Barstow [mailto:art.bars...@nokia.com]


Domenic - Mike Smith mentioned you have worked on a related spec. What is the 
URL?

We are working on a streams specification which addresses the appropriate 
requirements at https://github.com/whatwg/streams.

It is still a work in progress, but the most important differences in approach 
and API can be seen. In particular, the extensive Requirements section details 
the problems a streaming API should solve; very few of them are solved by the 
draft this CfC was targeted at.

I will be continuing to work on it throughout the week, as time permits, to 
flesh out more of the ideas that are currently sketches or one-sentence 
summaries, and instead making them complete APIs.


Also, are you interested and willing to work with Feras and Takeshi on a 
joint/converged spec in the context of WebApps?

I welcome any input and help from Feras, Takeshi, or any others who wish to be 
involved. I am already getting great feedback and input from many quarters, 
including the Node.js community, the web developer community, a couple 
implementers, and a few editors of related specifications (such as the serial 
port API, the raw socket API, the XHR standard, and the service worker spec). 
Pull requests or discussion in the issue tracker would definitely be welcome, 
as there is much work left to do!

As for *where* the work is done, I will be working within the context of the 
WHATWG to produce this specification. My understanding is that usually the W3C 
picks some point in time to fork WHATWG specifications into W3C ones, changes 
some minor details (such as removing authorship information and changing the 
genders used in examples), then advancing it through the usual 
ED/WD/LCWD/CR/PR/REC track in order to get patent disclosure. I'm very 
interested in ensuring patent disclosure for the streams specification, so I 
hope someone takes on this work, but I do not think it would be a good use of 
my time to do so, as from what I understand there are people at the W3C who 
have this process down to an art.


--
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms




Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-03 Thread Arthur Barstow

On 11/1/13 8:52 AM, ext Arthur Barstow wrote:

On 10/31/13 11:52 PM, ext Feras Moussa wrote:
Agreed. Some of the points listed appear to be things already 
addressed. Takeshi and I have some feedback on the initial mail, but 
will wait and provide thoughts on the proposal instead. Looking 
forward to seeing it.




From: tyosh...@google.com
Date: Fri, 1 Nov 2013 12:18:47 +0900
To: ann...@annevk.nl
CC: dome...@domenicdenicola.com; art.bars...@nokia.com; 
public-webapps@w3.org

Subject: Re: publish WD of Streams API; deadline Nov 3

OK. There seems to be some disconnect, but I'm fine with waiting for
Domenic's proposal first.



Feras, Takeshi - for the sake of clarity, are you proposing this CfC 
be stopped? [FWIW, I don't feel strongly either way, although I 
suspect one could argue that getting some additional exposure and 
comments could be helpful.]


Feras and Takeshi indicated they would like this CfC to continue, 
especially since their merged specs include improvements over the FPWD 
published 14-Feb-2013.


If people have technical comments on the latest ED, please send them to 
public-webapps and/or file bugs [Bugs].


-Thanks, ArtB

[Bugs] 
https://www.w3.org/Bugs/Public/buglist.cgi?component=Streams%20APIlist_id=29495product=WebAppsWGresolution=---







Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-03 Thread Arthur Barstow

Hi Aymeric,

On 10/29/13 7:22 AM, ext Aymeric Vitte wrote:
Who is coordinating each group that should get involved? 


I thought you agreed to do that ;).

MediaStream for example should be based on the Stream interface and 
all related streams proposals.


More seriously though, this is good to know, and if there is additional 
coordination that needs to be done, please let us know.


-Thanks, ArtB





Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-03 Thread Aymeric Vitte
Yes, with good results, groups are throwing the ball to others... I 
don't know right now all the groups that might need to be involved, 
that's the reason of my question.


4 days out without internet connection, usually one email every two 
weeks on the subject and suddendly tons of emails, looks like a 
conspiracy...


I will reread the threads (still perplex about some issues, a txt stream 
is a binary stream that should be piped to textEncoder/Decoder from my 
standpoint, making it a special case just complicates everything, maybe 
it's too late to revert this) but it looks like the consensus is to wait 
for Domenic's proposal, OK but as I mentioned he missed some points in 
the current proposal and it's interesting to read carefully the Overlap 
thread, and I find it important to have a simple way to handle 
ArrayBuffer, View, Blob without converting all the time.


Streams instantiations somewhere make me think to the structured clone 
algorithm, as I proposed before there should be a method like a 
createStream so you just need to say for a given API that it supports 
this method and you don't have to modify the API except for specific 
cases (xhr,ws,etc), like for the structured clone algo, and this is missing.


Regards

Aymeric

Le 03/11/2013 19:02, Arthur Barstow a écrit :

Hi Aymeric,

On 10/29/13 7:22 AM, ext Aymeric Vitte wrote:
Who is coordinating each group that should get involved? 


I thought you agreed to do that ;).

MediaStream for example should be based on the Stream interface and 
all related streams proposals.


More seriously though, this is good to know, and if there is 
additional coordination that needs to be done, please let us know.


-Thanks, ArtB




--
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms




RE: CfC: publish WD of Streams API; deadline Nov 3

2013-11-03 Thread Feras Moussa
 Streams instantiations somewhere make me think to the structured clone
 algorithm, as I proposed before there should be a method like a
 createStream so you just need to say for a given API that it supports
 this method and you don't have to modify the API except for specific
 cases (xhr,ws,etc), like for the structured clone algo, and this is missing.

This is an interesting idea. But I'm not entirely clear on your proposal. Is 
[1] where you mentioned it, or is there another thread I've missed?

You're not proposing changing the stream constructor, but rather also defining 
a generic way an API can add support for stream by implementing a 
strongly-defined createStream method?

Is your thinking to have this in order to give users a consistent way to obtain 
a stream from various APIs? 
On first thought I like the idea, but I think once we settle on a definition of 
'Stream', we can asses what is really required for other APIs to begin 
supporting it. If so, I can create a bug to track this concept.

[1] http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0246.html



 Date: Sun, 3 Nov 2013 23:16:12 +0100
 From: vitteayme...@gmail.com
 To: art.bars...@nokia.com
 CC: public-webapps@w3.org
 Subject: Re: CfC: publish WD of Streams API; deadline Nov 3

 Yes, with good results, groups are throwing the ball to others... I
 don't know right now all the groups that might need to be involved,
 that's the reason of my question.

 4 days out without internet connection, usually one email every two
 weeks on the subject and suddendly tons of emails, looks like a
 conspiracy...

 I will reread the threads (still perplex about some issues, a txt stream
 is a binary stream that should be piped to textEncoder/Decoder from my
 standpoint, making it a special case just complicates everything, maybe
 it's too late to revert this) but it looks like the consensus is to wait
 for Domenic's proposal, OK but as I mentioned he missed some points in
 the current proposal and it's interesting to read carefully the Overlap
 thread, and I find it important to have a simple way to handle
 ArrayBuffer, View, Blob without converting all the time.

 Streams instantiations somewhere make me think to the structured clone
 algorithm, as I proposed before there should be a method like a
 createStream so you just need to say for a given API that it supports
 this method and you don't have to modify the API except for specific
 cases (xhr,ws,etc), like for the structured clone algo, and this is missing.

 Regards

 Aymeric

 Le 03/11/2013 19:02, Arthur Barstow a écrit :
 Hi Aymeric,

 On 10/29/13 7:22 AM, ext Aymeric Vitte wrote:
 Who is coordinating each group that should get involved?

 I thought you agreed to do that ;).

 MediaStream for example should be based on the Stream interface and
 all related streams proposals.

 More seriously though, this is good to know, and if there is
 additional coordination that needs to be done, please let us know.

 -Thanks, ArtB



 --
 Peersm : http://www.peersm.com
 node-Tor : https://www.github.com/Ayms/node-Tor
 GitHub : https://www.github.com/Ayms

 


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-01 Thread Vic99999
EventSources are missing too

On 29/10/13 10:22 PM, Aymeric Vitte wrote:
 I have suggested some additions/changes in my latest reply to the 
 Overlap thread.

 The list of streams producers/consumers is not final but obviously 
 WebSockets are missing.

 Who is coordinating each group that should get involved? MediaStream 
 for example should be based on the Stream interface and all related 
 streams proposals.

 Regards,

 Aymeric

 Le 28/10/2013 16:29, Arthur Barstow a écrit :
 Feras and Takeshi have begun merging their Streams proposal and this 
 is a Call for Consensus to publish a new WD of Streams API using the 
 updated ED as the basis:

 https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

 Please note the Editors may update the ED before the TR is published 
 (but they do not intend to make major changes during the CfC).

 Agreement to this proposal: a) indicates support for publishing a new 
 WD; and b) does not necessarily indicate support of the contents of 
 the WD.

 If you have any comments or concerns about this proposal, please 
 reply to this e-mail by November 3 at the latest. Positive response 
 to this CfC is preferred and encouraged and silence will be assumed 
 to mean agreement with the proposal.

 -Thanks, ArtB






Re: CfC: publish WD of Streams API; deadline Nov 3

2013-11-01 Thread Arthur Barstow

On 10/31/13 11:52 PM, ext Feras Moussa wrote:

Agreed. Some of the points listed appear to be things already addressed. 
Takeshi and I have some feedback on the initial mail, but will wait and provide 
thoughts on the proposal instead. Looking forward to seeing it.



From: tyosh...@google.com
Date: Fri, 1 Nov 2013 12:18:47 +0900
To: ann...@annevk.nl
CC: dome...@domenicdenicola.com; art.bars...@nokia.com; public-webapps@w3.org
Subject: Re: publish WD of Streams API; deadline Nov 3

OK. There seems to be some disconnect, but I'm fine with waiting for
Domenic's proposal first.



Feras, Takeshi - for the sake of clarity, are you proposing this CfC be 
stopped? [FWIW, I don't feel strongly either way, although I suspect one 
could argue that getting some additional exposure and comments could be 
helpful.]


Domenic - Mike Smith mentioned you have worked on a related spec. What 
is the URL? Also, are you interested and willing to work with Feras and 
Takeshi on a joint/converged spec in the context of WebApps?


-Thanks, AB





RE: Defining generic Stream than considering only bytes (Re: CfC: publish WD of Streams API; deadline Nov 3)

2013-10-31 Thread Feras Moussa
A few comments inline below -


 From: tyosh...@google.com 
 Date: Thu, 31 Oct 2013 13:23:26 +0900 
 To: d...@deanlandolt.com 
 CC: art.bars...@nokia.com; public-webapps@w3.org 
 Subject: Defining generic Stream than considering only bytes (Re: CfC: 
 publish WD of Streams API; deadline Nov 3) 
 
 Hi Dean, 
 
 On Thu, Oct 31, 2013 at 11:30 AM, Dean Landolt 
 d...@deanlandolt.commailto:d...@deanlandolt.com wrote: 
 I really like this general concepts of this proposal, but I'm confused 
 by what seems like an unnecessary limiting assumption: why assume all 
 streams are byte streams? This is a mistake node recently made in its 
 streams refactor that has led to an objectMode and added cruft. 
 
 Forgive me if this has been discussed -- I just learned of this today. 
 But as someone who's been slinging streams in javascript for years I'd 
 really hate to see the standard stream hampered by this bytes-only 
 limitation. The node ecosystem clearly demonstrates that streams are 
 for more than bytes and (byte-encoded) strings. 
 
 
 To glue Streams with existing binary handling infrastructure such as 
 ArrayBuffer, Blob, we should have some specialization for Stream 
 handling bytes rather than using generalized Stream that would 
 accept/output an array or single object of the type. Maybe we can 
 rename Streams API to ByteStream not to occupy the name Stream that 
 sounds like more generic, and start standardizing generic Stream. 

Dean, it sounds like your concern isnt just around the naming, but rather how 
data is read out of a stream. I've reviewed both the Node Streams and Buffer 
APIs previously, and from my understanding the data is provided as either a 
Buffer or String. This is on-par with ArrayBuffer/String. What data do you want 
to obtain that is missing, and for what scenario? Are these data types that 
already exist in the web platform, or new types you think are missing?

 
 In my perfect world any arbitrary iterator could be used to 
 characterize stream chunks -- this would have some really interesting 
 benefits -- but I suspect this kind of flexibility would be overkill 
 for now. But there's good no reason bytes should be the only thing 
 people can chunk up in streams. And if we're defining streams for the 
 whole platform they shouldn't just be tied to a few very specific 
 file-like use cases. 
 If streams could also consist of chunks of strings (real, native 
 strings) a huge swath of the API could disappear. All of readType, 
 readEncoding and charset could be eliminated, replaced with simple, 
 composable transforms that turn byte streams (of, say, utf-8) into 
 string streams. And vice versa. 
 
 
 So, for example, XHR would be the point of decoding and it returns a 
 Stream of DOMStrings? 
 
 Of course the real draw of this approach would be when chunks are 
 neither blobs nor strings. Why couldn't chunks be arrays? The arrays 
 could contain anything (no need to reserve any value as a sigil). 
 Regardless of the chunk type, the zero object for any given type 
 wouldn't be `null` (it would be something like '' or []). That means we 
 can use null to distinguish EOF, and `chunk == null` would make a 
 perfectly nice (and unambiguous) EOF sigil, eliminating yet more API 
 surface. This would give us a clean object mode streams for free, and 
 without node's arbitrary limitations. 
 
 For several reasons, I chose to use .eof than using null. One of them 
 is to allow the non-empty final chunk to signal EOF than requiring one 
 more read() call. 
 
 This point can be re-discussed. 

I thought EOF made sense here as well, but it's something that can be changed. 
Your proposal is interesting - is something like this currently implemented 
anywhere? This behavior feels like it'd require several changes elsewhere, 
since some APIs and libraries may explicitly look for an EOF.

 
 The `size` of an array stream would be the total length of all array 
 chunks. As I hinted before, we could also leave the door open to 
 specifying chunks as any iterable, where `size` (if known) would just 
 be the `length` of each chunk (assuming chunks even have a `length`). 
 This would also allow individual chunks to be built of generators, 
 which could be particularly interesting if the `size` argument to 
 `read` was specified as a maximum number of bytes rather than the total 
 to return -- completely sensible considering it has to behave this way 
 near the end of the stream anyway... 
 
 I don't really understand the last point. Could you please elaborate 
 the story and benefit? 
 
 IIRC, it's considered to be useful and important to be able to cut an 
 exact requested size of data into an ArrayBuffer object and get 
 notified (the returned Promise gets resolved) only when it's ready. 
 
 This would lead to a pattern like `stream.read(Infinity)`, which would 
 essentially say give me everything you've got soon as you can. 
 
 In current proposal, read() i.e. read

Re: Splitting Stream into InputStream and OutputStream (was Re: CfC: publish WD of Streams API; deadline Nov 3)

2013-10-31 Thread François REMY
Yes, having the InputStream and OutputStream interfaces would be great, and the 
“Stream” class could inherit from both. The important thing is that an external 
API can return either a readable or a writable stream, depending on what make 
sense for it.



Since JavaScript does not provide a way to check if an object implements an 
interface, there should probably exist a way to check that from the API, like:


   var stream = InputStream(s) // returns “s” if it’s an input stream, convert 
is into a stream if possible, or return null

   if(stream) { … } else { … }


That’s a great way to convert a string into a stream, for example, in the case 
of an API that requires an InputStream and must integrate with older code that 
returns a String or a Blob.









De : Takeshi Yoshino
Envoyé : ‎jeudi‎ ‎31‎ ‎octobre‎ ‎2013 ‎04‎:‎45
À : François REMY
Cc : Arthur Barstow, public-webapps@w3.org





Hi François



On Thu, Oct 31, 2013 at 6:16 AM, François REMY francois.remy@outlook.com 
wrote:


- Streams should exist in at least two fashions: InputStream and OutputStream. 
Both of them serve different purposes and, while some stream may actually be 
both, this remains an exceptional behavior worth being noted. Finally, a 
Stream is not equal to a InMemoryStream as the constructor may seem to 
indicate. A stream is a much lower-level concept, which may actually have 
nothing to do with InMemory operations.




Yes. I initially thought it'll be clearer to split in/out interface. E.g. a 
Stream obtained from XHR to receive a response should not be writable. It's 
reasonable to make network-to-Stream transfer happen in background 
asynchronously to JavaScript, and then it doesn't make much sense to keep it 
writable from JavaScript.




It has a unified IDL now but I'm designing write side and read side 
independently. We could decouple it into two separate IDLs? concepts? if it 
make sense. Stream would inherit from both and provides a constructor.

Re: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Anne van Kesteren
On Wed, Oct 30, 2013 at 6:04 PM, Domenic Denicola
dome...@domenicdenicola.com wrote:
 I have concrete suggestions as to what such an API could look like—and, more 
 importantly, how its semantics would significantly differ from this one—which 
 I hope to flesh out and share more broadly by the end of this weekend. 
 However, since the call for comments phase has commenced, I thought it 
 important to voice these objections as soon as possible.

Given how long we have been trying to figure out streams, waiting a
little longer to see Domenic's proposal should be fine I think. No
need to start rushing things through the process now. (Although on the
flipside at some point we will need to start shipping something.)


-- 
http://annevankesteren.nl/



Re: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Takeshi Yoshino
OK. There seems to be some disconnect, but I'm fine with waiting for
Domenic's proposal first.

Takeshi


On Thu, Oct 31, 2013 at 7:41 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Wed, Oct 30, 2013 at 6:04 PM, Domenic Denicola
 dome...@domenicdenicola.com wrote:
  I have concrete suggestions as to what such an API could look like—and,
 more importantly, how its semantics would significantly differ from this
 one—which I hope to flesh out and share more broadly by the end of this
 weekend. However, since the call for comments phase has commenced, I
 thought it important to voice these objections as soon as possible.

 Given how long we have been trying to figure out streams, waiting a
 little longer to see Domenic's proposal should be fine I think. No
 need to start rushing things through the process now. (Although on the
 flipside at some point we will need to start shipping something.)


 --
 http://annevankesteren.nl/




Re: Splitting Stream into InputStream and OutputStream (was Re: CfC: publish WD of Streams API; deadline Nov 3)

2013-10-31 Thread Takeshi Yoshino
On Thu, Oct 31, 2013 at 4:48 PM, François REMY 
francois.remy@outlook.com wrote:

  Since JavaScript does not provide a way to check if an object implements
 an interface, there should probably exist a way to check that from the API,
 like:


Basically it should be sufficient if each API can check type, but yeah
probably useful.


 var stream = InputStream(s) // returns “s” if it’s an input stream,
 convert is into a stream if possible, or return null
if(stream) { … } else { … }

 That’s a great way to convert a string into a stream, for example, in the
 case of an API that requires an InputStream and must integrate with older
 code that returns a String or a Blob.


Interesting. Maybe also accept an array of strings?


RE: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Feras Moussa
Agreed. Some of the points listed appear to be things already addressed. 
Takeshi and I have some feedback on the initial mail, but will wait and provide 
thoughts on the proposal instead. Looking forward to seeing it.


 From: tyosh...@google.com 
 Date: Fri, 1 Nov 2013 12:18:47 +0900 
 To: ann...@annevk.nl 
 CC: dome...@domenicdenicola.com; art.bars...@nokia.com; public-webapps@w3.org 
 Subject: Re: publish WD of Streams API; deadline Nov 3 
 
 OK. There seems to be some disconnect, but I'm fine with waiting for 
 Domenic's proposal first. 
 
 Takeshi 
 
 
 On Thu, Oct 31, 2013 at 7:41 PM, Anne van Kesteren 
 ann...@annevk.nlmailto:ann...@annevk.nl wrote: 
 On Wed, Oct 30, 2013 at 6:04 PM, Domenic Denicola 
 dome...@domenicdenicola.commailto:dome...@domenicdenicola.com wrote: 
 I have concrete suggestions as to what such an API could look 
 like—and, more importantly, how its semantics would significantly 
 differ from this one—which I hope to flesh out and share more broadly 
 by the end of this weekend. However, since the call for comments phase 
 has commenced, I thought it important to voice these objections as soon 
 as possible. 
 
 Given how long we have been trying to figure out streams, waiting a 
 little longer to see Domenic's proposal should be fine I think. No 
 need to start rushing things through the process now. (Although on the 
 flipside at some point we will need to start shipping something.) 
 
 
 -- 
 http://annevankesteren.nl/ 
 
 


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Rob Manson
Along with WebSockets as Aymeric mentioned...WebRTC DataChannels are 
also missing.


And I think Aymeric's point about MediaStream is important too...but 
there is very strong push-back from within the Media Capture  Streams 
TF that they don't think this is relevant 8/


Also, here's a couple of links for things I've shared/proposed recently 
related to this.


public message
http://lists.w3.org/Archives/Public/public-media-capture/2013Sep/0229.html

presentation
http://www.slideshare.net/robman/web-standards-for-ar-workshop-at-ismar13

code
https://github.com/buildar/getting_started_with_webrtc#image_processing_pipelinehtml 



All thoughts and feedback welcome.


roBman


On 29/10/13 10:22 PM, Aymeric Vitte wrote:
I have suggested some additions/changes in my latest reply to the 
Overlap thread.


The list of streams producers/consumers is not final but obviously 
WebSockets are missing.


Who is coordinating each group that should get involved? MediaStream 
for example should be based on the Stream interface and all related 
streams proposals.


Regards,

Aymeric

Le 28/10/2013 16:29, Arthur Barstow a écrit :
Feras and Takeshi have begun merging their Streams proposal and this 
is a Call for Consensus to publish a new WD of Streams API using the 
updated ED as the basis:


https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Please note the Editors may update the ED before the TR is published 
(but they do not intend to make major changes during the CfC).


Agreement to this proposal: a) indicates support for publishing a new 
WD; and b) does not necessarily indicate support of the contents of 
the WD.


If you have any comments or concerns about this proposal, please 
reply to this e-mail by November 3 at the latest. Positive response 
to this CfC is preferred and encouraged and silence will be assumed 
to mean agreement with the proposal.


-Thanks, ArtB








RE: CfC: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Feras Moussa
Yes, WebSockets was missing - I've gone ahead and updated the spec to include 
it.

Thanks for sharing the links, the content is well thought out. In particular, 
your diagram does a good job summarizing some of the key consumers and 
producers that come to play regarding Streams. I'll review it in detail.

DataChannels also seem like a possible candidate, although I'm not yet very 
familiar with them. This can be something reviewed and thought through, and 
added accordingly.

 Who is coordinating each group that should get involved? MediaStream for 
example should be based on the Stream interface and all related streams 
proposals.

Once we come to a consensus in the WG what Streams look like and their role, we 
can begin to coordinate what the impact is on other groups.

Thanks,
Feras



 Date: Fri, 1 Nov 2013 16:05:22 +1100
 From: rob...@mob-labs.com
 To: public-webapps@w3.org
 Subject: Re: CfC: publish WD of Streams API; deadline Nov 3

 Along with WebSockets as Aymeric mentioned...WebRTC DataChannels are
 also missing.

 And I think Aymeric's point about MediaStream is important too...but
 there is very strong push-back from within the Media Capture  Streams
 TF that they don't think this is relevant 8/

 Also, here's a couple of links for things I've shared/proposed recently
 related to this.

 public message
 http://lists.w3.org/Archives/Public/public-media-capture/2013Sep/0229.html

 presentation
 http://www.slideshare.net/robman/web-standards-for-ar-workshop-at-ismar13

 code
 https://github.com/buildar/getting_started_with_webrtc#image_processing_pipelinehtml


 All thoughts and feedback welcome.


 roBman


 On 29/10/13 10:22 PM, Aymeric Vitte wrote:
 I have suggested some additions/changes in my latest reply to the
 Overlap thread.

 The list of streams producers/consumers is not final but obviously
 WebSockets are missing.

 Who is coordinating each group that should get involved? MediaStream
 for example should be based on the Stream interface and all related
 streams proposals.

 Regards,

 Aymeric

 Le 28/10/2013 16:29, Arthur Barstow a écrit :
 Feras and Takeshi have begun merging their Streams proposal and this
 is a Call for Consensus to publish a new WD of Streams API using the
 updated ED as the basis:

 https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

 Please note the Editors may update the ED before the TR is published
 (but they do not intend to make major changes during the CfC).

 Agreement to this proposal: a) indicates support for publishing a new
 WD; and b) does not necessarily indicate support of the contents of
 the WD.

 If you have any comments or concerns about this proposal, please
 reply to this e-mail by November 3 at the latest. Positive response
 to this CfC is preferred and encouraged and silence will be assumed
 to mean agreement with the proposal.

 -Thanks, ArtB



 


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-10-31 Thread Rob Manson

Sounds good.

Thanks.

roBman


On 1/11/13 4:43 PM, Feras Moussa wrote:

Yes, WebSockets was missing - I've gone ahead and updated the spec to include 
it.

Thanks for sharing the links, the content is well thought out. In particular, 
your diagram does a good job summarizing some of the key consumers and 
producers that come to play regarding Streams. I'll review it in detail.

DataChannels also seem like a possible candidate, although I'm not yet very 
familiar with them. This can be something reviewed and thought through, and 
added accordingly.


  Who is coordinating each group that should get involved? MediaStream for 
example should be based on the Stream interface and all related streams 
proposals.

Once we come to a consensus in the WG what Streams look like and their role, we 
can begin to coordinate what the impact is on other groups.

Thanks,
Feras




Date: Fri, 1 Nov 2013 16:05:22 +1100
From: rob...@mob-labs.com
To: public-webapps@w3.org
Subject: Re: CfC: publish WD of Streams API; deadline Nov 3

Along with WebSockets as Aymeric mentioned...WebRTC DataChannels are
also missing.

And I think Aymeric's point about MediaStream is important too...but
there is very strong push-back from within the Media Capture  Streams
TF that they don't think this is relevant 8/

Also, here's a couple of links for things I've shared/proposed recently
related to this.

public message
http://lists.w3.org/Archives/Public/public-media-capture/2013Sep/0229.html

presentation
http://www.slideshare.net/robman/web-standards-for-ar-workshop-at-ismar13

code
https://github.com/buildar/getting_started_with_webrtc#image_processing_pipelinehtml


All thoughts and feedback welcome.


roBman


On 29/10/13 10:22 PM, Aymeric Vitte wrote:

I have suggested some additions/changes in my latest reply to the
Overlap thread.

The list of streams producers/consumers is not final but obviously
WebSockets are missing.

Who is coordinating each group that should get involved? MediaStream
for example should be based on the Stream interface and all related
streams proposals.

Regards,

Aymeric

Le 28/10/2013 16:29, Arthur Barstow a écrit :

Feras and Takeshi have begun merging their Streams proposal and this
is a Call for Consensus to publish a new WD of Streams API using the
updated ED as the basis:

https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Please note the Editors may update the ED before the TR is published
(but they do not intend to make major changes during the CfC).

Agreement to this proposal: a) indicates support for publishing a new
WD; and b) does not necessarily indicate support of the contents of
the WD.

If you have any comments or concerns about this proposal, please
reply to this e-mail by November 3 at the latest. Positive response
to this CfC is preferred and encouraged and silence will be assumed
to mean agreement with the proposal.

-Thanks, ArtB








RE: publish WD of Streams API; deadline Nov 3

2013-10-30 Thread Domenic Denicola
From: Arthur Barstow [mailto:art.bars...@nokia.com]

 If you have any comments or concerns about this proposal, please reply to
 this e-mail by November 3 at the latest.

I have some concerns about this proposal, and do not think it is solving the 
problem at hand in an appropriate fashion. I believe it does not provide the 
appropriate primitives the web platform needs for a streams API. Most 
seriously, I think it has ignored the majority of lessons learned from existing 
JavaScript streaming APIs.

Here are specific critiques, in ascending order of importance.

- It has a read(n) interface, which is not valuable [1] but constraints the API 
in several awkward ways.

- It assumes MIME types are at all relevant to a streaming data abstraction, 
when this is not at all the case.

- In general, is far too backward-looking in attempting to integrate things 
like blobs or object URLs into what is supposed to be a forward-looking 
primitive for the future of the extensible web. Replacing these various 
disparate concepts is what developers want from streams [2].

- It conflates text streams and binary data. As outlined in previous messages 
[1], what type of data the stream contains *must* be an *immutable* property of 
the stream. In contrast, the proposed API actively encourage mixing multiple 
data types within a stream via readType, readEncoding, and the overloaded write 
method.

- It conflates readable and writable streams, which prevents a whole class of 
abstractions and use cases like: read-only file streams; write-only HTTP 
requests; and duplex streams which read to one channel and write to another 
(e.g. a websocket, where writing pushes data to the server and reading reads 
data from the server---not the written data, but data that the server writes to 
you). Indeed, the only use case this proposal supports is transform streams, 
which take data in one end and output new data on the other end.

- It provides no mechanism for backpressure signaling to readable stream data 
sources or from writable stream sinks. As we have heard previously, any stream 
API that does not treat backpressure as a primary issue is not a stream API at 
all. [3]

- More generally, it does not operate at the correct level of abstraction, 
which should be close to the I/O primitives streams are meant to expose. This 
is evident in the general lack of APIs  for handing interaction with and 
signaling to underlying I/O sources or sinks.

- It's pipe mechanism is poorly thought out, and does not build on top of the 
existing primitives; indeed, it seems to install some kind of mutex lock that 
prevents the other primitives from being used until the pipe is complete. The 
primitives do not support multiple consumers, so the pipe mechanism handles 
that case in an ad-hoc way. It is not composable into chains in the fashion of 
traditional stream piping. Its lack of backpressure support prevents expression 
of key use cases such as piping a fast data connection (e.g. disk) to a slow 
data connection (e.g. push to a slow mobile device); piping a slow connection 
to a fast one; piping through encoders/compressers/decoders/decompressors; and 
so on. In general, it appears to take no inspiration from prior art, which is a 
shame since existing stream implementations all agree on how pipe should work. 

In light of these critiques, I feel that this API is not worth pursuing and 
should not proceed to Working Draft status. If we are to bring streaming data 
to the web platform, we should instead do it correctly, learning the lessons of 
the JavaScript stream APIs that came before us, and provide a powerful 
primitive that gives developers what they have asked for and serves as 
something we can layer the web's many streaming I/O interfaces on top of.

I have concrete suggestions as to what such an API could look like—and, more 
importantly, how its semantics would significantly differ from this one—which I 
hope to flesh out and share more broadly by the end of this weekend. However, 
since the call for comments phase has commenced, I thought it important to 
voice these objections as soon as possible.

Thanks,
-Domenic

[1]: http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0355.html
[2]: http://imgur.com/a/9vFGa#11
[3]: http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0275.html


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-10-30 Thread François REMY

| If you have any comments or concerns about this proposal, please reply
| to this e-mail by November 3 at the latest.

While adding streams to the platform seems a good idea to me, I've a few 
concern with this proposal.



My biggest concerns are articulated over two issues:

- Streams should exist in at least two fashions: InputStream and 
OutputStream. Both of them serve different purposes and, while some stream 
may actually be both, this remains an exceptional behavior worth being 
noted. Finally, a Stream is not equal to a InMemoryStream as the 
constructor may seem to indicate. A stream is a much lower-level concept, 
which may actually have nothing to do with InMemory operations.


- Secondly, the Stream interface is mixing the Stream and the 
StreamReader/StreamWriter concepts. I do not have a problem, if this is done 
properly, to mix the two concepts (aka most applications will want to use a 
StreamReader/StreamWriter anyway) but the current incarnation is not 
powerfull enough to be really useful, while still managing to be confusing.



As an actionable advice to the authors of the spec, I would recommend to 
have a look to the Stream API of other modern languages and how those API 
evolved over time. Streams exist for a very long time, it would be very 
unfortunate to repeat on the web platform the mistakes already made and 
fixed the hard way in competing platforms.





| Agreement to this proposal: a) indicates support for publishing a new
| WD; and b) does not necessarily indicate support of the contents of the 
WD.


Then, I agree to publish a new Working Draft, but this draft will need much 
futher refinement before being a w3c-recommendable specification. 





Re: publish WD of Streams API; deadline Nov 3

2013-10-30 Thread Aymeric Vitte
As you mention streams are existing since the begining of times and it's 
incredible that this does not exist on a web platform, but apparently 
the subject is still not so easy, node changed its stream quite a lot of 
time.


I probably will not be able to answer back in the coming days but your 
judgement is tough, this API is the concatenation of thoughts of the 
Overlap thread. Indeed it must clarify the writable/readable aspects, as 
well as the flow control/congestion, moreover than, despite of what you 
are saying, the API does support multiple consumers, and I/O, it's close 
to node streams.


Of course you must conflate text and binary, you can not spend your 
time converting to text, to ArrayBuffer, to ArrayBufferView, to Blob, 
the user API knows what it is streaming.


Regards,

Aymeric




Le 30/10/2013 19:04, Domenic Denicola a écrit :

From: Arthur Barstow [mailto:art.bars...@nokia.com]


If you have any comments or concerns about this proposal, please reply to
this e-mail by November 3 at the latest.

I have some concerns about this proposal, and do not think it is solving the 
problem at hand in an appropriate fashion. I believe it does not provide the 
appropriate primitives the web platform needs for a streams API. Most 
seriously, I think it has ignored the majority of lessons learned from existing 
JavaScript streaming APIs.

Here are specific critiques, in ascending order of importance.

- It has a read(n) interface, which is not valuable [1] but constraints the API 
in several awkward ways.

- It assumes MIME types are at all relevant to a streaming data abstraction, 
when this is not at all the case.

- In general, is far too backward-looking in attempting to integrate things 
like blobs or object URLs into what is supposed to be a forward-looking 
primitive for the future of the extensible web. Replacing these various 
disparate concepts is what developers want from streams [2].

- It conflates text streams and binary data. As outlined in previous messages 
[1], what type of data the stream contains *must* be an *immutable* property of 
the stream. In contrast, the proposed API actively encourage mixing multiple 
data types within a stream via readType, readEncoding, and the overloaded write 
method.

- It conflates readable and writable streams, which prevents a whole class of 
abstractions and use cases like: read-only file streams; write-only HTTP 
requests; and duplex streams which read to one channel and write to another 
(e.g. a websocket, where writing pushes data to the server and reading reads 
data from the server---not the written data, but data that the server writes to 
you). Indeed, the only use case this proposal supports is transform streams, 
which take data in one end and output new data on the other end.

- It provides no mechanism for backpressure signaling to readable stream data 
sources or from writable stream sinks. As we have heard previously, any stream 
API that does not treat backpressure as a primary issue is not a stream API at 
all. [3]

- More generally, it does not operate at the correct level of abstraction, 
which should be close to the I/O primitives streams are meant to expose. This 
is evident in the general lack of APIs  for handing interaction with and 
signaling to underlying I/O sources or sinks.

- It's pipe mechanism is poorly thought out, and does not build on top of the 
existing primitives; indeed, it seems to install some kind of mutex lock that 
prevents the other primitives from being used until the pipe is complete. The 
primitives do not support multiple consumers, so the pipe mechanism handles 
that case in an ad-hoc way. It is not composable into chains in the fashion of 
traditional stream piping. Its lack of backpressure support prevents expression 
of key use cases such as piping a fast data connection (e.g. disk) to a slow 
data connection (e.g. push to a slow mobile device); piping a slow connection 
to a fast one; piping through encoders/compressers/decoders/decompressors; and 
so on. In general, it appears to take no inspiration from prior art, which is a 
shame since existing stream implementations all agree on how pipe should work.

In light of these critiques, I feel that this API is not worth pursuing and 
should not proceed to Working Draft status. If we are to bring streaming data 
to the web platform, we should instead do it correctly, learning the lessons of 
the JavaScript stream APIs that came before us, and provide a powerful 
primitive that gives developers what they have asked for and serves as 
something we can layer the web's many streaming I/O interfaces on top of.

I have concrete suggestions as to what such an API could look like—and, more 
importantly, how its semantics would significantly differ from this one—which I 
hope to flesh out and share more broadly by the end of this weekend. However, 
since the call for comments phase has commenced, I thought it important to 
voice these objections as soon 

Re: CfC: publish WD of Streams API; deadline Nov 3

2013-10-30 Thread Dean Landolt
I really like this general concepts of this proposal, but I'm confused by
what seems like an unnecessary limiting assumption: why assume all streams
are byte streams? This is a mistake node recently made in its streams
refactor that has led to an objectMode and added cruft.

Forgive me if this has been discussed -- I just learned of this today. But
as someone who's been slinging streams in javascript for years I'd really
hate to see the standard stream hampered by this bytes-only limitation.
The node ecosystem clearly demonstrates that streams are for more than
bytes and (byte-encoded) strings.

In my perfect world any arbitrary iterator could be used to characterize
stream chunks -- this would have some really interesting benefits -- but I
suspect this kind of flexibility would be overkill for now. But there's
good no reason bytes should be the only thing people can chunk up in
streams. And if we're defining streams for the whole platform they
shouldn't *just* be tied to a few very specific file-like use cases.

If streams could also consist of chunks of strings (real, native strings) a
huge swath of the API could disappear. All of readType, readEncoding and
charset could be eliminated, replaced with simple, composable transforms
that turn byte streams (of, say, utf-8) into string streams. And vice versa.

The `size` of a stream (if it exists) would be specified as the total
`length` of all chunks concatenated together. So if chunks were in bytes,
`size` would be the total bytes (as currently specified). But if chunks
consisted of real strings, `size` would be the total length of all string
chunks. Interestingly, if your source stream is in utf-8 the total bytes
wouldn't be meaningful, and the total string size couldn't be known without
iterating the whole stream. But if the source stream is utf-16 and the
`size` is known, the new `size` could also be known ahead of time -- `bytes
/ 2` (thanks to javascript's ucs-2 strings).

Of course the real draw of this approach would be when chunks are neither
blobs nor strings. Why couldn't chunks be arrays? The arrays could contain
anything (no need to reserve any value as a sigil). Regardless of the chunk
type, the zero object for any given type wouldn't be `null` (it would be
something like '' or []). That means we can use null to distinguish EOF,
and `chunk == null` would make a perfectly nice (and unambiguous) EOF
sigil, eliminating yet more API surface. This would give us a clean object
mode streams for free, and without node's arbitrary limitations.

The `size` of an array stream would be the total length of all array
chunks. As I hinted before, we could also leave the door open to specifying
chunks as any iterable, where `size` (if known) would just be the `length`
of each chunk (assuming chunks even have a `length`). This would also allow
individual chunks to be built of generators, which could be particularly
interesting if the `size` argument to `read` was specified as a maximum
number of bytes rather than the total to return -- completely sensible
considering it has to behave this way near the end of the stream anyway...

This would lead to a pattern like `stream.read(Infinity)`, which would
essentially say *give me everything you've got soon as you can*. This is
closer to node's semantics (where read is async, for added scheduling
flexibility). It would drain streams faster rather than pseudo-blocking for
a specific (and arbitrary) size chunk which ultimately can't be guaranteed
anyway, so you'll always have to do length checks.

(On a somewhat related note: why is a 0-sized stream specified to throw?
And why a SyntaxError of all things? A 0-sized stream seems perfectly
reasonable to me.)

What's particularly appealing to me about the chunk-as-generator idea is
that these chunks could still be quite large -- hundreds megabytes, even.
Just because a potentially large amount of data has become available since
the last chunk was processed doesn't mean you should have to bring it all
into memory at once.

I know this is a long email and it may sound like a lot of suggestions, but
I think it's actually a relatively minor tweak (and simplification) that
would unlock the real power of streams for their many other use cases. I've
been thinking about streams and promises (and streams with promises) for
years now, and this is the first approach that really feels right to me.






On Mon, Oct 28, 2013 at 11:29 AM, Arthur Barstow art.bars...@nokia.comwrote:

 Feras and Takeshi have begun merging their Streams proposal and this is a
 Call for Consensus to publish a new WD of Streams API using the updated ED
 as the basis:

 https://dvcs.w3.org/hg/**streams-api/raw-file/tip/**Overview.htmhttps://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm
 

 Please note the Editors may update the ED before the TR is published (but
 they do not intend to make major changes during the CfC).

 Agreement to this proposal: a) indicates support for publishing a new WD;
 and b) 

Splitting Stream into InputStream and OutputStream (was Re: CfC: publish WD of Streams API; deadline Nov 3)

2013-10-30 Thread Takeshi Yoshino
Hi François

On Thu, Oct 31, 2013 at 6:16 AM, François REMY 
francois.remy@outlook.com wrote:

 - Streams should exist in at least two fashions: InputStream and
 OutputStream. Both of them serve different purposes and, while some stream
 may actually be both, this remains an exceptional behavior worth being
 noted. Finally, a Stream is not equal to a InMemoryStream as the
 constructor may seem to indicate. A stream is a much lower-level concept,
 which may actually have nothing to do with InMemory operations.


Yes. I initially thought it'll be clearer to split in/out interface. E.g. a
Stream obtained from XHR to receive a response should not be writable. It's
reasonable to make network-to-Stream transfer happen in background
asynchronously to JavaScript, and then it doesn't make much sense to keep
it writable from JavaScript.

It has a unified IDL now but I'm designing write side and read side
independently. We could decouple it into two separate IDLs? concepts? if it
make sense. Stream would inherit from both and provides a constructor.


Defining generic Stream than considering only bytes (Re: CfC: publish WD of Streams API; deadline Nov 3)

2013-10-30 Thread Takeshi Yoshino
Hi Dean,

On Thu, Oct 31, 2013 at 11:30 AM, Dean Landolt d...@deanlandolt.com wrote:

 I really like this general concepts of this proposal, but I'm confused by
 what seems like an unnecessary limiting assumption: why assume all streams
 are byte streams? This is a mistake node recently made in its streams
 refactor that has led to an objectMode and added cruft.

 Forgive me if this has been discussed -- I just learned of this today. But
 as someone who's been slinging streams in javascript for years I'd really
 hate to see the standard stream hampered by this bytes-only limitation.
 The node ecosystem clearly demonstrates that streams are for more than
 bytes and (byte-encoded) strings.


To glue Streams with existing binary handling infrastructure such as
ArrayBuffer, Blob, we should have some specialization for Stream handling
bytes rather than using generalized Stream that would accept/output an
array or single object of the type. Maybe we can rename Streams API to
ByteStream not to occupy the name Stream that sounds like more generic, and
start standardizing generic Stream.


 In my perfect world any arbitrary iterator could be used to characterize
 stream chunks -- this would have some really interesting benefits -- but I
 suspect this kind of flexibility would be overkill for now. But there's
 good no reason bytes should be the only thing people can chunk up in
 streams. And if we're defining streams for the whole platform they
 shouldn't *just* be tied to a few very specific file-like use cases.

 If streams could also consist of chunks of strings (real, native strings)
 a huge swath of the API could disappear. All of readType, readEncoding and
 charset could be eliminated, replaced with simple, composable transforms
 that turn byte streams (of, say, utf-8) into string streams. And vice versa.


So, for example, XHR would be the point of decoding and it returns a Stream
of DOMStrings?


 Of course the real draw of this approach would be when chunks are neither
 blobs nor strings. Why couldn't chunks be arrays? The arrays could contain
 anything (no need to reserve any value as a sigil). Regardless of the chunk
 type, the zero object for any given type wouldn't be `null` (it would be
 something like '' or []). That means we can use null to distinguish EOF,
 and `chunk == null` would make a perfectly nice (and unambiguous) EOF
 sigil, eliminating yet more API surface. This would give us a clean object
 mode streams for free, and without node's arbitrary limitations.


For several reasons, I chose to use .eof than using null. One of them is to
allow the non-empty final chunk to signal EOF than requiring one more
read() call.

This point can be re-discussed.


 The `size` of an array stream would be the total length of all array
 chunks. As I hinted before, we could also leave the door open to specifying
 chunks as any iterable, where `size` (if known) would just be the `length`
 of each chunk (assuming chunks even have a `length`). This would also allow
 individual chunks to be built of generators, which could be particularly
 interesting if the `size` argument to `read` was specified as a maximum
 number of bytes rather than the total to return -- completely sensible
 considering it has to behave this way near the end of the stream anyway...


I don't really understand the last point. Could you please elaborate the
story and benefit?

IIRC, it's considered to be useful and important to be able to cut an exact
requested size of data into an ArrayBuffer object and get notified (the
returned Promise gets resolved) only when it's ready.


 This would lead to a pattern like `stream.read(Infinity)`, which would
 essentially say *give me everything you've got soon as you can*.


In current proposal, read() i.e. read() with no argument does this.


  This is closer to node's semantics (where read is async, for added
 scheduling flexibility). It would drain streams faster rather than
 pseudo-blocking for a specific (and arbitrary) size chunk which ultimately
 can't be guaranteed anyway, so you'll always have to do length checks.

 (On a somewhat related note: why is a 0-sized stream specified to throw?
 And why a SyntaxError of all things? A 0-sized stream seems perfectly
 reasonable to me.)


0-sized Stream is not prohibited.

Do you mean 0-sized read()/pipe()/skip()? I don't think they make much
sense. It's useful only when you want to sense EOF and it can be done by
read(1).


 What's particularly appealing to me about the chunk-as-generator idea is
 that these chunks could still be quite large -- hundreds megabytes, even.
 Just because a potentially large amount of data has become available since
 the last chunk was processed doesn't mean you should have to bring it all
 into memory at once.


It's interesting. Could you please list some concrete example of such a
generator?


Re: CfC: publish WD of Streams API; deadline Nov 3

2013-10-29 Thread Aymeric Vitte
I have suggested some additions/changes in my latest reply to the 
Overlap thread.


The list of streams producers/consumers is not final but obviously 
WebSockets are missing.


Who is coordinating each group that should get involved? MediaStream for 
example should be based on the Stream interface and all related streams 
proposals.


Regards,

Aymeric

Le 28/10/2013 16:29, Arthur Barstow a écrit :
Feras and Takeshi have begun merging their Streams proposal and this 
is a Call for Consensus to publish a new WD of Streams API using the 
updated ED as the basis:


https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Please note the Editors may update the ED before the TR is published 
(but they do not intend to make major changes during the CfC).


Agreement to this proposal: a) indicates support for publishing a new 
WD; and b) does not necessarily indicate support of the contents of 
the WD.


If you have any comments or concerns about this proposal, please reply 
to this e-mail by November 3 at the latest. Positive response to this 
CfC is preferred and encouraged and silence will be assumed to mean 
agreement with the proposal.


-Thanks, ArtB



--
Peersm : http://www.peersm.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms




CfC: publish WD of Streams API; deadline Nov 3

2013-10-28 Thread Arthur Barstow
Feras and Takeshi have begun merging their Streams proposal and this is 
a Call for Consensus to publish a new WD of Streams API using the 
updated ED as the basis:


https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Please note the Editors may update the ED before the TR is published 
(but they do not intend to make major changes during the CfC).


Agreement to this proposal: a) indicates support for publishing a new 
WD; and b) does not necessarily indicate support of the contents of the WD.


If you have any comments or concerns about this proposal, please reply 
to this e-mail by November 3 at the latest. Positive response to this 
CfC is preferred and encouraged and silence will be assumed to mean 
agreement with the proposal.


-Thanks, ArtB