Re: [websockets] Making optional extensions mandatory in the API (was RE: Getting WebSockets API to Last Call)

[Takeshi Yoshino]

Resending. I've made mistake in processing public-webapps@w3.org confirmation
page.

On Fri, Jul 22, 2011 at 19:45, Takeshi Yoshino tyosh...@google.com wrote:

 On Fri, Jul 22, 2011 at 18:54, Anne van Kesteren ann...@opera.com wrote:

 On Fri, 22 Jul 2011 07:03:34 +0200, Takeshi Yoshino tyosh...@google.com
 wrote:

 Think of these facts, the only dominant implementation concern I can
 come up with for WebSocket compression is that big service provides may take
 an aggressive policy that they require users to use only WebSocket client
 with compression capability to reduce bandwidth. As a result clients with no
 compression capability may, in effect, be kicked out of WebSocket world.

 I can understand that concern. Is this true for the HTTP world? I.e.
 clients that send Accept-Encoding: \r\n cannot live?


 Yes. Optional features for browsers do not work. For servers it is fine if
 they do not support compression or whatnot, but browsers pretty much need to
 align on feature set.


 In summary, your point seems to be that we must choose feature set that is
 considered to be taken by majority as optimal like HTTP/gzip, and ask
 browsers to support that by specifying it the W3C spec (*). I see that. It
 might make sense. However deflate-stream is not yet considered as the
 optimal choice in the HyBi WG and we're trying to introduce better one. Even
 some are doubting if it's worth using deflate-stream compared to identity
 stream.

 Requiring all browsers request (== implement) deflate-stream can be asking
 everyone to do thrown-away work as a result. Is this acceptable?

 Based on W3C's stance (*) and IETF's stance, the only landing point is
 initially enforcing browsers to use identify encoding except for
 experimenting new compression, and when IETF provides new compression
 extension good enough to recommend, update the API spec to switch to that.

 Takeshi

Re: [websockets] Making optional extensions mandatory in the API (was RE: Getting WebSockets API to Last Call)

[Takeshi Yoshino]

Is new XHR spec going to make gzip mandatory for its underlying HTTP?

On Tue, Jul 26, 2011 at 06:05, Aryeh Gregor simetrical+...@gmail.comwrote:

 From the discussion here, it sounds like there are problems with
 WebSockets compression as currently defined.  If that's the case, it
 might be better for the IETF to just drop it from the protocol for now
 and leave it for a future version, but that's up to them.  As far as
 we're concerned, if the option is really a bad idea to start with, it
 might make sense for us to prohibit it rather than require it, but
 there's no reason at all we have to leave it optional for web browsers
 just because it's optional for other WebSockets implementations.


Regarding deflate-stream, I think prohibiting is better than requiring.

But I still don't understand the benefit of banning any extension other than
what specified in the API spec.

There are two different assertions made by W3C side.

(a) it's not acceptable to make support (== request) of good-compression
optional
(b) it's not acceptable to allow any other compression/extension than
specified in the API spec

(a) is supported by the discussion you and Anne made by using analogy with
HTTP. I may agree with this. (b) is what Hixie was asserting in the bug
entry. I'd like to see clear support for (b).

No one knows what kind of compression will be finally the win for WebSocket.
I'd like to see ideas about how the evolution of WebSocket will be like.
With (b), to experimentally implement some extension/compression not
specified in the API spec, we have to make our browser non-compliant with
W3C spec.

I'd suggest that, once better-deflate is ready by IETF, W3C uses text like 
the user agent MUST request better-deflate extension instead of JUST
better-deflate extension.

Thanks,
Takeshi

Re: [websockets] Making optional extensions mandatory in the API (was RE: Getting WebSockets API to Last Call)

[Takeshi Yoshino]

On Thu, Jul 28, 2011 at 00:06, Anne van Kesteren ann...@opera.com wrote:

 On Wed, 27 Jul 2011 03:35:03 -0700, Takeshi Yoshino tyosh...@google.com
 wrote:

 Is new XHR spec going to make gzip mandatory for its underlying HTTP?


 I do not think that analogy makes sense. The WebSocket protocol can only be
 used through the WebSocket API, HTTP is prevalent in more places.


What do you mean by more places?


 Having said that, XMLHttpRequest does place constraints on HTTP. E.g. it
 requires redirects to be followed, it does not expose 1xx responses, only
 works cross-origin in combination with CORS, etc.


I agree that there're some constrains must be placed on underlying protocol
to make it useful/secure on browser.

Re: [websockets] Making optional extensions mandatory in the API (was RE: Getting WebSockets API to Last Call)

[Takeshi Yoshino]

On Thu, Jul 28, 2011 at 02:03, Anne van Kesteren ann...@opera.com wrote:

 On Wed, 27 Jul 2011 08:49:57 -0700, Takeshi Yoshino tyosh...@google.com
 wrote:

 What do you mean by more places?


 XMLHttpRequest is not the sole API for HTTP, there's also a, form, etc.
 So indicating what parts of the HTTP protocol are mandatory for browsers is
 not really in scope for XMLHttpRequest.


So, let me correct my text by s/XHR/HTML5 http://www.w3.org/TR/html5//.

Re: Proposal: add websocket close codes for server not found and/or too many websockets open

[Takeshi Yoshino]

The protocol spec has defined 1015, but I think we should not pass through
it to the WebSocket API.
http://lists.w3.org/Archives/Public/public-webapps/2011OctDec/0437.html

I think 1006 is the right code for all of WebSocket handshake failure, TLS
failure and TCP connection failure. If the language in
http://tools.ietf.org/html/rfc6455#section-7.4.1 is not good, we can add
cannot be opened or before closed abnormally for clarification.

Chrome's onerror issue will be fixed soon. We agree with Simon that it's
clearly specified in http://tools.ietf.org/html/rfc6455#section-4.1. Thanks
for reporting Jason.
http://code.google.com/p/chromium/issues/detail?id=128057

Re: RfC: LCWD of WebSocket API; deadline August 30

[Takeshi Yoshino]

No technical comments.

A few editorial comments.

 CLOSING (numeric value 2)
 The connection is going through the closing handshake.

The readyState can enter CLOSING also when close() is called before
establishment. In that case, it's not going through closing handshake.

  // networking
   attribute EventHandleronopen;

insert an SP between EventHandler and onopen
(already fixed on the editor's draft)

 When the user agent validates the server's response during the establish
a WebSocket connection algorithm, if the status code received from the
server is not 101 (e.g. it is a redirect), the user agent must fail the
websocket connection.

websocket - WebSocket

 If the user agent was required to fail the websocket connection or the
WebSocket connection is closed with prejudice, fire a simple event named
error at the WebSocket object. [WSP]

websocket - WebSocket

 interface CloseEvent : Event {
   readonly attribute boolean wasClean;
   readonly attribute unsigned short code;
   readonly attribute DOMString reason;
 };

missing anchor on reason to its description

Overlap between StreamReader and FileReader

[Takeshi Yoshino]

StreamReader proposed in the Streams API spec is almost the same as
FileReader. By adding the maxSize argument to the readAs methods (new
methods or just add it to existing methods as an optional argument) and
adding the readAsBlob method, FileReader can cover all what StreamReader
provides. Has this already been discussed here?

I heard that some people who had this concern discussed briefly and were
worrying about derailing File API standardization.

We're planning to implement it on Chromium/Blink shortly.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

I skimmed the thread before starting this and saw that you were pointing
out some issues but didn't think you're opposing so much.



Let me check requirements.

a) We don't want to introduce a completely new object for streaming HTTP
read/write, but we'll realize it by adding some extension to XHR.

b) The point to connect the I/O API and XHR should be only the send()
method argument and xhr.response attribute if possible.

c) The semantics (attribute X is valid when state is ..., etc.) should be
kept same as other modes.

d) The I/O API needs to work with synchronous XHR.

e) Resource for already processed data should be able to be released
explicitly as the user instructs.

f) Reading with maxSize argument (don't read too much).

g) The I/O API should allow for skipping unnecessary data without creating
a new object for that.

Not requirement

h) Some people wanted Stream to behave like not an object to store the data
but kinda dam put between response attribute and XHR's internal buffer (and
network stack) expecting that XHR doesn't consume data from the network
until read operation is invoked on Stream object. (i.e. Stream controls
data flow in addition to callback invocation timing). But it's no longer
considered to be a requirement.

i) Reading with size argument (invoke callback only when data of the
specified amount is ready. Only data of the specified size at the head of
stream is passed to the handler)


On Fri, May 17, 2013 at 2:41 AM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, May 16, 2013 at 6:31 PM, Travis Leithead
 travis.leith...@microsoft.com wrote:
  Since we have Streams implemented to some degree, I'd love to hear
 suggestions to improve it relative to IO. Anne can you summarize the points
 you've made on the other various threads?

 I recommend reading through

 http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/thread.html#msg569

 Problems:

 * Too much complexity for being an Blob without synchronous size.
 * The API is bad. The API for File is bad too, but we cannot change
 it, this however is new.

 And I think we really want an IO API that's not about incremental, but
 can actively discard incoming data once it's processed.


 --
 http://annevankesteren.nl/

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

Sorry, I just took over this work and so was misunderstanding some point in
the Streams API spec.

On Fri, May 17, 2013 at 6:09 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, May 16, 2013 at 10:14 PM, Takeshi Yoshino tyosh...@google.com
 wrote:
  I skimmed the thread before starting this and saw that you were pointing
 out
  some issues but didn't think you're opposing so much.

 Well yes. I removed integration from XMLHttpRequest a while back too.


  Let me check requirements.
 
  d) The I/O API needs to work with synchronous XHR.

 I'm not sure this is a requirement. In particular in light of
 http://infrequently.org/2013/05/the-case-against-synchronous-worker-apis-2/
 and synchronous being worker-only it's not entirely clear to me this
 needs to be a requirement from the get-go.


  e) Resource for already processed data should be able to be released
  explicitly as the user instructs.

 Can't this happen transparently?


Yes. Read data is automatically released model is simple and good.

I thought the spec is clear about this but sorry it isn't. In the spec we
should say that StreamReader invalidates consumed data in Stream and buffer
for the invalidated bytes will be released at that point. Right?


  g) The I/O API should allow for skipping unnecessary data without
 creating a
  new object for that.

 This would be equivalent to reading and discarding?


I wanted to understand clearly what you meant by discard in your posts. I
wondered if you were suggesting that we have some method to skip incoming
data without creating any object holding received data. I.e. something like

s.skip(10);
s.readFrom(10);

not like

var useless_data_at_head_remaining = 256;
ondata = function(evt) {
  var bytes_received = evt.data.size();
  if (useless_data_at_head_remaining  bytes_received) {
useless_data_at_head_remaining -= bytes_received;
return;
  }

  processUsefulData(evt.data.slice(uselesss_data_at_head_remaining));
}

If you meant the latter, I'm ok. I'd also call the latter reading and
discarding.


  Not requirement
 
  h) Some people wanted Stream to behave like not an object to store the
 data
  but kinda dam put between response attribute and XHR's internal buffer
 (and
  network stack) expecting that XHR doesn't consume data from the network
  until read operation is invoked on Stream object. (i.e. Stream controls
 data
  flow in addition to callback invocation timing). But it's no longer
  considered to be a requirement.

 I'm not sure what this means. It sounds like something that indeed
 should be transparent from an API point-of-view, but it's hard to
 tell.


In the thread, Glenn was discussing what's consumer and what's producer,
IIRC.

I supposed that the idea behind Stream is providing a flow control
interface to control XHR has internal buffer. When the internal buffer is
full, it stops reading data from the network (e.g. BSD socket). The buffer
will be drained when and only when read operation is made on the Stream
object.

Stream has infinite length, but shouldn't have infinite capacity. It'll
swell up if the consumer (e.g. media stream?) is slow.

Of course, browsers would set some limit, but it should rather be well
discussed in the spec. Unless the limit is visible to scripts, they cannot
know if it can watch only load event or need to handle progress event
and consume arrived data progressively to process all data.


 We also need to decide whether a stream supports multiple readers or
 whether you need to explicitly clone a stream somehow. And as far as
 the API goes, we should study existing libraries.


What use cases do you have in your mind? Your example in the thread was
passing one to video but also accessing it manually using StreamReader. I
think it's unknown in what timing and how much video consumes data from
the Stream to the script and it's really hard make such coordination
successful.

Are you thinking of use case like mixing chat data and video contents in
the same HTTP response body?

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Fri, May 17, 2013 at 6:15 PM, Anne van Kesteren ann...@annevk.nl wrote:

 The main problem is that Stream per Streams API is not what you expect
  from an IO stream, but it's more what Blob should've been (Blob
 without synchronous size). What we want I think is a real IO stream.
 If we also need Blob without synchronous size is less clear to me.


Forgetting File API completely, for example, ... how about simple socket
like interface?

// Downloading big data

var remaining;
var type = null;
var payload = '';
function processData(data) {
  var offset = 0;
  while (offset  data.length) {
if (!type) {
  var type = data.substr(offset, 1);
  remaining = payloadSize(type);
} else if (remaining  0) {
  var consume = Math.min(remaining, data.length - offset);
  payload += data.substr(offset, consume);
  offset += consume;
} else if (remaining == 0) {
  if (type == FOO) {
foo(payload);
  } else if (type == BAR) {
bar(payload);
  }
  type = null;
}
  }
}

var client = new XMLHttpRequest();
client.onreadystatechange = function() {
  if (this.readyState == this.LOADING) {
var responseStream = this.response;
responseStream.setBufferSize(1024);
responseStream.ondata = function(evt) {
  processData(evt.data);
  // Consumed data will be invalidated and memory used for the data
will be released.
};
responseStream.onclose = function() {
  // Reached end of response body
  ...
};
responseStream.start();
// Now responseStream starts forwarding events happen on XHR to its
callbacks.
  }
};
client.open(GET, /foobar);
client.responseType = stream;
client.send();

// Uploading big data

var client = new XMLHttpRequest();
client.open(POST, /foobar);

var requestStream = new WriteStream(1024);

var producer = new Producer();
producer.ondata = function(evt.data) {
  requestStream.send(evt.data);
};
producer.onclose = function() {
  requestStream.close();
};

client.send(requestStream);

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Sat, May 18, 2013 at 1:38 PM, Jonas Sicking jo...@sicking.cc wrote:

 For File reading I would now instead do something like

 partial interface Blob {
   AbortableProgressFutureArrayBuffer readBinary(BlobReadParams);
   AbortableProgressFutureDOMString readText(BlobReadTextParams);
   Stream readStream(BlobReadParams);


I'd name it asStream. readStream operation here isn't intended to do any
read, i.e. moving data between buffers, (like ArrayBufferView for
ArrayBuffer) right?

Or it's gonna clone the Blob's contents and wrap with the Stream interface
as we cannot discard contents of a Blob and it'll be inconsistent with
the semantics (implication?) we're going to give to the Stream interface?

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Sat, May 18, 2013 at 1:56 PM, Jonas Sicking jo...@sicking.cc wrote:

 On Fri, May 17, 2013 at 9:38 PM, Jonas Sicking jo...@sicking.cc wrote:
  For Stream reading, I think I would do something like the following:
 
  interface Stream {
AbortableProgressFutureArrayBuffer readBinary(optional unsigned
  long long size);
AbortableProgressFutureString readText(optional unsigned long long
  size, optional DOMString encoding);
AbortableProgressFutureBlob readBlob(optional unsigned long long
 size);
 
ChunkedData readBinaryChunked(optional unsigned long long size);
ChunkedData readTextChunked(optional unsigned long long size);
  };
 
  interface ChunkedData : EventTarget {
attribute EventHandler ondata;
attribute EventHandler onload;
attribute EventHandler onerror;
  };

 Actually, we could even get rid of the ChunkedData interface and do
 something like

 interface Stream {
   AbortableProgressFutureArrayBuffer readBinary(optional unsigned
 long long size);
   AbortableProgressFutureString readText(optional unsigned long long
 size, optional DOMString encoding);
   AbortableProgressFutureBlob readBlob(optional unsigned long long size);

   AbortableProgressFuturevoid readBinaryChunked(optional unsigned
 long long size);
   AbortableProgressFuturevoid readTextChunked(optional unsigned long
 long size);
 };

 where the ProgressFutures returned from
 readBinaryChunked/readBinaryChunked delivers the data in the progress
 notifications only, and no data is delivered when the future is
 actually resolved. Though this might be abusing Futures a bit?


This is also clear read-only-once interface as well as onmessage() approach
because there's no attribute to accumulate the result value. The fact that
the argument for accept callback is void strikes at least me that the value
passed to progress callback is not an accumulated result but each chunk
separately.

As the state transition of Stream would be simple enough to match Future, I
think technically it's ok and even better to employ it than readyState +
callback approach.

But is everyone fine with making it mandatory to get used to programming
with Future to use Stream?

Re: [XHR] Event firing order. XMLHttpRequestUpload then XMLHttpRequest or reverse

[Takeshi Yoshino]

On Fri, Aug 2, 2013 at 2:13 AM, Anne van Kesteren ann...@annevk.nl wrote:

 On Tue, Jul 30, 2013 at 10:25 AM, Takeshi Yoshino tyosh...@google.com
 wrote:
  Change on 2010/09/13
 
 http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest-2/Overview.src.html.diff?r1=1.138;r2=1.139;f=h
  reversed the order of event firing for request error algorithm and
 send()
  method to XHRUpload-then-XHR.
 
  send() (only loadstart event) and abort() method are still specified to
 fire
  events in XHR-then-XHRUpload order. Is this intentional or we should make
  them consistent?

 We should make them consistent in some manner. Firing on the main
 object last makes sense to me. It also makes some amount of conceptual
 sense to do the reverse for when the fetching starts, but I feel less
 strongly about that.


In the spec, we have three cancels
- cancel an instance of fetch algorithm
- cancel network activity
- cancel a request

The spec says Cancel a request is an abort error which fires events in
XHR-XHRUpload order, but abort() fires events in XHR-XHRUpload order. It
was confusing so I filed this bug. First and at least, I'd like to make
this clear.

What does cancel a request correspond to?

Re: loadstart, I don't have strong opinion, too.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Fri, Aug 23, 2013 at 2:41 AM, Isaac Schlueter i...@izs.me wrote:

 1. Drop the read n bytes part of the API entirely.  It is hard to do


I'm ok with that. But then, instead we need to evolve ArrayBuffer to have
powerful concat/slice functionality for performance. Re: slicing, we can
just make APIs to accept ArrayBufferView. How should we deal with concat
operation? You suggested that we add unshift(), but repeating read and
unshift until we get enough data sound not so good.

For example, currently TextDecoder (http://encoding.spec.whatwg.org/)
accepts one ArrayBufferView and outputs one DOMString. We can use stream
mode of TextDecoder to get multiple output DOMStrings and then concatenate
them to get the final result.

As we still don't have StringBuilder, it's not considered to be a big deal
to have ArrayBufferBuilder (Stream.read(size) is kinda ArrayBuffer
builder)?

Is any of you guys thinking about introducing something like Node.js's
Buffer class for decoding and tokenization? TextDecoder+Stream would be a
kind of such classes.

I also considered making read() operation to accept pre-allocated
ArrayBuffer and return the number of bytes written.

  stream.read(buffer)

If written data is insufficient, the user can continue to pass the same
buffer to fill the unused space. But, since DOMString is immutable, we
can't take the same approach for readText() op.


 see in Node), and complicates the internal mechanisms.  People think

they need it, but what they really need is readUntil(delimiterChar).


What if implementing length header based protocol, e.g. msgpack?


 2. Reading strings vs ArrayBuffers or other types of things MUST be a

property of the stream,


Fixed property or mutable via readType attribute?

If readType, the sequence of UTF8/binary mixed read() problem remains.


 3. Sync vs async read().  Let's dig into the issue of
 `var d = s.read()` vs `s.read(function(d) {})` for getting data out of
 a stream.

...snip...

 buffering to occur if you have pipe chains of streams that are
 processing at different speeds, where one is bursty and the other is
 consistent.


Clarification. You're saying that always posting cb to task queue is
wasteful. Right?

Anyway, I think it makes sense. If read is designed to invoke cb
synchronously, it'll be difficult to avoid stack overflow. So the only
options is to always run cb in the next task.


 stream.poll(function ondata() {


What happens if unshift() is called? poll() invokes ondata() only when new
data (unshift()-ed data is not included) is available?


   var d = stream.read();
   while (stream.state === 'OK') {
 processData(d);
 d = stream.read();
   }


Is Jonas right about the reason why we need loop here? I.e. to avoid
automatic merge/serialization of buffered chunks?


   switch (stream.state) {
 case 'EOF': onend(); break;
 case 'EWOULDBLOCK': stream.poll(ondata); break;
 default: onerror(new Error('Stream read error: ' + stream.state));


We could distinguish these three states by null, empty
ArrayBuffer/DOMString, and non-empty ArrayBuffer/DOMString?


 ReadableStream.prototype.readAll = function(onerror, ondata, onend) {
   onpoll();
   function onpoll() {


If we decide not to allow multiple concurrent read operations on a stream,
can we just use event handler approach.

stream.onerror = ...
stream.ondata = ...


 4. Passive data listening.  In Node v0.10, it is not possible to
 passively listen to the data passing through a stream without
 affecting the state of the stream.  This is corrected in v0.12, by
 making the read() method also emit a 'data' event whenever it returns
 data, so v0.8-style APIs work as they used to.

 The takeaway here is not to do what Node did, but to learn what Node
 learned: the passive-data-listening use-case is relevant.


What's the use case?


 5. Piping.  It's important to consider how any proposed readable
 stream API will allow one to respond to backpressure, and how it
 relates to a *writable* stream API.  Data management from a source to
 a destination is the fundamental reason d'etre for streams, after all.


I'd have onwritable and onreadable handler, make their threshold
configurable and let pipe to setup them.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

Here's my all-in-one strawman proposal including some new stuff for flow
control. Yes, it's too big, but may be useful for glancing what features
are requested.



enum StreamReadType {
  ,
  arraybuffer,
  text
};

[Constructor(optional DOMString mime, optional [Clamp] long long
writeBufferSize, optional [Clamp] long long readBufferSize)]
interface Stream : EventTarget {
  readonly attribute DOMString type;  // MIME type

  // Writing interfaces

  readonly attribute unsigned long long writableSize;  // Bytes can be
written synchronously
  attribute unsigned long long writeBufferSize;

  attribute EventHandler onwritable;
  attribute unsigned long long writableThreshold;

  attribute EventHandler onpulled;

  attribute EventHandler onreadaborted;

  void write((DOMString or ArrayBufferView or Blob)? data);
  void flush();
  void closeWrite();
  void abortWrite();

  // Reading interfaces

  attribute StreamReadType readType;  // Must not be set after the first
read()
  attribute DOMString readEncoding;

  readonly attribute unsigned long long readableSize;  // Bytes can be read
synchronously
  attribute unsigned long long readBufferSize;

  attribute EventHandler onreadable;
  attribute unsigned long long readableThreshold;

  attribute EventHandler onflush;

  attribute EventHandler onclose;  // Receives clean flag

  any read(optional [Clamp] long long size);
  any peek(optional [Clamp] long long size, optional [Clamp] long long
offset);
  void skip([Clamp] long long size);
  void pull();
  void abortRead();

  // Async interfaces

  attribute EventHandler ondoneasync;  // Receives bytes skipped or Blob or
undefined (when done pipeTo)

  void readAsBlob(optional [Clamp] long long size);
  void longSkip([Clamp] long long size);
  void pipeTo(Stream destination, optional [Clamp] long long size);
};



- Encoding for text mode reading is determined by the type attribute. Can
be overridden by setting the readEncoding attribute.

- Invoking read() repeatedly to pull data into the stream is annoying. So,
instead I used writable/readableThreshold approach.

- Not to bloat the API anymore, limited error/close signaling interface to
only EventHandlers.

- stream.read() means stream.read(stream.readableSize).

- After onclose invocation, it's guaranteed that all written bytes are
available for read.

- read() is non-blocking. It returns only what is synchronously readable.
If there isn't enough bytes (investigate the readableSize attribute), an
app should wait until the next invocation of onreadable. readBufferSize and
readableThreshold may be modified accordingly and pull() may be called.
- stream.read(size) returns an ArrayBuffer or DOMString of min(size,
stream.readableSize) bytes that is synchronously readable now.

- When readType is set to text, read() throws an EncodingError if an
invalid sequence is found. Incomplete sequence will be left unconsumed. If
there's an incomplete sequence at the end of stream, the app can know that
by checking the size attribute after onclose invocation and read() call.

- readableSize attribute returns (number of readable bytes as of the last
time the event loop started executing a task) - (bytes consumed by read()
method).

- onflush is separated from onreadable since it's possible that an
intermediate Stream in a long change has no data to flush but the next or
later Streams have.
- Invocation order is onreadable - onflush or onclose.
- Flush handling code must be implemented on both onflush and onclose. On
close() call, only onclose is invoked to reduce event propagation cost.

- Pass read/writeBufferSize of -1 to constructor or set -1 to
stream.read/writeBufferSize for unlimited buffering.

- Instead of having write(buffer, cb), made write() accept any size of data
regardless of writeBufferSize. XHR should respect writeBufferSize and write
only writableSize bytes of data and set onwritable if necessary and
possibly also set writableThreashold.

- {writable,readable}Threshold are 0 by default meaning that onwritable and
onreadable are invoked immediately when there's space/data available.

- If {writable,readable}Threshold are greater than capacity, it's
considered to be set to capacity.

- onwritable/onreadable is invoked asynchronously when
-- new space/data is available as a result of read()/write() operation that
satisfies writable/readableThreshold
onreadable is invoked asynchronously when
-- flush()-ed or close()-ed

- onwritable/onreadable is invoked synchronously when
-- onwritable/onreadable is updated and there's space/data available that
satisfies writable/readableThreshold
-- writable/readableThreshold is updated and there's space/data available
that satisfies the new writable/readableThreshold
-- new space/data is available as a result of updating capacity that
satisfies writable/readableThreshold

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

I forgot to add an attribute to specify the max size of backing store.
Maybe it should be added to the constructor.

On Wed, Sep 11, 2013 at 11:24 PM, Takeshi Yoshino tyosh...@google.comwrote:

   any peek(optional [Clamp] long long size, optional [Clamp] long long
 offset);


peek with offset doesn't make sense for text mode reading. Some exception
should be throw for that case.


 - readableSize attribute returns (number of readable bytes as of the last
 time the event loop started executing a task) - (bytes consumed by read()
 method).


+ (bytes added by write() and transferred to read buffer synchronously)



The concept of this interface is
- to allow bulk transfer from internal asynchronous storage (e.g. network,
disk based backing store) to JS world but delay conversion (e.g. into
DOMString, ArrayBuffer).
- not to ask an app to do such transfer explicitly

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Thu, Sep 12, 2013 at 10:58 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Apparently we are not talking about the same thing, while I am thinking
 to a high level interface your interface is taking care of the underlying
 level.


How much low level stuff to expose would basically affect high level
interface design, I think.


 Like node's streams, node had to define it since it was not existing (but
 is someone using node's streams as such or does everybody use

...snip...

 So, to understand where the mismatch comes from, could you please
 highlight a web use case/code example based on your proposal?


I'm still thinking how much we should include in the API, too. This
proposal is just a trial to address the requirements Isaac listed. So, each
feature should correspond to some of his example code.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Fri, Sep 13, 2013 at 5:15 AM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Isaac said too So, just to be clear, I'm **not** suggesting that
 browser streams copy Node streams verbatim..


I know. I wanted to kick the discussion which was stopped for 2 weeks.


 Unless you want to do node inside browsers (which would be great but seems
 unlikely) I still don't see the relation between this kind of proposal and
 existing APIs.


What do you mean by existing APIs? I was thinking that we've been
discussing what Stream read/write API for manual consuming/producing by
JavaScript code should be like.


 Could you please give an example very different from the ones I gave
 already?


Sorry, which mail?

One of what I was imaging is protocol parsing. Such as msgpack, protocol
buffer. It's good that ArrayBuffers of exact size is obtained.

OTOH, as someone pointed out, Stream should have some flow control
mechanism not to pull unlimited amount of data from async storage, network,
etc. readableSize in my proposal is an example of how we make the limit
controllable by an app.

We could also depend on the size argument of read() call. But thinking of
protocol parsing again, it's common that we have small fields such as 4, 8,
16 bytes. If read(size) is configured to pull size bytes from async
storage, it's inefficient. Maybe we could have some hard coded limit, e.g.
1MiB and use max(hardCodedLimit, requestedReadSize).

I'm fine with the latter.


 You have reverted to EventTarget too instead of promises, why?


There was no intention to object against use of Promise. Sorry that I
wasn't clear. I'm rather interested in receiving sequence of data as they
become available (corresponds to Jonas's ChunkedData version read methods)
with just one read call. Sorry that I didn't mention explicitly, but
listeners on the proposed API came from ChunkedData object. I thought we
can put them on Stream itself by giving up multiple read scenario.

writeable/readableThreshold can be safely removed from the API if we agree
it's not important. If the threshold stuff are removed, flush() and pull()
will also be removed.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Fri, Sep 13, 2013 at 6:08 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

 Now you want a stream interface so you can code some js like mspack on top
 of it.

 I am still missing a part of the puzzle or how to use it: as you mention
 the stream is coming from somewhere (File, indexedDB, WebSocket, XHR,
 WebRTC, etc) you have a limited choice of APIs to get it, so msgpack will
 act on top of one of those APIs, no? (then back to the examples above)

 How can you get the data another way?


Do you mean that those data producer APIs should be changed to provide
read-by-delta-data, and manipulation of data by js code should happen there
instead of at the output of Stream?

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

Since I joined discussion recently, I don't know the original idea behind
the Stream+XHR integration approach (response returns Stream object) as in
current Streams API spec. But one advantage of it I come up with is that we
can keep change to those producer APIs small. If we decide to add methods
for example for flow control (though it's still under question), such stuff
go to Stream, not XHR, etc.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Fri, Sep 13, 2013 at 9:50 PM, Aymeric Vitte vitteayme...@gmail.comwrote:


 Le 13/09/2013 14:23, Takeshi Yoshino a écrit :

 Do you mean that those data producer APIs should be changed to provide
 read-by-delta-data, and manipulation of data by js code should happen there
 instead of at the output of Stream?



 Yes, exactly, except if you/someone see another way of getting the data
 inside the browser and turning the flow into a stream without using these
 APIs.


I agree that there're various states and things to handle for each of the
producer APIs, and it might be judicious not to convey such API specific
info/signal through Stream.

I don't think it's bad to convert xhr.DONE to stream.close() manually as in
your example
http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0453.html.

But, regarding flow control, as I said in the other mail just posted, if we
start thinking of flow control more seriously, maybe the right approach is
to have unified flow control method and the point to define such a
fine-grained flow control is Stream, not each API. If we're not, yes, maybe
your proposal (deltaResponse) should be enough.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Sat, Sep 14, 2013 at 12:03 AM, Aymeric Vitte vitteayme...@gmail.comwrote:


 I take this example to understand if this could be better with a built-in
 Stream flow control, if so, after you have defined the right parameters (if
 possible) for the streams flow control, you could process delta data while
 reading the file and restream them directly via WebSockets, and this would
 be great but again not sure that a universal solution can be found.


I think what we can do is just providing helper to make it easier to build
such an intelligent and app specific flow control logic.

Maybe one of the points of your example is that we're not always be able to
calculate good readableThreshold. I'm also not so sure how much of apps in
the world can benefit from this kind of APIs.

For consumers that can do flow control well on receive window basis, my API
should work well (unnecessary events are not dispatched. chunks are
consolidated. lazier ArrayBuffer creation). WebSocket has (broken)
bufferedAmount attribute for window based flow control. Are you using it as
a hint?

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Wed, Sep 25, 2013 at 12:41 AM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Did you see
 http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0593.html ?


Yes. This example seems to be showing how to connect only producer/consumer
APIs which support Stream. Right?

In such a case, all the flow control stuff would be basically hidden, and
if necessary each producer/consumer/transformer/filter/etc. may expose flow
control related parameter in their own form, and configure connected
input/output streams accordingly. E.g. stream_xhr may choose to have large
write buffer for performance, or have small one and make some backpressure
to stream_ws1 for memory efficiency.

My understanding is that the flow control APIs like mine are intended to be
used by JS code implementing some converter, consumer, etc. while built-in
stuff like WebCrypt would be evolved to accept Stream directly and handle
flow control in e.g. C++ world.



BTW, I'm discussing this to provide data points to decide whether to
include flow control API or not. I'm not pushing it. I appreciate if other
participants express opinions about this.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Wed, Sep 25, 2013 at 10:55 PM, Aymeric Vitte vitteayme...@gmail.comwrote:


  My understanding is that the flow control APIs like mine are intended to
 be used by JS code implementing some converter, consumer, etc. while
 built-in stuff like WebCrypt would be evolved to accept Stream directly and
 handle flow control in e.g. C++ world.


  

  BTW, I'm discussing this to provide data points to decide whether to
 include flow control API or not. I'm not pushing it. I appreciate if other
 participants express opinions about this.



 Not sure to get what you mean between your API flow control and built-in
 flow control... I think the main purpose of the Stream API should be to
 handle more efficiently streaming without having to handle ArrayBuffers
 copy, split, concat, etc, to abstract the use of ArrayBuffer,
 ArrayBufferView, Blob, txt so you don't spend your time converting things
 and to connect simply different streams.


JS flow control API is for JS code to manually control threshold, buffer
size, etc. so that JS code can consume, produce data to/from Stream.

Built-in flow control is C++ (or any other lang implementing the UA)
interface that will be used when streams are connected with pipe(). Maybe
it would have similar interface as JS flow control API.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

As we don't see any strong demand for flow control and sync read
functionality, I've revised the proposal.

Though we can separate state/error signaling from Stream and keep them done
by each API (e.g. XHR) as Aymeric said, EoF signal still needs to be
conveyed through Stream.



enum StreamReadType {
  ,
  blob,
  arraybuffer,
  text
};

interface StreamConsumeResult {
  readonly attribute boolean eof;
  readonly any data;
  readonly unsigned long long size;
};

[Constructor(optional DOMString mime)]
interface Stream {
  readonly attribute DOMString type;  // MIME type

  // Rejected on error. No more write op shouldn't be made.
  //
  // Fulfilled when the write completes. It doesn't guarantee that the
written data has been
  // read out successfully.
  //
  // The contents of ArrayBufferView must not be modified until the promise
is fulfilled.
  //
  // Fulfill may be delayed when the Stream considers itself to be full.
  //
  // write(), close() must not be called again until the Promise of the
last write() is fulfilled.
  Promisevoid write((DOMString or ArrayBufferView or Blob)? data);
  void close();

  attribute StreamReadType readType;
  attribute DOMString readEncoding;

  // read(), skip(), pipe() must not be called again until the Promise of
the last read(), skip(), pipe() is fulfilled.

  // Rejected on error. No more read op shouldn't be made.
  //
  // If size is specified,
  // - if EoF: fulfilled with data up to EoF
  // - otherwise: fulfilled with data of size bytes
  //
  // If size is omitted, (all or part of) data available for read now will
be returned.
  //
  // If readType is set to text, size of the result may be smaller than the
value specified for the size argument.
  PromiseStreamConsumeResult read(optional [Clamp] long long size);

  // Rejected on error. Fulfilled on completion.
  //
  // .data of result is not used. .size of result is the skipped amount.
  PromiseStreamConsumeResult skip([Clamp] long long size);  // .data is
skipped size

  // Rejected on error. Fulfilled on completion.
  //
  // If size is omitted, transfer until EoF is encountered.
  //
  // .data of result is not used. .size of result is the size of data
transferred.
  PromiseStreamConsumeResult pipe(Stream destination, optional [Clamp]
long long size);
};

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Thu, Sep 26, 2013 at 6:36 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

 Looks good, comments/questions :

 - what's the use of readEncoding?


Overriding charset specified in .type for read op. It's weird but we can
ask an app to overwrite .type instead.



 - StreamReadType: add MediaStream? (and others if existing)


Maybe, if there's clear rule to convert binary stream + MIME type into
MediaStream object.



 - would it be possible to pipe from StreamReadType to other StreamReadType?


pipe() tells the receiver with which value of StreamReadType the pipe() was
called. Receiver APIs may be designed to accept either mode or both modes.



 - would it be possible to pipe from a source to different targets (my
 example of encrypt/hash at the same time)?


I missed it. Your mirroring method (making pipe accept multiple Stream)
looks good.

The problem is what to do when one of destinations is write blocked. Maybe
we want to read data from the source as the fastest consumer consumes and
save read data for slowest one. When should we fulfill the promise?
Completion of read from the source, completion of write to all
destinations, etc.



 - what is the link between the API and the Stream (responseType='stream')?
 How do you handle this for APIs where responseType does not really apply
 (mspack, crypto...)


- make APIs to return a Stream for read (write) like
XHR.responseType='stream'
- make APIs to accept a Stream for read (write)

Either should work as we have pipe().

E.g.

var sourceStream = xhr.response;
var resultStream = new Stream();
var fileWritingPromise = fileWriter.write(resultStream);
var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey,
sourceStream, resultStream);
Promise.all(fileWritingPromise, encryptionPromise).then(
  ...
);



I also found a point needs clarification

- pipe() does eof or not. I think we don't want automatic eof.

Re: [XHR] Event firing order. XMLHttpRequestUpload then XMLHttpRequest or reverse

[Takeshi Yoshino]

Sorry for the delay.

On Tue, Sep 3, 2013 at 9:00 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Tue, Sep 3, 2013 at 9:18 AM, Takeshi Yoshino tyosh...@google.com
 wrote:
  In the spec, we have three cancels
  - cancel an instance of fetch algorithm
  - cancel network activity

 These are the same. Attempted to clarify.


Verified that the order of events are the same for abort() and abort error.

Thanks for factoring out termination algorithm. It's clearer now.



  - cancel a request

 This is the end user terminate, correct?


Yes. So, this includes any kind of event resulting in termination of fetch
algorithm (network, termination by some user's instruction to UA)?


 Would you like to be
 acknowledged as Takeshi Yoshino? If you can give me your name in
 kanji I can include that too. See e.g.
 http://encoding.spec.whatwg.org/#acknowledgments for some examples.


Thank you. Just alphabetical name is fine.


 See http://xhr.spec.whatwg.org/ for the updated text. And
 https://github.com/whatwg/xhr/commits for an overview of the changes.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

Formatted and published my latest proposal at github after incorporating
Aymeric's multi-dest idea.

http://htmlpreview.github.io/?https://github.com/tyoshino/stream/blob/master/streams.html


On Sat, Sep 28, 2013 at 11:45 AM, Kenneth Russell k...@google.com wrote:

 This looks nice. It looks like it should already handle the flow control
 issues mentioned earlier in the thread, simply by performing the read on
 demand, though reporting the result asynchronously.


Thanks, Kenneth for reviewing.

Re: [XHR] Event firing order. XMLHttpRequestUpload then XMLHttpRequest or reverse

[Takeshi Yoshino]

On Fri, Oct 4, 2013 at 8:46 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, Oct 3, 2013 at 6:35 AM, Takeshi Yoshino tyosh...@google.com
 wrote:
  On Tue, Sep 3, 2013 at 9:00 PM, Anne van Kesteren ann...@annevk.nl
 wrote:
  This is the end user terminate, correct?
 
  Yes. So, this includes any kind of event resulting in termination of
 fetch
  algorithm (network, termination by some user's instruction to UA)?

 No, if you look at
 http://xhr.spec.whatwg.org/#infrastructure-for-the-send%28%29-method
 If the end user cancels the request is about the end user, If there
 is a network error is about everything else.


 Sorry. I included network by mistake. I wanted to understand the abort
error well.

Q: cancel by abort() is abort error?
Q: any kind of network activity cancellation not due to network/timeout are
abort error?

Re: [XHR] Event firing order. XMLHttpRequestUpload then XMLHttpRequest or reverse

[Takeshi Yoshino]

On Sat, Oct 5, 2013 at 1:26 AM, Anne van Kesteren ann...@annevk.nl wrote:

 On Fri, Oct 4, 2013 at 3:12 PM, Takeshi Yoshino tyosh...@google.com
 wrote:
  Sorry. I included network by mistake. I wanted to understand the abort
 error
  well.
 
  Q: cancel by abort() is abort error?

 It's not the same condition. abort() has its own set of steps.
 Although we might be able to merge these and probably should...



OK.


   Q: any kind of network activity cancellation not due to network/timeout
 are
  abort error?

 No.


OK. I didn't understand what you meant by end user correctly. Never mind.

Re: [streams-api] Seeking status and plans

[Takeshi Yoshino]

On Thu, Oct 10, 2013 at 11:34 PM, Feras Moussa feras.mou...@hotmail.comwrote:

 Apologies for the delay, I had broken one of my mail rules and didn't see
 this initially.

 Asymeric is correct - there have been a few threads and revisions. A more
 up-to-date version is the one Asymeric linked -
 http://htmlpreview.github.io/?https://github.com/tyoshino/stream/blob/master/streams.html
 The above version incorporates both promises and streams and is a more
 refined version of Streams.

 From other threads on Stream, it became apparent that there were a few
 pieces of the current Streams API ED that were designed around older
 paradigms and needed refining to be better aligned with current APIs.  I
 think it does not make sense to have two different specs, and instead have
 a combined one that we move forward.

 I can work with Takeshi on getting his version incorporated into the
 Streams ED, which we can then move forward with.


I'm happy to.



 Thanks,
 Feras


  Date: Thu, 10 Oct 2013 09:32:20 -0400
  From: art.bars...@nokia.com
  To: vitteayme...@gmail.com; feras.mou...@hotmail.com;
 tyosh...@google.com
  CC: public-webapps@w3.org
  Subject: Re: [streams-api] Seeking status and plans
 
  On 10/10/13 6:26 AM, ext Aymeric Vitte wrote:
  I think the plan should be more here now:
  http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0049.html
 
  There are indeed at least two specs here:
 
  [1] Feras' https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm
 
 
  [2] Takeshi's
  
 http://htmlpreview.github.io/?https://github.com/tyoshino/stream/blob/master/streams.html
 
 
  Among the Qs I have here are ...
 
  * What is the implementation status of these specs?


There's a Blink intent-to-implement for Streams API I sent. I plan to
implement it based on [2]. By turning on a flag to turn on experimental
features, XHR in Chrome can return a response as a Stream object. It's
based on 10. Extension of XMLHttpRequest of [1]. Nothing done yet for the
rest. We're preparing for it (such as implementation of Promises).

I agree that we should merge [1] and [2] as Feras said.

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

Sorry for blank of ~2 weeks.

On Fri, Oct 4, 2013 at 5:57 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  I am still not very familiar with promises, but if I take your preceeding
 example:


 var sourceStream = xhr.response;
 var resultStream = new Stream();
 var fileWritingPromise = fileWriter.write(resultStream);
 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey,
 sourceStream, resultStream);
  Promise.all(fileWritingPromise, encryptionPromise).then(
   ...
 );


I made a mistake. The argument of Promise.all should be an Array. So,
[fileWritingPromise, encryptionPromise].




 shoud'nt it be more something like:

 var sourceStream = xhr.response;
 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey);
 var resultStream=sourceStream.pipe(encryptionPromise);
 var fileWritingPromise = fileWriter.write(resultStream);
  Promise.all(fileWritingPromise, encryptionPromise).then(
   ...
 );


Promises just tell the user completion of each operation with some value
indicating the result of the operation. It's not destination of data.

Do you think it's good to create objects representing each encrypt
operation? So, some objects called filter is introduced and the code
would be like:

var pipeToFilterPromise;

var encryptionFilter;
var fileWriter;

xhr.onreadystatechange = function() {
  ...
  } else if (this.readyState == this.LOADING) {
 if (this.status != 200) {
  ...
}

var sourceStream = xhr.response;

encryptionFilter =
crypto.subtle.createEncryptionFilter(aesAlgorithmEncrypt, aesKey);
// Starts the filter.
var encryptionPromise = encryptionFilter.encrypt();
// Also starts pouring data but separately from promise creation.
pipeToFilterPromise = sourceStream.pipe(encryptionFilter);

fileWriter = ...;
// encryptionFilter works as data producer for FileWriter.
var fileWritingPromise = fileWriter.write(encryptionFilter);

// Set only handler for rejection now.
pipeToFilterPromise.catch(
  function(result) {
xhr.abort();
encryptionFilter.abort();
fileWriter.abort();
  }
);

encryptionPromise.catch(
  function(result) {
xhr.abort();
fileWriter.abort();
  }
);

fileWritingPromise.catch(
  function(result) {
xhr.abort();
encryptionFilter.abort();
  }
);

// As encryptionFilter will be (successfully) closed only
// when XMLHttpRequest and pipe() are both successful.
// So, it's ok to set handler for fulfillment now.
Promise.all([encryptionPromise, fileWritingPromise]).then(
  function(result) {
// Done everything successfully!
// We come here only when encryptionFilter is close()-ed.
fileWriter.close();
processFile();
  }
);
  } else if (this.readyState == this.DONE) {
 if (this.status != 200) {
  encryptionFilter.abort();
  fileWriter.abort();
} else {
  // Now we know that XHR was successful.
  // Let's close() the filter to finish encryption
  // successfully.
  pipeToFilterPromise.then(
function(result) {
  // XMLHttpRequest closes sourceStream but pipe()
  // resolves pipeToFilterPromise without closing
  // encryptionFilter.
  encryptionFilter.close();
}
  );
}
  }
};
xhr.send();

encryptionFilter has the same interface as normal stream but encrypts piped
data. Encrypted data is readable from it. It has special methods, encrypt()
and abort().

processFile() is hypothetical function must be called only when all of
loading, encryption and saving to file were successful.



 or

 var sourceStream = xhr.response;
 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey);
 var hashPromise = crypto.subtle.digest(hash);
 var resultStream = sourceStream.pipe([encryptionPromise,hashPromise]);
 var fileWritingPromise = fileWriter.write(resultStream);
 Promise.all([fileWritingPromise, resultStream]).then(
   ...
 );


and this should be:

var sourceStream = xhr.response;

encryptionFilter =
crypto.subtle.createEncryptionFilter(aesAlgorithmEncrypt, aesKey);
var encryptionPromise = encryptionFilter.crypt();

hashFilter = crypto.subtle.createDigestFilter(hash);
var hashPromise = hashFilter.digest();

pipeToFiltersPromise = sourceStream.pipe([encryptionFilter, hashFilter]);

var encryptedDataWritingPromise = fileWriter.write(encryptionFilter);

var hashWritingPromise =
  Promise.all([encryptionPromise, encryptedDataWritingPromise]).then(
function(result) {
  return fileWriter.write(hashFilter)
},
...
  );

Promise.all([hashPromise, hashWritingPromise]).then(
  function(result) {
fileWriter.close();
processFile();
  },
  ...
);

Or, we can also choose to let the writer API to create a special object
that has the Stream interface for receiving input and then let
encryptionFilter and hashFilter to pipe() to it.

...
pipeToFiltersPromise = 

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Wed, Oct 23, 2013 at 11:42 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Your filter idea seems to be equivalent to a createStream that I
 suggested some time ago (like node), what about:

 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt, aesKey,
 sourceStream).createStream();

 So you don't need to modify the APIs where you can not specify the
 responseType.

 I was thinking to add stop/resume and pause/unpause:

 - stop: insert eof in the stream


close() does this.


  Example : finalize the hash when eof is received

 - resume: restart from where the stream stopped
 Example : restart the hash from the state the operation was before
 receiving eof (related to Issue22 in WebCrypto that was closed without any
 solution, might imply to clone the state of the operation)


Should it really be a part of Streams API? How about just making the filter
(not Stream itself) returned by WebCrypto reusable and add some method to
recycle it?


 - pause: pause the stream, do not send eof


Sorry, what will be paused? Output?


  - unpause: restart the stream

 And flow control should be back and explicit, not sure right now how to
 define it but I think it's impossible for a js app to do a precise flow
 control, and for existing APIs like WebSockets it's not easy to control the
 flow and avoid in some situations to overload the UA.

 Regards,

 Aymeric

 Le 21/10/2013 13:14, Takeshi Yoshino a écrit :

  Sorry for blank of ~2 weeks.

  On Fri, Oct 4, 2013 at 5:57 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  I am still not very familiar with promises, but if I take your
 preceeding example:


 var sourceStream = xhr.response;
 var resultStream = new Stream();
 var fileWritingPromise = fileWriter.write(resultStream);
 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt,
 aesKey, sourceStream, resultStream);
  Promise.all(fileWritingPromise, encryptionPromise).then(
   ...
 );


  I made a mistake. The argument of Promise.all should be an Array. So,
 [fileWritingPromise, encryptionPromise].




  shoud'nt it be more something like:

 var sourceStream = xhr.response;
 var encryptionPromise = crypto.subtle.encrypt(aesAlgorithmEncrypt,
 aesKey);
 var resultStream=sourceStream.pipe(encryptionPromise);
 var fileWritingPromise = fileWriter.write(resultStream);
  Promise.all(fileWritingPromise, encryptionPromise).then(
   ...
 );


  Promises just tell the user completion of each operation with some value
 indicating the result of the operation. It's not destination of data.

  Do you think it's good to create objects representing each encrypt
 operation? So, some objects called filter is introduced and the code
 would be like:

  var pipeToFilterPromise;

  var encryptionFilter;
  var fileWriter;

  xhr.onreadystatechange = function() {
   ...
   } else if (this.readyState == this.LOADING) {
  if (this.status != 200) {
   ...
 }

  var sourceStream = xhr.response;

  encryptionFilter =
 crypto.subtle.createEncryptionFilter(aesAlgorithmEncrypt, aesKey);
 // Starts the filter.
 var encryptionPromise = encryptionFilter.encrypt();
 // Also starts pouring data but separately from promise creation.
  pipeToFilterPromise = sourceStream.pipe(encryptionFilter);

  fileWriter = ...;
 // encryptionFilter works as data producer for FileWriter.
 var fileWritingPromise = fileWriter.write(encryptionFilter);

  // Set only handler for rejection now.
  pipeToFilterPromise.catch(
function(result) {
 xhr.abort();
 encryptionFilter.abort();
  fileWriter.abort();
   }
  );

  encryptionPromise.catch(
function(result) {
  xhr.abort();
  fileWriter.abort();
   }
  );

  fileWritingPromise.catch(
function(result) {
  xhr.abort();
  encryptionFilter.abort();
   }
  );

   // As encryptionFilter will be (successfully) closed only
  // when XMLHttpRequest and pipe() are both successful.
 // So, it's ok to set handler for fulfillment now.
  Promise.all([encryptionPromise, fileWritingPromise]).then(
function(result) {
 // Done everything successfully!
 // We come here only when encryptionFilter is close()-ed.
 fileWriter.close();
 processFile();
   }
 );
   } else if (this.readyState == this.DONE) {
  if (this.status != 200) {
   encryptionFilter.abort();
   fileWriter.abort();
 } else {
   // Now we know that XHR was successful.
   // Let's close() the filter to finish encryption
   // successfully.
   pipeToFilterPromise.then(
 function(result) {
   // XMLHttpRequest closes sourceStream but pipe()
   // resolves pipeToFilterPromise without closing
   // encryptionFilter.
   encryptionFilter.close();
 }
   );
 }
}
  };
 xhr.send();

  encryptionFilter has the same interface as normal stream

Re: Overlap between StreamReader and FileReader

[Takeshi Yoshino]

On Wed, Oct 30, 2013 at 8:14 PM, Takeshi Yoshino tyosh...@google.comwrote:

 On Wed, Oct 23, 2013 at 11:42 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

 - pause: pause the stream, do not send eof



 Sorry, what will be paused? Output?


http://lists.w3.org/Archives/Public/public-webrtc/2013Oct/0059.html
http://www.w3.org/2011/04/webrtc/wiki/Transport_Control#Pause.2Fresume

So, you're suggesting that we make Stream be a convenient point where we
can dam up data flow and skip adding methods to pausing data producing and
consuming to producer/consumer APIs? I.e. we make it able to prevent data
queued in a Stream from being read. This typically means asynchronously
suspending ongoing pipe() or read() call on the Stream with no-argument or
very large argument.




  - unpause: restart the stream

 And flow control should be back and explicit, not sure right now how to
 define it but I think it's impossible for a js app to do a precise flow
 control, and for existing APIs like WebSockets it's not easy to control the
 flow and avoid in some situations to overload the UA.

Splitting Stream into InputStream and OutputStream (was Re: CfC: publish WD of Streams API; deadline Nov 3)

[Takeshi Yoshino]

Hi François

On Thu, Oct 31, 2013 at 6:16 AM, François REMY 
francois.remy@outlook.com wrote:

 - Streams should exist in at least two fashions: InputStream and
 OutputStream. Both of them serve different purposes and, while some stream
 may actually be both, this remains an exceptional behavior worth being
 noted. Finally, a Stream is not equal to a InMemoryStream as the
 constructor may seem to indicate. A stream is a much lower-level concept,
 which may actually have nothing to do with InMemory operations.


Yes. I initially thought it'll be clearer to split in/out interface. E.g. a
Stream obtained from XHR to receive a response should not be writable. It's
reasonable to make network-to-Stream transfer happen in background
asynchronously to JavaScript, and then it doesn't make much sense to keep
it writable from JavaScript.

It has a unified IDL now but I'm designing write side and read side
independently. We could decouple it into two separate IDLs? concepts? if it
make sense. Stream would inherit from both and provides a constructor.

Defining generic Stream than considering only bytes (Re: CfC: publish WD of Streams API; deadline Nov 3)

[Takeshi Yoshino]

Hi Dean,

On Thu, Oct 31, 2013 at 11:30 AM, Dean Landolt d...@deanlandolt.com wrote:

 I really like this general concepts of this proposal, but I'm confused by
 what seems like an unnecessary limiting assumption: why assume all streams
 are byte streams? This is a mistake node recently made in its streams
 refactor that has led to an objectMode and added cruft.

 Forgive me if this has been discussed -- I just learned of this today. But
 as someone who's been slinging streams in javascript for years I'd really
 hate to see the standard stream hampered by this bytes-only limitation.
 The node ecosystem clearly demonstrates that streams are for more than
 bytes and (byte-encoded) strings.


To glue Streams with existing binary handling infrastructure such as
ArrayBuffer, Blob, we should have some specialization for Stream handling
bytes rather than using generalized Stream that would accept/output an
array or single object of the type. Maybe we can rename Streams API to
ByteStream not to occupy the name Stream that sounds like more generic, and
start standardizing generic Stream.


 In my perfect world any arbitrary iterator could be used to characterize
 stream chunks -- this would have some really interesting benefits -- but I
 suspect this kind of flexibility would be overkill for now. But there's
 good no reason bytes should be the only thing people can chunk up in
 streams. And if we're defining streams for the whole platform they
 shouldn't *just* be tied to a few very specific file-like use cases.

 If streams could also consist of chunks of strings (real, native strings)
 a huge swath of the API could disappear. All of readType, readEncoding and
 charset could be eliminated, replaced with simple, composable transforms
 that turn byte streams (of, say, utf-8) into string streams. And vice versa.


So, for example, XHR would be the point of decoding and it returns a Stream
of DOMStrings?


 Of course the real draw of this approach would be when chunks are neither
 blobs nor strings. Why couldn't chunks be arrays? The arrays could contain
 anything (no need to reserve any value as a sigil). Regardless of the chunk
 type, the zero object for any given type wouldn't be `null` (it would be
 something like '' or []). That means we can use null to distinguish EOF,
 and `chunk == null` would make a perfectly nice (and unambiguous) EOF
 sigil, eliminating yet more API surface. This would give us a clean object
 mode streams for free, and without node's arbitrary limitations.


For several reasons, I chose to use .eof than using null. One of them is to
allow the non-empty final chunk to signal EOF than requiring one more
read() call.

This point can be re-discussed.


 The `size` of an array stream would be the total length of all array
 chunks. As I hinted before, we could also leave the door open to specifying
 chunks as any iterable, where `size` (if known) would just be the `length`
 of each chunk (assuming chunks even have a `length`). This would also allow
 individual chunks to be built of generators, which could be particularly
 interesting if the `size` argument to `read` was specified as a maximum
 number of bytes rather than the total to return -- completely sensible
 considering it has to behave this way near the end of the stream anyway...


I don't really understand the last point. Could you please elaborate the
story and benefit?

IIRC, it's considered to be useful and important to be able to cut an exact
requested size of data into an ArrayBuffer object and get notified (the
returned Promise gets resolved) only when it's ready.


 This would lead to a pattern like `stream.read(Infinity)`, which would
 essentially say *give me everything you've got soon as you can*.


In current proposal, read() i.e. read() with no argument does this.


  This is closer to node's semantics (where read is async, for added
 scheduling flexibility). It would drain streams faster rather than
 pseudo-blocking for a specific (and arbitrary) size chunk which ultimately
 can't be guaranteed anyway, so you'll always have to do length checks.

 (On a somewhat related note: why is a 0-sized stream specified to throw?
 And why a SyntaxError of all things? A 0-sized stream seems perfectly
 reasonable to me.)


0-sized Stream is not prohibited.

Do you mean 0-sized read()/pipe()/skip()? I don't think they make much
sense. It's useful only when you want to sense EOF and it can be done by
read(1).


 What's particularly appealing to me about the chunk-as-generator idea is
 that these chunks could still be quite large -- hundreds megabytes, even.
 Just because a potentially large amount of data has become available since
 the last chunk was processed doesn't mean you should have to bring it all
 into memory at once.


It's interesting. Could you please list some concrete example of such a
generator?

Re: publish WD of Streams API; deadline Nov 3

[Takeshi Yoshino]

OK. There seems to be some disconnect, but I'm fine with waiting for
Domenic's proposal first.

Takeshi


On Thu, Oct 31, 2013 at 7:41 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Wed, Oct 30, 2013 at 6:04 PM, Domenic Denicola
 dome...@domenicdenicola.com wrote:
  I have concrete suggestions as to what such an API could look like—and,
 more importantly, how its semantics would significantly differ from this
 one—which I hope to flesh out and share more broadly by the end of this
 weekend. However, since the call for comments phase has commenced, I
 thought it important to voice these objections as soon as possible.

 Given how long we have been trying to figure out streams, waiting a
 little longer to see Domenic's proposal should be fine I think. No
 need to start rushing things through the process now. (Although on the
 flipside at some point we will need to start shipping something.)


 --
 http://annevankesteren.nl/

Re: Splitting Stream into InputStream and OutputStream (was Re: CfC: publish WD of Streams API; deadline Nov 3)

[Takeshi Yoshino]

On Thu, Oct 31, 2013 at 4:48 PM, François REMY 
francois.remy@outlook.com wrote:

  Since JavaScript does not provide a way to check if an object implements
 an interface, there should probably exist a way to check that from the API,
 like:


Basically it should be sufficient if each API can check type, but yeah
probably useful.


 var stream = InputStream(s) // returns “s” if it’s an input stream,
 convert is into a stream if possible, or return null
if(stream) { … } else { … }

 That’s a great way to convert a string into a stream, for example, in the
 case of an API that requires an InputStream and must integrate with older
 code that returns a String or a Blob.


Interesting. Maybe also accept an array of strings?

Thoughts behind the Streams API ED

[Takeshi Yoshino]

I'd like to summarize my ideas behind this API surface since the overlap
thread is too long. We'll put these into bug entries soon.

Feedback on Overlap thread, especially Issac's exhaustive list of
considerations and conversation with Aymeric were very helpful. In reply to
his mail, I drafted my initial proposal [2] in past which addresses almost
all of them. Since the API surface was so big, I tried to compact it while
incorporating Promises. Current ED [3] addresses not all but some of
important requirements. I think it's a good (re)starting point.

* Flow control
read() and write() in the ED does provide flow control by controlling the
timing of resolution of the returned Promise. A Stream would have a window
to limit data to be buffered in it. If a big value is passed as size
parameter of read(), it may extend the window if necessary.

In reading data as a DOMString, the size param of read() doesn't specify
exact raw size of data to be read out. It just works as throttle to prevent
internal buffer from being drained too fast. StreamReadResult tells how
many bytes were actually consumed.

If more explicit and precise flow control is necessary, we could cherry
pick some from my old big API proposal [1]. For example, making window size
configurable.

If it makes sense, size can be generalized to be cost of each element. It
would be useful when trying to generalize Stream to various objects.

To make window dynamically adjustable, we could introduce methods such as
drainCapacity(), expandCapacity() to it.

* Passive producer
Thinking of producers like a random number generator, it's not always good
to ask a producer to prepare data and push it to a Stream using write().
This was possible in [2], but not in the ED. This can be addressed for
example by adding one overload to write().
- write() and write(size) doesn't write data but wait until the Stream can
accept some amount or the specified size of data.

* Conversion from existing active and unstoppable producer API
E.g. WebSocket invokes onmessage immediately when new data is available.
For this kind of API, finite size Stream cannot absorb the production. So,
there'll be need to buffer read data manually. In [2], Stream always
accepted write() even if buffer is full assuming that if necessary the
producer should be using onwritable method.

Currently, only one write() can be issued concurrently, but we can choose
to have Stream queue write() requests in it.

* Sync read if possible
By adding sync flag to StreamReadResult and introducing StreamWriteResult
to signal if read was done sync (data is the actual result) or async (data
is a Promise) to save Promise post tasking cost.

I estimated that post tasking overhead should be negligible for bulk
reading, and when to read small fields, we often read some amount into
ArrayBuffer and then parse it directly. So it's currently excluded.

* Multiple consumers
pipe() can take multiple destination Streams. This allows for mirroring
Stream's output into two or more Streams. I also considered making Stream
itself consumable, but I thought it complicates API and implementation.
- Elements in internal buffer need to be reference counted.
- It must be able to distinguish consumers.

If one of consumers is fast and the other is slow, we need to wait for the
slower one before starting processing the rest in the original Stream. We
can choose to allow multiple consumers to address this by introducing a new
concept Cursor that represents reading context. Cursor can be implemented
as a new interface or Stream that refers to (and adds reference count to
elements of) the original Stream's internal buffer.

Needs some more study to figure out if context approach is really better
than pipe()-ing to new Stream instance.

* Splitting InputStream (ReadableStream) and OutputStream (WritableStream)
Writing part and reading part of the API can be split into two separate
interfaces. It's designed to allow for such decoupling. The constructor and
most of internal flags are to define a plain Stream. I'll give it a try
soon.

* StreamCosumeResult
I decided to have this interface for returning results since
- Notify EOF by just one read call if possible
- Tell the size of raw binary data consumed when readType=text

* read(n)
There're roughly two ways to encode structured data, length header based
and separator based. For the former, people basically don't want to get
notified when enough data is not ready. It's also inconvenient if we get
small ArrayBuffer chunks and need to concatenate them manually. For the
latter, call read() or read(n) if you need flow control.

Small chunk problem is also bothering for separator based protocol.
unshift() or peek() may help.

* In-band success/fail signaling
I excluded abort() kind method. Any error signal and other additional
information are conveyed manually outside of Stream. We can revisit this
point. If it's turned to be essential, we could put abort() back or add an
argument to close() to 

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Tue, Nov 5, 2013 at 2:11 AM, Takeshi Yoshino tyosh...@google.com wrote:

 Feedback on Overlap thread, especially Issac's exhaustive list of
 considerations


Deleted citation by mistake. [1] is the Issac's post.


 [1]
 http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0355.html

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

FYI, I added a branch named Preview version into which suggestions are
incorporated aggressively to see how the API surface would be change.
https://dvcs.w3.org/hg/streams-api/raw-file/tip/preview.html
Please take a look if you're interested in.

For stabler version edited after having discussion, check ED as usual.

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Wed, Nov 6, 2013 at 7:33 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  I have seen the different bugs too, some comments:

 - maybe I have missed some explaination or some obvious thing but I don't
 understand very well right now the difference/use between
 readable/writablebytestream and bytestream


ReadableByteStream and WritableByteStream are defining interfaces not only
for ByteStream but more generally for other APIs. For example, we discussed
how WebCrypto's encryption method should work with Stream concept recently,
and one idea you showed was making WebCrypto.subtle return an object (which
I called filter) to which we can pipe data. By defining a protocol how to
pass data to consumer as the WritableByteStream interface, we can reuse it
later for defining IDL for those filters. Similarly, ReadableByteStream can
provide uniform protocol how data producing APIs should communicate with
consumers.

ByteStream is now a class inheriting both ReadableByteStream and
WritableByteStream (sorry, I forgot to include inheritance info in the IDL).


 - pause/unpause: as far as I understand the whatwg spec does not recommend
 it but I don't understand the reasons. As I previously mentionned the idea
 is to INSERT a pause signal in the stream, you can not control the stream
 and therefore know when you are pausing it.


Maybe after decoupling the interface, pause/unpause are things to be added
to ByteStream? IIUC, pause prevents data from being read from a ByteStream,
and unpause removes the dam?


 - stop/resume: same, see my previous post, the difference is that the
 consuming API should clone the state of the operation and close the current
 operation as if eof was received, then restart from the clone on resume


Sorry that I haven't replied to your one.

Your post about those methods:
http://lists.w3.org/Archives/Public/public-webapps/2013OctDec/0343.html
WebCrypto ISSUE-22: http://www.w3.org/2012/webcrypto/track/issues/22

Maybe I still don't quite understand your ideas. Let me confirm.

stop() tells the consumer API implementing WritableByteStream that it
should behave as if it received EOF, but when resume() is called, restart
processing the data written between stop() call and resume() call as if the
API received data for the first time?

How should stop() work for ByteStream? ByteStream's read() method will
receive EOF at least once when all data written before stop() call has been
read, and it keeps returning EOF until resume() tells the ByteStream to
restart outputting?

I've been feeling that your use case is very specific to WebCrypto. Saving
state and restoring it sounds more like feature request for WebCrypto, not
a Stream.

But I'm a bit interested in what your stop()/resume() enables. With this
feature, ByteStream becomes message stream which is convenient for handling
WebSocket.


  - pipe []/fork: I don't see why the fast stream should wait for the slow
 one, so maybe the stream is forked and pause can be used for the slow one


There could be apps that want to limit memory usage strictly. We think
there're two strategies fork() can take.
a) wait until the slowest substream consumes
b) grow not to block the fastest substream while keeping data for the
slowest

a) is useful to limit memory usage. b) is more performance oriented.



 - flow control: could it be possible to advertise a maximum bandwidth rate
 for a stream?


It's currently communicated as window similar to TCP. Consumer can adjust
size argument of read() and frequency of read() call to match the
consumer's processing speed.

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  stop/resume:

 Indeed as I mentioned this is related to WebCrypto Issue22 but I don't
 think this is a unique case. Issue22 was closed because of lack of
 proposals to solve it, apparently I was the only one to care about it (but
 I saw recently some other messages that seem to be related), and finally
 this would involve a public clone method with associated security concerns.

 But with Streams it could be different, the application will internally
 clone the state of the operation probably eliminating the security issues,
 as simple as that.

 To describe simply the use case, let's take a progressive hash computing 4
 bytes by 4 bytes:

 incoming stream: ABCDE bytes
 hash operation: process ABCD, keep E for the next computation
 incoming stream: FGHI bytes + STOP-EOF
 hash operation: process EFGH, process STOP-EOF: clone the state of the
 hash, close the operation: digest hash with I


So, here, partial hash for ABCDEFGH is output



 resume:
 incoming stream: JKLF
 hash operation (clone): process IJKL, keep F for next computation
 etc...


and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P is
padding). Right?


  So you do not restart the operation as if it was the first time it was
 receiving data, you just continue it from the state it was when stop was
 received.

 That's not so unusual to do this, it has been requested many times in node.

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Thu, Nov 7, 2013 at 6:27 PM, Aymeric Vitte vitteayme...@gmail.comwrote:


 Le 07/11/2013 10:21, Takeshi Yoshino a écrit :

  On Thu, Nov 7, 2013 at 6:05 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  stop/resume:

 Indeed as I mentioned this is related to WebCrypto Issue22 but I don't
 think this is a unique case. Issue22 was closed because of lack of
 proposals to solve it, apparently I was the only one to care about it (but
 I saw recently some other messages that seem to be related), and finally
 this would involve a public clone method with associated security concerns.

 But with Streams it could be different, the application will internally
 clone the state of the operation probably eliminating the security issues,
 as simple as that.

 To describe simply the use case, let's take a progressive hash computing
 4 bytes by 4 bytes:

 incoming stream: ABCDE bytes
 hash operation: process ABCD, keep E for the next computation
 incoming stream: FGHI bytes + STOP-EOF
 hash operation: process EFGH, process STOP-EOF: clone the state of the
 hash, close the operation: digest hash with I


  So, here, partial hash for ABCDEFGH is output


 No, you get the digest for ABCDEFGHI and you get a cloned operation which
 will restart from ABCDEFGH



OK.





 resume:
 incoming stream: JKLF
 hash operation (clone): process IJKL, keep F for next computation
 etc...


  and if we close the stream here we'll get a hash for ABCDEFGHIJKLFPPP (P
 is padding). Right?


 If you close the stream here you get the digest for ABCDEFGHIJKLF


resume happens implicitly when new data comes in without explicit method
call say resume()?





  So you do not restart the operation as if it was the first time it was
 receiving data, you just continue it from the state it was when stop was
 received.

 That's not so unusual to do this, it has been requested many times in
 node.


 --
 Peersm : http://www.peersm.com
 node-Tor : https://www.github.com/Ayms/node-Tor
 GitHub : https://www.github.com/Ayms

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Fri, Nov 8, 2013 at 5:38 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Please see here https://github.com/whatwg/streams/issues/33, I realized
 that this would apply to operations like textDecoder too without the need
 of an explicit stream option, so that's no more WebCrypto only related.


Similar but a bit different?

For clarification, could you review the following?

textDecoderStream.write(araybuffer of 0xd0 0xa0 0xd0 0xbe 0xd1);
textDecoderStream.stop();
textDecoderStream.write(arraybuffer of 0x81 0xd1 0x81 0xd0 0xb8 0xd1 0x8f)

This generates DOMString stream [Рос, сия]. Right? Or you want to get
[Рос, Россия]?

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

Sorry. I've cut the input at wrong position.

textDecoderStream.write(arraybuffer of 0xd0 0xa0 0xd0 0xbe 0xd1 0x81 0xd1);
textDecoderStream.stop();
textDecoderStream.write(arraybuffer of 0x81 0xd0 0xb8 0xd1 0x8f)

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Fri, Nov 8, 2013 at 8:54 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

 I would expect Poc (stop, keep 0xd1 for the next data) and сия

 It can be seen a bit different indeed, while with crypto you expect the
 finalization of the operation since the begining (but only by computing the
 latest bytes), here you can not expect the string since the begining of
 course.

 It just depends how the Operation (here TextDecoder) handles stop but I
 find it very similar, TextEncoder closes the operation with the bytes it
 has and clone its state (ie do nothing here except clearing resolved
 bytes and keeping unresolved ones for data to come).


I'd say more generally that stop() is kinda in-band control signal that is
inserted between elements of the stream and distinguishable from the
elements. As you said, interpretation of the stop() symbol depends on what
the destination is.

One thing I'm still not sure is that I think you can just add stop()
equivalent method to the destination, and
- pipe() data until the point you were calling stop()
- call the stop() equivalent on e.g. hash
- restart pipe()

At least our spec allows for this. Of course, it's convenient that Stream
can carry such a signal. But there's trade-off between the convenience and
API size. Similar to decision whether we include abort() to
WritableByteStream or not.

Extremely, abort(), close() and stop() can be merged into one method
(unless abort() method has a functionality to abandon already written
data). They're all signal inserting methods.

close() - signal(FIN)
stop(info) - signal(CONTROL, info)
abort(error) - signal(ABORT, error)

and the signal is packed and inserted into the stream's internal buffer.

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

Aymeric,

Re: pause()/resume(),

I've moved flow control functionality for non-exact read() method to a
separate attribute pullAmount [1] [2]. pullAmount limits the max size of
data to be read by read() method. Currently the pipe() method is specified
not to respect pullAmount but we can choose to have it to respect
pullAmount, i.e. pausing pipe()'s transfer when pullAmount is set to 0.
Does this work for your use case?

[1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=23790
[2] https://dvcs.w3.org/hg/streams-api/rev/8a7f99536516

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Tue, Nov 12, 2013 at 5:23 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  No, see my previous reply, unless I am proven incorrect, I still think we
 should have:

 - pause/unpause
 - stop/(implicit resume)

 Regards,

 Aymeric

 Le 11/11/2013 22:06, Takeshi Yoshino a écrit :

  Aymeric,

  Re: pause()/resume(),


Sorry. This is typo. I meant pause()/unpause().



  I've moved flow control functionality for non-exact read() method to a
 separate attribute pullAmount [1] [2]. pullAmount limits the max size of
 data to be read by read() method. Currently the pipe() method is specified
 not to respect pullAmount but we can choose to have it to respect
 pullAmount, i.e. pausing pipe()'s transfer when pullAmount is set to 0.
 Does this work for your use case?

  [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=23790
 [2] https://dvcs.w3.org/hg/streams-api/rev/8a7f99536516


 --
 Peersm : http://www.peersm.com
 node-Tor : https://www.github.com/Ayms/node-Tor
 GitHub : https://www.github.com/Ayms

Re: Thoughts behind the Streams API ED

[Takeshi Yoshino]

On Tue, Nov 12, 2013 at 5:20 PM, Aymeric Vitte vitteayme...@gmail.comwrote:

  Takeshi,

 See discussion here too: https://github.com/whatwg/streams/issues/33

 The problem with stop again is that I need to handle myself the clone
 operations, the advantage of stop-eof is:

 - clone the operation
 - close it
 - restart from the clone

 And as I mentioned before this would work for any operation that has
 unresolved bytes (TextDecoder, etc) without the need of modifying the
 operation API for explicit clones or options.


OK. I understood your needs.

As explained in the previous mail, if we decide to take in your suggestion,
I'd implement your stop-eof as a more generic in-band control signal. In
other words, changing the draft to propose message stream than byte stream.

I estimate that this feature sophisticates implementation much. So, I want
to hear more support and justification.

For now, I filed a bug to track discussion.
https://www.w3.org/Bugs/Public/show_bug.cgi?id=23799

Re: CfC: publish Proposed Recommendation of Progress Events; deadline November 25

[Takeshi Yoshino]

two minor comments
- add semicolons to lines of the example code in the introduction section?
- 2nd paragraph in the conformance section, quote must?

Re: Request for feedback: Streams API

[Takeshi Yoshino]

Thanks. ByteStream is already partially implemented in Blink/Chromium. As
one of implementors, I'll continue prototyping and share issues here.

I haven't got time for, but writing some polyfill might be also good thing
to do.

Takeshi


On Thu, Dec 5, 2013 at 2:38 AM, Kenneth Russell k...@google.com wrote:

 Looks great! Seems very well thought through.

 The API seems large enough that it would be worth prototyping it and
 writing test applications to make sure it addresses key use cases
 before finalizing the spec.

 -Ken


 On Wed, Dec 4, 2013 at 8:27 AM, Feras Moussa feras.mou...@hotmail.com
 wrote:
  The editors of the Streams API have reached a milestone where we feel
 many
  of the major issues that have been identified thus far are now resolved
 and
  incorporated in the editors draft.
 
  The editors draft [1] has been heavily updated and reviewed the past few
  weeks to address all concerns raised, including:
  1. Separation into two distinct types -ReadableByteStream and
  WritableByteStream
  2. Explicit support for back pressure management
  3. Improvements to help with pipe( ) and flow-control management
  4. Updated spec text and diagrams for further clarifications
 
  There are still a set of bugs being tracked in bugzilla. We would like
  others to please review the updated proposal, and provide any feedback
 they
  may have (or file bugs).
 
  Thanks.
  -Feras
 
 
  [1] https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

Re: Comments on version web-apps specs from 2013-10-31

[Takeshi Yoshino]

Thanks for feedback.

On Thu, Nov 21, 2013 at 4:56 PM, Feras Moussa feras.mou...@hotmail.comwrote:

 Hi Francois,
 Thanks for the feedback.


  From: francois-xavier.kowal...@hp.com
  To: public-webapps@w3.org
  Date: Wed, 20 Nov 2013 20:30:47 +
  Subject: Comments on version web-apps specs from 2013-10-31

 
  Hello
 
  I have a few comments on:
 https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm from
 2013-10-31. Apologies wether it is not the latest version: It took me some
 time to figure-out where was the right forum to send these comments to.
 
  Section 2.1.3:
  1. Close(): For writeable streams, the close() method does not provide a
 data-completion hook (all-data-flushed-to-destination), unlike the close
 method that resolved the Promise returned by read().
 The version of the spec you linked doesn't differentiate
 writeable/readable streams, but is something we are considering adding in a
 future version. I don't quite understand what you're referring to here.
 close is independent of future reads - you can call a read after close, and
 once EOF is reached, the promise is resolved and you get a result with
 eof=true.


writeClose() we have now is still void.

In current API, fulfillment of writePromise doesn't necessarily mean
acknowledgement (data written has been successfully processed) but
implmentor may give such a meaning to it in the following two ways:
- fulfill writePromise when the written data is successfully consumed
- make writeClose return a Promise, say closePromise. writePromise may be
fulfilled before processing but writeClose is fulfilled only when all the
data written has been successfully consumed

I think it makes sense to change writeClose() type to Promiseundefined so
that derived classes may choose to use it to notify the writer of success.


  2. Pipe(): the readable Steam (the one that provides the pipe() method)
 is neutered in case of I/O error, but the state change of the writeable
 stream is not indicated. What if the write operation failed?
 Are you asking what the chain of error propagation is when multiple
 streams are chained?


Error handling is not yet well documented but I plan to make write
operation failure result in rejection of pipePromise.


 
  Section 3.2:
  1. Shouldn't a FileWrite also be capable of consuming a Stream? (Like
 XHR-pipe-to-FileWriter)
 Yes, I think so - this is a use case we can add.

 Added to the possible consumer list.


  2. Shouldn't an XMLHttpRequest also be capable of consuming a Stream?
 (eg. chunked PUT/POST)?
 Section 5.4 covers this - support for posting a Stream. That said, this is
 a section I think will need to be flushed out more.


Added in recent commit.


 
  br.
 
  —FiX
 
  PS: My request to join this group is pending, so please Cc me in any
 reply/comment you may have until membership is fixed.
 
 

Re: Request for feedback: Streams API

[Takeshi Yoshino]

Hi,

Implementation of back pressure is important for handling large data
stably. The Streams need to ask the consumer code to notify it when it can
consume more data. Triggering this signal by method invocation is one of
possible options. Promises fit this well.

To address smaug___'s concern, we could choose not to 1-to-1 correspond
delivery of data and pull operation. Deliver data using callback
invocation, while allowing the consuming code to tell amount it want to
pull via pullAmount. Like WritableByteStream allows write() ignoring back
pressure, we allow ReadableByteStream to push available data to consuming
code ignoring back pressure. As far as the producer inside
WritableByteStream does work respecting pullAmount, back pressure mechanism
should still work well.

One concern Domenic showed in the chat will remain un-addressed. That is an
issue with callback setting timing. If we take Promise read() approach,
arrived data never vanishes into the void. I understand this
error-proneness.

Takeshi


On Mon, Dec 16, 2013 at 9:21 PM, Olli Pettay olli.pet...@helsinki.fiwrote:

 On 12/04/2013 06:27 PM, Feras Moussa wrote:

 The editors of the Streams API have reached a milestone where we feel
 many of the major issues that have been identified thus far are now
 resolved and
 incorporated in the editors draft.

 The editors draft [1] has been heavily updated and reviewed the past few
 weeks to address all concerns raised, including:
 1. Separation into two distinct types -ReadableByteStream and
 WritableByteStream
 2. Explicit support for back pressure management
 3. Improvements to help with pipe( ) and flow-control management
 4. Updated spec text and diagrams for further clarifications

 There are still a set of bugs being tracked in bugzilla. We would like
 others to please review the updated proposal, and provide any feedback they
 may
 have (or file bugs).

 Thanks.
 -Feras


 [1] https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm



 So per https://www.w3.org/Bugs/Public/show_bug.cgi?id=24054
 it is not clear to me why the API is heavily Promise based.
 Event listeners tend to work better with stream like APIs.


 (The fact the Promises are hip atm is not a reason to use them for
 everything ;) )

 -Olli

Re: [promises] Guidance on the usage of promises for API developers

[Takeshi Yoshino]

Nice writing! Both the shorthand phrases and guidance look very useful for
writing Promise based specs.

I have only one comment on this section.
https://github.com/domenic/promises-unwrapping/blob/master/docs/writing-specifications-with-promises.md#maintain-a-normal-control-flow

I agree with your point in the first paragraph, but the suggested way looks
rather confusing. The asynchronous operation abstracted by Promises is well
described, but selling this convention (writing what usually happens later
before return without any annotation) to everywhere sounds too much, I
think. It's good to move Return p to the end of the steps. But how about
also keeping do blah asynchronously text?

1. Let p be a newly-created promise.
2. These steps will be run asynchronously.
  1. If blah, reject p with undefined.
  2. If blah, resolve p with foobar.
3. Return p.

Thanks

Re: [promises] Guidance on the usage of promises for API developers

[Takeshi Yoshino]

To address, jgraham's point, let's go back to just use Run ... style
phrasing.

I agree that we shouldn't mislead readers that task queuing is always
necessary. Moreover, I think algorithm may be allowed to be done either
synchronously or asynchronously for some cases. Maybe we could add
possibly to mean it's not forced to be run later for such a case?

How about introducing some markups such as [Async] and a note of detailed
explanation (no need to queue a task, etc.)? Readers are guided to read the
note to understand what they mean, and actually more likely to do so
compared to just writing do X asynchronously. I imagine. Not sure.

1. sync_work_A.
2. sync_work_B.
3. Let p be a newly-created promise.
4. [PossiblyAsync] possibly_async_work_S.
5. [PossiblyAsync] possibly_async_work_T.
6. [Async] async_work_X.
7. [Async] async_work_Y.
8. Return p.

Right after returned from this method, we observe side effect of sync_work
and may observe possibly_async_work's but not async_work's.

If too noisy, modularize as Boris said (
http://lists.w3.org/Archives/Public/public-webapps/2014JanMar/0066.html).

Re: [promises] Guidance on the usage of promises for API developers

[Takeshi Yoshino]

New texts look nice. Two comments...

- XMLHttpRequest.send() is a good example, but as we go through multiple
states and invokes onreadystatechange multiple times, I prefer not just
quoting onreadystatechange but point the one-and-done aspect of XHR by
using phrase like which triggers onreadystatechange with readyState of
DONE
- Could you please refrain from pointing the streams API or list both in
the doc until we finish the discussion now we have?

Re: Fetch API

[Takeshi Yoshino]

http://fetch.spec.whatwg.org/#dom-request
Add steps to set client and context?

http://fetch.spec.whatwg.org/#cors-preflight-fetch-0
Add steps to set client and context?

http://fetch.spec.whatwg.org/#concept-legacy-fetch
http://fetch.spec.whatwg.org/#concept-legacy-potentially-cors-enabled-fetch
Steps to set url, client and context are missing here too. But not worth
updating this section anymore?

Termination reason is not handled intentionally (only supposed to be used
by XHR's functionalities and nothing would be introduced for Fetch API?)?

The promise is rejected with a TypeError which seems inconsistent with XHR.
Is this intentional?

Re: Fetch API

[Takeshi Yoshino]

On Mon, Jun 2, 2014 at 6:59 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, May 29, 2014 at 4:25 PM, Takeshi Yoshino tyosh...@google.com
 wrote:
  http://fetch.spec.whatwg.org/#dom-request
  Add steps to set client and context?

 That happens as part of the restricted copy. However, that might
 still change around a bit.


Ah, ok.


  http://fetch.spec.whatwg.org/#cors-preflight-fetch-0
  Add steps to set client and context?

 That's an internal algorithm never directly used. You can only get
 there from http://fetch.spec.whatwg.org/#concept-fetch and that can
 only be reached through an API such as fetch().


Right. But preflight's client and context are not initialized before
invoking HTTP fetch (http://fetch.spec.whatwg.org/#concept-http-fetch).
client is referred in HTTP fetch.

BTW, handle a fetch is a dead link.
https://slightlyoff.github.io/ServiceWorker/spec/service_worker/handle-a-fetch


  The promise is rejected with a TypeError which seems inconsistent with
 XHR.
   Is this intentional?

 Yes. I wanted to stick to JavaScript exceptions. However, I suspect at
 some point once we have FormData integration and such there might be
 quite a bit of dependencies on DOM in general, so maybe that is moot.


Got it. Thanks.

Re: Fetch API

[Takeshi Yoshino]

For XHR.send(), we've finally chosen to accept only ArrayBufferView.
http://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0141.html

Do we want to do the same for FetchBody body of RequestInit?

Re: XMLHttpRequest: uppercasing method names

[Takeshi Yoshino]

On Tue, Aug 12, 2014 at 10:55 PM, Anne van Kesteren ann...@annevk.nl
wrote:

 On Tue, Aug 12, 2014 at 3:37 PM, Brian Kardell bkard...@gmail.com wrote:
  If there's no really good reason to change it, least change is better IMO

 All I can think of is that it would be somewhat more consistent to not
 have this list and always uppercase,


Ideally


 but yeah, I guess I'll just align
 fetch() with XMLHttpRequest.


Isn't it an option that we use stricter rule (all uppercase) for the
newly-introduced fetch() API but keep the list for XHR? Aligning XHR and
fetch() is basically good but making fetch() inherit the whitelist is a
little sad.



Some archaeology:

- Blink recently reduced the whitelist to conform to the latest WHATWG XHR
spec. http://src.chromium.org/viewvc/blink?view=revisionrevision=176592
- Before that, used this list ported to WebKit from Firefox's behavior
http://trac.webkit.org/changeset/13652/trunk/WebCore/xml/xmlhttprequest.cpp
- Anne introduced the initial version of the part of the spec in Aug 2006
http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.html.diff?r1=1.12;r2=1.13;f=h
-- http://lists.w3.org/Archives/Public/public-webapi/2006Apr/0124.html
-- http://lists.w3.org/Archives/Public/public-webapi/2006Apr/0126.html

Re: XMLHttpRequest: uppercasing method names

[Takeshi Yoshino]

On Wed, Aug 13, 2014 at 12:15 AM, Brian Kardell bkard...@gmail.com wrote:


 On Aug 12, 2014 11:12 AM, Takeshi Yoshino tyosh...@google.com wrote:
 
  On Tue, Aug 12, 2014 at 10:55 PM, Anne van Kesteren ann...@annevk.nl
 wrote:
 
  On Tue, Aug 12, 2014 at 3:37 PM, Brian Kardell bkard...@gmail.com
 wrote:
   If there's no really good reason to change it, least change is better
 IMO
 
  All I can think of is that it would be somewhat more consistent to not
  have this list and always uppercase,
 
 
  Ideally
 
 
  but yeah, I guess I'll just align
  fetch() with XMLHttpRequest.
 
 
  Isn't it an option that we use stricter rule (all uppercase) for the
 newly-introduced fetch() API but keep the list for XHR? Aligning XHR and
 fetch() is basically good but making fetch() inherit the whitelist is a
 little sad.
 
 
 
  Some archaeology:
 
  - Blink recently reduced the whitelist to conform to the latest WHATWG
 XHR spec.
 http://src.chromium.org/viewvc/blink?view=revisionrevision=176592
  - Before that, used this list ported to WebKit from Firefox's behavior
 http://trac.webkit.org/changeset/13652/trunk/WebCore/xml/xmlhttprequest.cpp
  - Anne introduced the initial version of the part of the spec in Aug
 2006
 http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.html.diff?r1=1.12;r2=1.13;f=h
  -- http://lists.w3.org/Archives/Public/public-webapi/2006Apr/0124.html
  -- http://lists.w3.org/Archives/Public/public-webapi/2006Apr/0126.html
 

 fetch should explain magic in XMLHttpRequest et all.. I don't see how it
 could differ in the way you are suggesting and match

Which do you mean by fetch? http://fetch.spec.whatwg.org/#dom-global-fetch
or http://fetch.spec.whatwg.org/#concept-fetch.

fetch() and XHR share the fetch algorithm but have different bootstrap and
hooks.

Re: XMLHttpRequest: uppercasing method names

[Takeshi Yoshino]

On Wed, Aug 13, 2014 at 1:05 AM, Anne van Kesteren ann...@annevk.nl wrote:

 On Tue, Aug 12, 2014 at 5:15 PM, Brian Kardell bkard...@gmail.com wrote:
  fetch should explain magic in XMLHttpRequest et all.. I don't see how it
  could differ in the way you are suggesting and match

 Well fetch() will never be able to explain synchronous fetches. But in
 general I agree that it would be sad if XMLHttpRequest could do more
 than fetch(). Per bz it seems we should just align fetch() with
 XMLHttpRequest and call it a day.


wfm

seems Safari is also going to be compliant to the latest spec.
https://bugs.webkit.org/show_bug.cgi?id=134264



 --
 http://annevankesteren.nl/

[Streams API] Add load-using-streams functionality to XHR or only to Fetch API?

[Takeshi Yoshino]

We're implementing Streams API [1] and integration of it and XHR [2]
experimentally in Blink [3][4].

Anne suggested that we look into adding new fetch-layer features to the
Fetch API [5][6] rather than XHR in Blink mailing list (blink-dev) [6].
There's a concern if we can ship the Fetch API to non-ServiceWorker scope
soon [7].

I'd like to hear your opinions on these issues:
- add the feature only to the Fetch, or both XHR and Fetch?
- how the integration should be done?

Thanks


[1] https://github.com/whatwg/streams
[2] http://xhr.spec.whatwg.org/
[3]
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/core/streams/
[4] https://github.com/tyoshino/streams_integration/blob/master/README.md
[5] http://fetch.spec.whatwg.org/#fetch-method
[6] http://fetch.spec.whatwg.org/#body-stream-concept
[7]
https://groups.google.com/a/chromium.org/d/msg/blink-dev/GoFbe0yLO50/SXpYMdYn0A4J
[8]
https://groups.google.com/a/chromium.org/d/msg/blink-dev/GoFbe0yLO50/vviGEQ5Z-KoJ

Re: [Streams API] Add load-using-streams functionality to XHR or only to Fetch API?

[Takeshi Yoshino]

Streams API itself still has many issues, right.

We'd like to continue prototyping in Blink anyway to provide feedback to
the spec as browser developer's point of view, but if we're sure that we
won't add it to XHR, we want to stop the work on XHR (both spec and impl)
at some early stage after figuring out very basic issues and focus on the
Fetch.

Takeshi


On Thu, Aug 14, 2014 at 4:03 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, Aug 14, 2014 at 6:20 AM, Takeshi Yoshino tyosh...@google.com
 wrote:
  We're implementing Streams API and integration of it and XHR
  experimentally in Blink.

 I think the question is not so much how far along XMLHttpRequest and
 fetch() are, but how far along streams is. E.g. a chat I had yesterday
 http://krijnhoetmer.nl/irc-logs/whatwg/20140813#l-592 suggests there's
 still many things to sort through.


 --
 http://annevankesteren.nl/

Re: [Streams API] Add load-using-streams functionality to XHR or only to Fetch API?

[Takeshi Yoshino]

On Thu, Aug 14, 2014 at 4:29 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, Aug 14, 2014 at 9:19 AM, Takeshi Yoshino tyosh...@google.com
 wrote:
  We'd like to continue prototyping in Blink anyway to provide feedback to
 the
  spec as browser developer's point of view, but if we're sure that we
 won't
  add it to XHR, we want to stop the work on XHR (both spec and impl) at
 some
  early stage after figuring out very basic issues and focus on the Fetch.

 I recommend putting all your energy on Fetch and Streams then. We can
 still add features to XMLHttpRequest if there's compelling reasons
 (e.g. we hit deployment issues with fetch()), but we should not put
 our focus there.


Thanks. I'd like to hear opinions from others too.



BTW, Anne, as the fetch() is planned to be available in non-ServiceWorker
scopes, could you please do either of:
a) discuss issues about fetch() at public-webapps@
b) add a link to https://github.com/slightlyoff/ServiceWorker/labels/fetch
in http://fetch.spec.whatwg.org/ as well as the link to public-webapps@

I guess you don't want to do (a). (b) is fine. Then, readers of
http://fetch.spec.whatwg.org/ can easily know that discussions about
fetch() are happening at the ServiceWorker issue tracker even if he/she is
not familiar with ServiceWorker, and join it if want.

Re: File API: reading a Blob

[Takeshi Yoshino]

On Thu, Sep 11, 2014 at 8:47 AM, Aymeric Vitte vitteayme...@gmail.com
wrote:

  Does your experimentation pipe the XHR stream to MSE? Obviously that
 should be the target for yt, this would be a first real application of the
 Streams API.


It's not yet updated to use the new Streams. Here's our layout test for
MSE. responseType = 'legacystream' makes the XHR return the old version of
the stream.

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-append-legacystream.htmlq=createMediaXHRsq=package:chromiumtype=csl=12

You can find the following call in the file.

sourceBuffer.appendStream(xhr.response);



 Because the Streams API does not define how to apply back pressure to XHR,
 but does define how to apply back pressure between XHR and MSE.

 Probably the spec should handle on a per case basis what should be the
 behavior in term of back pressure for things coming from outside of the
 browser (xhr, websockets, webrtc, etc , not specified as far as I know) and
 for things going on inside the browser (already specified)

 Le 08/09/2014 06:58, Takeshi Yoshino a écrit :

  On Thu, Sep 4, 2014 at 7:02 AM, Aymeric Vitte vitteayme...@gmail.com
 wrote:

 The fact is that most of the W3C groups impacted by streams (File,
 indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream,
 etc, I must forget a lot here) seem not to care a lot about it and maybe
 just expect streams to land in the right place in the APIs when they are
 available, by some unknown magic.

 I still think that the effort should start from now for all the APIs (as
 well as the implementation inside browsers, which apparently has started
 for Chrome, but Chrome was supposed to have started some implementation of
 the previous Streams APIs, so it's not very clear), and that it should be
 very clearly synchronized, disregarding vague assumptions from the groups
 about low/high level and Vx releases, eluding the issue.


  Chrome has an experimental implementation [1] of the new Streams API [2]
 integrated with XHR [3] behind a flag.

  We receive data from the browser process over IPC (both network and blob
 case). The size of data chunks and arrival timing depend on various
 factors. The received chunks are passed to the XMLHttpRequest class on the
 same thread as JavaScript runs. We create a new instance of ReadableStream
 [4] on arrival of the first chunk. On every chunk arrival, we create an
 ArrayBuffer from the chunk and then call [[enqueue]](chunk) [5] equivalent
 C++ function to put it into the ReadableStream.

  The ReadableStream is available from the response attribute in the
 LOADING and DONE state (if no error). The chunks pushed to the
 ReadableStream become available for read immediately.

  Any problem occurs while loading data from network/blob, we call
 [[error]](e) [6] equivalent C++ function with an exception as defined in
 the XHR spec for sync XHR.

  Currently, XMLHttpRequest doesn't exert any back pressure. We plan to do
 something not to read too much data from disk/network. It might be worth
 specifying something about the flow control in the abstract read from
 blob/network operation at standard level.

  [1]
 https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/xmlhttprequest/response-stream.html
  [2] https://github.com/whatwg/streams
 https://github.com/whatwg/streams#readablestream
 [3] https://github.com/tyoshino/streams_integration/
  [4] https://github.com/whatwg/streams#readablestream
 [5] https://github.com/whatwg/streams#enqueuechunk
  [6] https://github.com/whatwg/streams#errore



 --
 Peersm : http://www.peersm.com
 torrent-live: https://github.com/Ayms/torrent-live
 node-Tor : https://www.github.com/Ayms/node-Tor
 GitHub : https://www.github.com/Ayms

Re: File API: reading a Blob

[Takeshi Yoshino]

+Aaron

On Thu, Sep 11, 2014 at 6:48 PM, Aymeric Vitte vitteayme...@gmail.com
wrote:

  But I suppose that should be one of the first use case for Google to
 introduce streams with MSE, no?


We're (and will) sharing updates about Streams with MSE folk (Aaron). MSE
side would be updated once Streams API becomes stable enough for them to
start revising.


 To be more clear about what I mean by back pressure for things coming
 from outside of the browser:

 - XHR: the Streams API should define how xhr gets chunks using Range
 according to the flow and adapt accordingly transparently for the users


We're not thinking of involving multiple HTTP transactions for one send().
That's beyond current XHR semantics. I think even if we implement that kind
of feature, we should add it as a new mode or API, not as a new
responseType.

What we plan to do is connecting the Streams API's back pressure with TCP
congestion control. Since socket API doesn't expose much details to user,
we'll just control frequency and the size of buffer to pass to read(2).


 - WebSockets: use something like bufferedAmount but that can be notified
 to the users, so the users can adapt the flow, currently bufferedAmount is
 not extremely usefull since you might need to do some polling to check it.


For WebSockets, there're two approaches of Streams integration.
ReadableStream of messages or ReadableStream of ArrayBuffer representing
chunks of the payload of a message.

We need to discuss that first.



 Le 11/09/2014 08:36, Takeshi Yoshino a écrit :

  On Thu, Sep 11, 2014 at 8:47 AM, Aymeric Vitte vitteayme...@gmail.com
 wrote:

  Does your experimentation pipe the XHR stream to MSE? Obviously that
 should be the target for yt, this would be a first real application of the
 Streams API.


  It's not yet updated to use the new Streams. Here's our layout test for
 MSE. responseType = 'legacystream' makes the XHR return the old version of
 the stream.


 https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/media/media-source/mediasource-append-legacystream.htmlq=createMediaXHRsq=package:chromiumtype=csl=12

  You can find the following call in the file.

  sourceBuffer.appendStream(xhr.response);



 Because the Streams API does not define how to apply back pressure to
 XHR, but does define how to apply back pressure between XHR and MSE.

 Probably the spec should handle on a per case basis what should be the
 behavior in term of back pressure for things coming from outside of the
 browser (xhr, websockets, webrtc, etc , not specified as far as I know) and
 for things going on inside the browser (already specified)

 Le 08/09/2014 06:58, Takeshi Yoshino a écrit :

  On Thu, Sep 4, 2014 at 7:02 AM, Aymeric Vitte vitteayme...@gmail.com
 wrote:

 The fact is that most of the W3C groups impacted by streams (File,
 indexedDB, MSE, WebRTC, WebCrypto, Workers, XHR, WebSockets, Media Stream,
 etc, I must forget a lot here) seem not to care a lot about it and maybe
 just expect streams to land in the right place in the APIs when they are
 available, by some unknown magic.

 I still think that the effort should start from now for all the APIs (as
 well as the implementation inside browsers, which apparently has started
 for Chrome, but Chrome was supposed to have started some implementation of
 the previous Streams APIs, so it's not very clear), and that it should be
 very clearly synchronized, disregarding vague assumptions from the groups
 about low/high level and Vx releases, eluding the issue.


  Chrome has an experimental implementation [1] of the new Streams API
 [2] integrated with XHR [3] behind a flag.

  We receive data from the browser process over IPC (both network and
 blob case). The size of data chunks and arrival timing depend on various
 factors. The received chunks are passed to the XMLHttpRequest class on the
 same thread as JavaScript runs. We create a new instance of ReadableStream
 [4] on arrival of the first chunk. On every chunk arrival, we create an
 ArrayBuffer from the chunk and then call [[enqueue]](chunk) [5] equivalent
 C++ function to put it into the ReadableStream.

  The ReadableStream is available from the response attribute in the
 LOADING and DONE state (if no error). The chunks pushed to the
 ReadableStream become available for read immediately.

  Any problem occurs while loading data from network/blob, we call
 [[error]](e) [6] equivalent C++ function with an exception as defined in
 the XHR spec for sync XHR.

  Currently, XMLHttpRequest doesn't exert any back pressure. We plan to
 do something not to read too much data from disk/network. It might be worth
 specifying something about the flow control in the abstract read from
 blob/network operation at standard level.

  [1]
 https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/http/tests/xmlhttprequest/response-stream.html
  [2] https://github.com/whatwg/streams
 https

Re: [streams-api] Seeking status of the Streams API spec

[Takeshi Yoshino]

Re: establishing integration plan for the consumers and producers listed in
the W3C spec, we haven't done anything than what Domenic introduced in this
thread.

I wrote some draft of XHR+ReadableStream integration spec and is
implemented on Chrome, but the plan is not to ship it but wait for the
Fetch API as discussed at WHATWG.


On Wed, Oct 15, 2014 at 9:10 AM, Paul Cotton paul.cot...@microsoft.com
wrote:

 This thread was recently re-started at
 http://lists.w3.org/Archives/Public/public-webapps/2014OctDec/0084.html

 Domenic's latest document is at https://streams.spec.whatwg.org/  The W3C
 document has NOT been updated since
 http://www.w3.org/TR/2013/WD-streams-api-20131105/ .


Not to confuse people, too late but I replaced the W3C Streams API spec WD
with a pointer to the WHATWG Streams spec and a few sections discussing
what we should add to the spec for browser use cases.



 /paulc

 Paul Cotton, Microsoft Canada
 17 Eleanor Drive, Ottawa, Ontario K2E 6A3
 Tel: (425) 705-9596 Fax: (425) 936-7329


 -Original Message-
 From: Jerry Smith (WINDOWS)
 Sent: Tuesday, October 14, 2014 8:03 PM
 To: Domenic Denicola; Aaron Colwell
 Cc: Anne van Kesteren; Paul Cotton; Takeshi Yoshino; public-webapps;
 Arthur Barstow; Feras Moussa; public-html-me...@w3.org
 Subject: RE: [streams-api] Seeking status of the Streams API spec

 Where is the latest Streams spec?
 https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm doesn't have
 much about WritableStreams.

 Jerry

 -Original Message-
 From: Domenic Denicola [mailto:dome...@domenicdenicola.com]
 Sent: Tuesday, October 14, 2014 10:18 AM
 To: Aaron Colwell
 Cc: Anne van Kesteren; Paul Cotton; Takeshi Yoshino; public-webapps;
 Arthur Barstow; Feras Moussa; public-html-me...@w3.org
 Subject: RE: [streams-api] Seeking status of the Streams API spec

 From: Aaron Colwell [mailto:acolw...@google.com]

  MSE is just too far along, has already gone through a fair amount of
 churn, and has major customers like YouTube and Netflix that I just don't
 want to break or force to migrate...again.

 Totally understandable.

  I haven't spent much time looking at the new Stream spec so I can't
 really say yet whether I agree with you or not. The main reason why people
 wanted to be able to append a stream is to handle larger, open range,
 appends without having to make multiple requests or wait for an XHR to
 complete before data could be appended. While I understand that you had
 your reasons to expand the scope of Streams to be more general, MSE really
 just needs them as a handle to route bytes being received with XHR to the
 SourceBuffer w/o having to actually surface them to JS. It would be really
 unfortunate if this was somehow lost in the conversion from the old spec.

 The way to do this in Streams is to pipe the fetch stream to a writable
 stream:

 fetch(url)
   .then(response = response.body.pipeTo(writableStream).closed)
   .then(() = console.log(all data written!))
   .catch(e = console.log(error fetching or piping!, e));

 By piping between two UA-controlled streams, you can establish an
 off-main-thread relationship between them. This is why it would be ideal
 for SourceBuffer (or a future alternative to it) to be WritableStream,
 especially given that it already has abort(), appendBuffer(), and various
 state-like properties that are very similar to what a WritableStream
 instance has. The benefit here being that people could then use
 SourceBuffers as generic destinations for any writable-stream-accepting
 code, since piping to a writable stream is idiomatic.

 But that said, given the churn issue I can understand it not being
 feasible or desirable to take that path.

  Perhaps, although I expect that there may be some resistance to dropping
 this at this point. Media folks were expecting the Streams API to progress
 in such a way that would at least allow appendStream() to still work
 especially since it only needs a stream for recieving bytes. I'll dig into
 the latest Streams spec so I can better understand the current state of
 things.

 One speculative future evolution would be for SourceBuffer to grow a
 `.stream` or `.writable` property, which exposes the actual stream. Then
 appendStream could essentially be redefined as

 SourceBuffer.prototype.appendStream = function (readableStream) {
   readableStream.pipeTo(this.writable);
 };


Needs porting some part of Prepare Append Algorithm in order not to run
two or more pipeTo at the same time?


 and similarly appendBuffer could be recast in terms of
 `this.writable.write`, etc. Then developers who wish to treat the
 SourceBuffer as a more generic writable stream can just do
 `myGenericWritableStreamLibrary(mediaSource.sourceBuffers[0].writable)` or
 similar.

Re: [streams-api] Seeking status of the Streams API spec

[Takeshi Yoshino]

Hi Arthur,

OK. Since I hurried, there're some odd texts left. Fixed:
https://dvcs.w3.org/hg/streams-api/rev/891635210233


On Tue, Oct 21, 2014 at 10:53 PM, Arthur Barstow art.bars...@gmail.com
wrote:

 On 10/14/14 11:06 PM, Takeshi Yoshino wrote:

 Not to confuse people, too late but I replaced the W3C Streams API spec
 WD with a pointer to the WHATWG Streams spec and a few sections discussing
 what we should add to the spec for browser use cases.


 Takeshi - given the magnitude of the changes in [changeset], a new WD
 should be published. I'll start a related PSA targeting a publication on
 October 23.

 (I'll start working on the draft WD if you don't have the time.)

 -Thanks, AB

 [changeset] https://dvcs.w3.org/hg/streams-api/rev/e5b689ded0d6

Re: [streams] Seeking status and plans [Was: [TPAC2014] Creating focused agenda]

[Takeshi Yoshino]

On Fri, Oct 24, 2014 at 8:24 PM, Arthur Barstow art.bars...@gmail.com
wrote:

 On 10/23/14 1:26 PM, Domenic Denicola wrote:

 From: Arthur Barstow [mailto:art.bars...@gmail.com]

  I think recent threads about Streams provided some useful information
 about the status and plans for Streams. I also think it could be useful if
 some set of you were available to answer questions raised at the meeting.
 Can any of you commit some time to be available? If so, please let me know
 some time slots you are available. My preference is Monday morning, if
 possible.

 I'd be happy to call in at that time or another. Just let me know so I
 can put it in my calendar.


 OK, I just added Streams to the 11:00-11:30 slot on Monday Oct 27:

 https://www.w3.org/wiki/Webapps/November2014Meeting#
 Agenda_Monday_October_27

 IRC and Zakim phone bridge info is:

 https://www.w3.org/wiki/Webapps/November2014Meeting#Meeting_Logistics

 Takeshi, Feras - please do join the call.

 If members of the p-html-media group want to join, that should be fine.


Thanks.

Me too over telephone and IRC.

Takeshi

Re: =[xhr]

[Takeshi Yoshino]

We're adding Streams API https://streams.spec.whatwg.org/ based response
body receiving feature to the Fetch API

See
- https://github.com/slightlyoff/ServiceWorker/issues/452
- https://github.com/yutakahirano/fetch-with-streams

Similarly, using WritableStream + Fetch API, we could allow for sending
partial chunks. It's not well discussed/standardized yet. Please join
discussion there.

Takeshi

On Sat, Nov 15, 2014 at 3:49 AM, Rui Prior rpr...@dcc.fc.up.pt wrote:

 AFAIK, there is currently no way of using XMLHttpRequest to make a POST
 using Transfer-Encoding: Chunked.  IMO, this would be a very useful
 feature for repeatedly sending short messages to a server.

 You can always make POSTs repeatedly, one per chunk, and arrange for the
 server to glue the chunks together, but for short messages this
 process adds a lot of overhead (a full HTTP request per chunk, with full
 headers for both the request and the response).  Another option would
 using websockets, but the protocol is no longer HTTP, which increases
 complexity and may bring some issues.

 Chunked POSTs using XMLHttpRequest would be a much better option, were
 they available.  An elegant way of integrating this feature in the API
 would be adding a second, optional boolean argument to the send()
 method, defaulting to false, that, when true, would trigger chunked
 uploading;  the last call to send() would have that argument set to
 true, indicating the end of the object to be uploaded.

 Is there any chance of such feature getting added to the standard in the
 future?

 Rui Prior

Re: =[xhr]

[Takeshi Yoshino]

On Tue, Nov 18, 2014 at 12:11 AM, Anne van Kesteren ann...@annevk.nl
wrote:

 On Mon, Nov 17, 2014 at 3:50 PM, Domenic Denicola d...@domenic.me wrote:
  What do we think of that kind of behavior for fetch requests?

 I'm not sure we want to give a potential hostile piece of script that
 much control over what goes out. Being able to lie about
 Content-Length would be a new feature that does not really seem
 desirable. Streaming should probably imply chunked given that.


Agreed.

stream.write(new ArrayBuffer(1024));
 setTimeout(() = stream.write(new ArrayBuffer(1024)), 100);
 setTimeout(() = stream.write(new ArrayBuffer(1024)), 200);
 setTimeout(() = stream.close(), 300);


And, for abort(), underlying transport will be destroyed. For TCP FIN
without last-chunk. For http2 maybe RST_STREAM with INTERNAL_ERROR? Need
consult httpbis.

Re: =[xhr]

[Takeshi Yoshino]

On Tue, Nov 18, 2014 at 1:45 PM, Domenic Denicola d...@domenic.me wrote:

 From: Takeshi Yoshino [mailto:tyosh...@google.com]

  On Tue, Nov 18, 2014 at 12:11 AM, Anne van Kesteren ann...@annevk.nl
 wrote:
  On Mon, Nov 17, 2014 at 3:50 PM, Domenic Denicola d...@domenic.me wrote:
  What do we think of that kind of behavior for fetch requests?
 
  I'm not sure we want to give a potential hostile piece of script that
 much control over what goes out. Being able to lie about Content-Length
 would be a new feature that does not really seem desirable. Streaming
 should probably imply chunked given that.
 
  Agreed.

 That would be very sad. There are many servers that will not accept
 chunked upload (for example Amazon S3). This would mean web apps would be
 unable to do streaming upload to such servers.


Hmm, is this kinda protection against DoS? It seems S3 SigV4 accepts
chunked but still requires a custom-header indicating the final size. This
may imply that even if sending with chunked T-E becomes popular with the
Fetch API they won't accept such requests without length info in advance.

Re: =[xhr]

[Takeshi Yoshino]

How about padding the remaining bytes forcefully with e.g. 0x20 if the
WritableStream doesn't provide enough bytes to us?

Takeshi

On Tue, Nov 18, 2014 at 7:01 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Tue, Nov 18, 2014 at 10:34 AM, Domenic Denicola d...@domenic.me wrote:
  I still think we should just allow the developer full control over the
 Content-Length header if they've taken full control over the contents of
 the request body (by writing to its stream asynchronously and piecemeal).
 It gives no more power than using CURL. (Except the usual issues of
 ambient/cookie authority, but those seem orthogonal to Content-Length
 mismatch.)

 Why? If a service behind a firewall is vulnerable to Content-Length
 mismatches, you can now attack such a service by tricking a user
 behind that firewall into visiting evil.com.


 --
 https://annevankesteren.nl/

Re: =[xhr]

[Takeshi Yoshino]

On Wed, Nov 19, 2014 at 1:45 AM, Domenic Denicola d...@domenic.me wrote:

 From: annevankeste...@gmail.com [mailto:annevankeste...@gmail.com] On
 Behalf Of Anne van Kesteren

  On Tue, Nov 18, 2014 at 12:50 PM, Takeshi Yoshino tyosh...@google.com
 wrote:
  How about padding the remaining bytes forcefully with e.g. 0x20 if the
 WritableStream doesn't provide enough bytes to us?
 
  How would that work? At some point when the browser decides it wants to
 terminate the fetch (e.g. due to timeout, tab being closed) it attempts to
 transmit a bunch of useless bytes? What if the value is really large?


It's a problem that we'll provide a very easy way (compared to building a
big ArrayBuffer by doubling its size repeatedly) to a malicious script to
have a user agent send very large data. So, we might want to place a limit
to the maximum size of Content-Length that doesn't hurt the benefit of
streaming upload too much.


 I think there are several different scenarios under consideration.

 1. The author says Content-Length 100, writes 50 bytes, then closes the
 stream.
 2. The author says Content-Length 100, writes 50 bytes, and never closes
 the stream.
 3. The author says Content-Length 100, writes 150 bytes, then closes the
 stream.
 4. The author says Content-Length 100 , writes 150 bytes, and never closes
 the stream.

 It would be helpful to know how most servers handle these. (Perhaps HTTP
 specifies a mandatory behavior.) My guess is that they are very capable of
 handling such situations. 2 in particular resembles a long-polling setup.

 As for whether we consider this kind of thing an attack, instead of just
 a new capability, I'd love to get some security folks to weigh in. If they
 think it's indeed a bad idea, then we can discuss mitigation strategies; 3
 and 4 are easily mitigatable, whereas 1 could be addressed by an idea like
 Takeshi's. I don't think mitigating 2 makes much sense as we can't know
 when the author intends to send more data.


The extra 50 bytes for the case 3 and 4 should definitely be ignored by the
user agent. The user agent should probably also error the WritableStream
when extra bytes are written.

2 is useful but new situation to web apps. I agree that we should consult
security experts.

Re: Allow custom headers (Websocket API)

[Takeshi Yoshino]

On Thu, Feb 5, 2015 at 10:57 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, Feb 5, 2015 at 2:48 PM, Bjoern Hoehrmann derhoe...@gmx.net
 wrote:
  A Websocket connection is established by making a HTTP Upgrade request,
  and the protocol is HTTP unless and until the connection is upgraded.

 Sure, but the server can get away with supporting a very limited
 subset of HTTP, no? Anyway, perhaps a combination of a CORS preflight
 followed by the HTTP Upgrade that then includes the headers is the
 answer, would probably be best to ask some WebSocket library
 developers what they think.


Agreed. Even if we don't make any change on the existing specs, we need to
standardize (or just announce to developers) that they need to make servers
understand that combination if they want to develop apps that uses custom
headers. Then, client vendors could implement that.

Re: Allow custom headers (Websocket API)

[Takeshi Yoshino]

On Thu, Feb 5, 2015 at 10:41 PM, Florian Bösch pya...@gmail.com wrote:

 On Thu, Feb 5, 2015 at 2:39 PM, Takeshi Yoshino tyosh...@google.com
 wrote:

 To prevent WebSocket from being abused to attack existing HTTP servers
 from malicious non-simple cross-origin requests, we need to have WebSocket
 clients to do some preflight to verify that the server is not an HTTP
 server that don't understand CORS. We could do e.g. when a custom header is
 specified,

 No further specification is needed because CORS already covers the case of
 endpoints that do not understand CORS (deny by default). Hence above
 assertion is superfluous.


IIUC, CORS prevents clients from issuing non-simple cross-origin request
(even idempotent methods) without verifying that the server understands
CORS. That's realized by preflight.




 So, anyway, I think we need to make some change on the WebSocket spec.

 Also bogus assertion.

Re: Allow custom headers (Websocket API)

[Takeshi Yoshino]

To prevent WebSocket from being abused to attack existing HTTP servers from
malicious non-simple cross-origin requests, we need to have WebSocket
clients to do some preflight to verify that the server is not an HTTP
server that don't understand CORS. We could do e.g. when a custom header is
specified,

(a) the client issues a CORS preflight to verify that the server does
understand CORS and it is willing to accept the request.
(b) the client issues a WebSocket preflight to verify that the server is a
WebSocket server.

(a) may work, needs change on the WebSocket spec to ask WebSocket servers
to understand CORS preflight. It's not required for now.

(b) may be implemented by issuing an extra WebSocket handshake without the
custom headers but with Sec-WebSocket-Key, etc. just for checking that the
server is WebSocket server, but this may be not no-op to the server. So, I
think we should specify something new that is specified to be no-op.

So, anyway, I think we need to make some change on the WebSocket spec.

Takeshi

On Thu, Feb 5, 2015 at 10:23 PM, Florian Bösch pya...@gmail.com wrote:

 Well,

 1) Clients do apply CORS to WebSocket requests already (and might've
 started doing so quite some time ago) and everything's fine and you don't
 need to change anything.

 2) Clients do not apply CORS to WebSocket requests, and you're screwed,
 because any change you make will break existing deployments.

 Either way, this will result in no change made, so you can burry it right
 here.

 On Thu, Feb 5, 2015 at 2:12 PM, Anne van Kesteren ann...@annevk.nl
 wrote:

 On Thu, Feb 5, 2015 at 1:27 PM, Florian Bösch pya...@gmail.com wrote:
  CORS is an adequate protocol to allow for additional headers, and
 websocket
  requests could be subjected to CORS (I'm not sure what the current
 client
  behavior is in that regard, but I'm guessing they enforce CORS on
 websocket
  requests as well).

 I think you're missing something. A WebSocket request is subject to
 the WebSocket protocol, which does not take the same precautions as
 the Fetch protocol does used elsewhere in the platform. And therefore
 we cannot provide this feature until the WebSocket protocol is fixed
 to take the same precautions.


 --
 https://annevankesteren.nl/

Re: Allow custom headers (Websocket API)

[Takeshi Yoshino]

http://www.w3.org/TR/cors/#cross-origin-request-0

 2. If the following conditions are true, follow the simple cross-origin
request algorithm:
 - The request method is a simple method and the force preflight flag is
unset.
 - Each of the author request headers is a simple header or author request
headers is empty.
 3. Otherwise, follow the cross-origin request with preflight algorithm.

https://fetch.spec.whatwg.org/#dfnReturnLink-7

 request's unsafe request flag is set and either request's method is not a
simple method or a header in request's header list is not a simple header
   Set request's response tainting to CORS.
   The result of performing an HTTP fetch using request with the CORS flag
and CORS preflight flag set.

Authorization header is not a simple header.


On Thu, Feb 5, 2015 at 10:48 PM, Florian Bösch pya...@gmail.com wrote:

 On Thu, Feb 5, 2015 at 2:44 PM, Takeshi Yoshino tyosh...@google.com
 wrote:

 IIUC, CORS prevents clients from issuing non-simple cross-origin request
 (even idempotent methods) without verifying that the server understands
 CORS. That's realized by preflight.


 Incorrect, the browser will perform idempotent requests (for instance
 img or XHR GET) across domains without a preflight request. It will
 however not make the data available to the client (javascript specifically)


That's the tainting part.


 unless CORS is satisfied (XHR GET will error out, and img will throw a
 glError on gl.texImage2D if CORS isn't satisfied).

Re: Allow custom headers (Websocket API)

[Takeshi Yoshino]

Usually,
- IETF HyBi ML
http://www.ietf.org/mail-archive/web/hybi/current/maillist.html for
protocol stuff
- Here or WHATWG ML
https://lists.w3.org/Archives/Public/public-whatwg-archive/ for API stuff


On Thu, Feb 5, 2015 at 11:07 PM, Michiel De Mey de.mey.mich...@gmail.com
wrote:

 Standardizing the approach would definitely help developers,
 however where will we communicate this?

Re: [charter] What is the plan for Streams API?

[Takeshi Yoshino]

+domenic

We've recently finished the ReadableStream part of the spec and
experimenting integration with the Fetch API. Most of the spec is still
unstable. I don't have bandwidth to maintain W3C version of the spec even
briefly currently...

Takeshi

On Tue, Aug 4, 2015 at 11:13 PM, Arthur Barstow art.bars...@gmail.com
wrote:

 On 7/30/15 8:46 AM, Arthur Barstow wrote:

 http://w3c.github.io/charter-html/group-charter.html


 The WebApps + HTML WG draft charter says the following about WebApps'
 Streams API:

 [[
 Streams API https://w3c.github.io/streams-api/
An API for representing a stream of data in web applications.
 ]]

 I believe the previously agreed plan of record for this spec was to
 create an API on top of https://streams.spec.whatwg.org/. Is that still
 something this group wants to do, and if so, who can commit to actually
 doing the work, in particular: editing, implementation, and test suite?

 If we no longer have committed resources for doing the above tasks then
 this spec should be removed from the draft charter.

 -Thanks, AB

Re: Art steps down - thank you for everything

[Takeshi Yoshino]

Thank you Art!

Takeshi

On Mon, Feb 1, 2016 at 12:39 AM, Tobie Langel  wrote:

> So long, Art, and thanks for all the fish.
>
> --tobie
>
> On Thu, 28 Jan 2016, at 16:45, Chaals McCathie Nevile wrote:
> > Hi folks,
> >
> > as you may have noticed, Art has resigned as a co-chair of the Web
> > Platform group. He began chairing the Web Application Formats group about
> > a decade ago, became the leading co-chair when it merged with Web APIs to
> > become the Web Apps working group, and was instrumental in making the
> > transition from Web Apps to the Web Platform Group. (He also chaired
> > various other W3C groups in that time).
> >
> > I've been very privileged to work with Art on the webapps group for so
> > many years - as many of you know, without him it would have been a much
> > poorer group, and run much less smoothly. He did a great deal of work for
> > the group throughout his time as co-chair, efficiently, reliably, and
> > quietly.
> >
> > Now we are three co-chairs, we will work between us to fill Art's shoes.
> > It won't be easy.
> >
> > Thanks Art for everything you've done for the group for so long.
> >
> > Good luck, and I hope to see you around.
> >
> > Chaals
> >
> > --
> > Charles McCathie Nevile - web standards - CTO Office, Yandex
> >   cha...@yandex-team.ru - - - Find more at http://yandex.com
> >
>
>