Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Kenneth Russell]

On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote:
 On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote:

  You can always call close() yourself, but Blob.close() should use the
  neuter mechanism already there, not make up a new one.

 Blobs aren't transferable, there is no existing mechanism that applies
 to them. Adding a blob.close() method is independent of making blob's
 transferable, the former is not prerequisite on the latter.


 There is an existing mechanism for closing objects.  It's called
 neutering.  Blob.close should use the same terminology, whether or not the
 object is a Transferable.

 On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote:

 I would be hesitant to impose a close() method on all future
 Transferable types.


 Why?  All Transferable types must define how to neuter objects; all close()
 does is trigger it.

 I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.


 That's often not possible, when the ArrayBuffer is returned to you from an
 API (eg. XHR2).

 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.


 No new semantics need to be defined; the semantics of Transferable are
 defined by postMessage and are the same for all transferable objects.
 That's already done.  The only thing that needs to be defined is how to
 neuter an object, which is what Blob.close() has to define anyway.

 Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future
 large, structured clonable objects to all be released with the same
 mechanisms: either pass them in the transfer argument to a postMessage
 call, or use the consistent, identical close() method inherited from
 Transferable.  This allows developers to think of the transfer list as a
 list of objects which won't be needed after the postMessage call.  It
 doesn't matter that the underlying optimizations are different; the visible
 side-effects are identical (the object can no longer be accessed).

Closing an object, and neutering it because it was transferred to a
different owner, are different concepts. It's already been
demonstrated that Blobs, being read-only, do not need to be
transferred in order to send them efficiently from one owner to
another. It's also been demonstrated that Blobs can be resource
intensive and that an explicit closing mechanism is needed.

I believe that we should fix the immediate problem and add a close()
method to Blob. I'm not in favor of adding a similar method to
ArrayBuffer at this time and therefore not to Transferable. There is a
high-level goal to keep the typed array specification as minimal as
possible, and having Transferable support leak in to the public
methods of the interfaces contradicts that goal.

-Ken

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Eric U]

On Wed, Mar 7, 2012 at 11:38 AM, Kenneth Russell k...@google.com wrote:
 On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote:
 On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote:

  You can always call close() yourself, but Blob.close() should use the
  neuter mechanism already there, not make up a new one.

 Blobs aren't transferable, there is no existing mechanism that applies
 to them. Adding a blob.close() method is independent of making blob's
 transferable, the former is not prerequisite on the latter.


 There is an existing mechanism for closing objects.  It's called
 neutering.  Blob.close should use the same terminology, whether or not the
 object is a Transferable.

 On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote:

 I would be hesitant to impose a close() method on all future
 Transferable types.


 Why?  All Transferable types must define how to neuter objects; all close()
 does is trigger it.

 I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.


 That's often not possible, when the ArrayBuffer is returned to you from an
 API (eg. XHR2).

 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.


 No new semantics need to be defined; the semantics of Transferable are
 defined by postMessage and are the same for all transferable objects.
 That's already done.  The only thing that needs to be defined is how to
 neuter an object, which is what Blob.close() has to define anyway.

 Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future
 large, structured clonable objects to all be released with the same
 mechanisms: either pass them in the transfer argument to a postMessage
 call, or use the consistent, identical close() method inherited from
 Transferable.  This allows developers to think of the transfer list as a
 list of objects which won't be needed after the postMessage call.  It
 doesn't matter that the underlying optimizations are different; the visible
 side-effects are identical (the object can no longer be accessed).

 Closing an object, and neutering it because it was transferred to a
 different owner, are different concepts. It's already been
 demonstrated that Blobs, being read-only, do not need to be
 transferred in order to send them efficiently from one owner to
 another. It's also been demonstrated that Blobs can be resource
 intensive and that an explicit closing mechanism is needed.

 I believe that we should fix the immediate problem and add a close()
 method to Blob. I'm not in favor of adding a similar method to
 ArrayBuffer at this time and therefore not to Transferable. There is a
 high-level goal to keep the typed array specification as minimal as
 possible, and having Transferable support leak in to the public
 methods of the interfaces contradicts that goal.

This makes sense to me.  Blob needs close independent of whether it's
in Transferable, and Blob has no need to be Transferable, so let's not
mix the two.

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Charles Pritchard]

On Mar 7, 2012, at 11:38 AM, Kenneth Russell k...@google.com wrote:

 On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote:
 On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote:
 
 You can always call close() yourself, but Blob.close() should use the
 neuter mechanism already there, not make up a new one.
 
 Blobs aren't transferable, there is no existing mechanism that applies
 to them. Adding a blob.close() method is independent of making blob's
 transferable, the former is not prerequisite on the latter.
 
 
 There is an existing mechanism for closing objects.  It's called
 neutering.  Blob.close should use the same terminology, whether or not the
 object is a Transferable.
 
 On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote:
 
 I would be hesitant to impose a close() method on all future
 Transferable types.
 
 
 Why?  All Transferable types must define how to neuter objects; all close()
 does is trigger it.
 
 I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.
 
 
 That's often not possible, when the ArrayBuffer is returned to you from an
 API (eg. XHR2).
 
 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.
 
 
 No new semantics need to be defined; the semantics of Transferable are
 defined by postMessage and are the same for all transferable objects.
 That's already done.  The only thing that needs to be defined is how to
 neuter an object, which is what Blob.close() has to define anyway.
 
 Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future
 large, structured clonable objects to all be released with the same
 mechanisms: either pass them in the transfer argument to a postMessage
 call, or use the consistent, identical close() method inherited from
 Transferable.  This allows developers to think of the transfer list as a
 list of objects which won't be needed after the postMessage call.  It
 doesn't matter that the underlying optimizations are different; the visible
 side-effects are identical (the object can no longer be accessed).
 
 Closing an object, and neutering it because it was transferred to a
 different owner, are different concepts. It's already been
 demonstrated that Blobs, being read-only, do not need to be
 transferred in order to send them efficiently from one owner to
 another. It's also been demonstrated that Blobs can be resource
 intensive and that an explicit closing mechanism is needed.
 
 I believe that we should fix the immediate problem and add a close()
 method to Blob. I'm not in favor of adding a similar method to
 ArrayBuffer at this time and therefore not to Transferable. There is a
 high-level goal to keep the typed array specification as minimal as
 possible, and having Transferable support leak in to the public
 methods of the interfaces contradicts that goal.

I think there's broad enough consensus amongst vendors to table the discussion 
about adding close to Transferable.

Would you please let me know why ypu believe ArrayBuffer should not have a 
close method?

I would like some clarity here. The Typed Array spec would not be cluttered by 
the addition of a simple close method.

I work much more with ArrayBuffer than Blob. I suspect others will too as they 
progress with more advanced and resource intensive applications.

What is the use-case distinction between close of immutable blob and close of a 
mutable buffer?

-Charles

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Greg Billock]

On Tue, Mar 6, 2012 at 1:18 PM, Kenneth Russell k...@google.com wrote:
 On Tue, Mar 6, 2012 at 12:04 PM, Greg Billock gbill...@google.com wrote:
 On Mon, Mar 5, 2012 at 6:46 PM, Charles Pritchard ch...@jumis.com wrote:
 On 3/5/2012 5:56 PM, Glenn Maynard wrote:

 On Mon, Mar 5, 2012 at 7:04 PM, Charles Pritchard ch...@jumis.com wrote:

 Do you see old behavior working something like the following?


 var blob = new Blob(my new big blob);
 var keepBlob = blob.slice(); destination.postMessage(blob, '*', [blob]);
 // is try/catch needed here?


 You don't need to do that.  If you don't want postMessage to transfer the
 blob, then simply don't include it in the transfer parameter, and it'll
 perform a normal structured clone.  postMessage behaves this way in part for
 backwards-compatibility: so exactly in cases like this, we can make Blob
 implement Transferable without breaking existing code.

 See http://dev.w3.org/html5/postmsg/#posting-messages and similar
 postMessage APIs.


 Web Intents won't have a transfer map argument.
 http://dvcs.w3.org/hg/web-intents/raw-file/tip/spec/Overview.html#widl-Intent-data

 For the Web Intents structured cloning algorithm, Web Intents would be
 inserting into step 3:
     If input is a Transferable object, add it to the transfer map.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#internal-structured-cloning-algorithm

 Then Web Intents would move the first section of the structured cloning
 algorithm to follow the internal cloning algorithm section, swapping their
 order.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#safe-passing-of-structured-data

 That's my understanding.

 We've been discussing the merits of this approach vs using a transfer
 array argument. There's a lot to like about this alternative -- it
 conserves arguments and looks simpler than the transfer map, as well
 as not having the headaches of whether you can do (null, [port]) or
 (port, [port]) and concerns like that.

 The advantage of using the transfer map param is that it is more
 contiguous with existing practice. We'd kind of hoped that this
 particular debate was finalized before we got to the point of needing
 to make a decision, so we bluffed and left it out of the web intents
 spec draft. :-) At this point, I'm leaning toward needing to add a
 transfer map parameter, and then dealing with that alongside other
 uses, given the state of thinking on Transferables support and the
 need to make this pretty consistent across structure clone
 invocations.

 I do think that complexity might be better solved by the type system
 (i.e. a new Transferable(ArrayBuffer)), which would require a
 different developer mechanic to set up clone vs transfer, but would
 relieve complexity in the invocation of structured clone itself:
 transferables could just always transfer transparently. I don't know
 if, given current practice with MessagePort, that kind of solution is
 available.

 A change like this would be feasible as long as it doesn't break
 compatibility. In other words, the current Transferable array would
 still need to be supported, but Transferable instances (or perhaps
 instances of some other type) wrapping another Transferable object
 would also express the intent.

 The current API for Transferable and postMessage was informed by the
 realization that the previous sequenceMessagePort argument to
 postMessage was essentially already expressing the Transferable
 concept.

 I'm not familiar with the Web Intents API, but at first glance it
 seems feasible to overload the constructor, postResult and postFailure
 methods to support passing a sequenceTransferable as the last
 argument. This would make the API look more like postMessage and avoid
 adding more transfer semantics. Is that possible?

Yes. That's our current plan.

-Greg

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Kenneth Russell]

On Wed, Mar 7, 2012 at 12:02 PM, Charles Pritchard ch...@jumis.com wrote:

 On Mar 7, 2012, at 11:38 AM, Kenneth Russell k...@google.com wrote:

 On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote:
 On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote:

 You can always call close() yourself, but Blob.close() should use the
 neuter mechanism already there, not make up a new one.

 Blobs aren't transferable, there is no existing mechanism that applies
 to them. Adding a blob.close() method is independent of making blob's
 transferable, the former is not prerequisite on the latter.


 There is an existing mechanism for closing objects.  It's called
 neutering.  Blob.close should use the same terminology, whether or not the
 object is a Transferable.

 On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote:

 I would be hesitant to impose a close() method on all future
 Transferable types.


 Why?  All Transferable types must define how to neuter objects; all close()
 does is trigger it.

 I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.


 That's often not possible, when the ArrayBuffer is returned to you from an
 API (eg. XHR2).

 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.


 No new semantics need to be defined; the semantics of Transferable are
 defined by postMessage and are the same for all transferable objects.
 That's already done.  The only thing that needs to be defined is how to
 neuter an object, which is what Blob.close() has to define anyway.

 Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future
 large, structured clonable objects to all be released with the same
 mechanisms: either pass them in the transfer argument to a postMessage
 call, or use the consistent, identical close() method inherited from
 Transferable.  This allows developers to think of the transfer list as a
 list of objects which won't be needed after the postMessage call.  It
 doesn't matter that the underlying optimizations are different; the visible
 side-effects are identical (the object can no longer be accessed).

 Closing an object, and neutering it because it was transferred to a
 different owner, are different concepts. It's already been
 demonstrated that Blobs, being read-only, do not need to be
 transferred in order to send them efficiently from one owner to
 another. It's also been demonstrated that Blobs can be resource
 intensive and that an explicit closing mechanism is needed.

 I believe that we should fix the immediate problem and add a close()
 method to Blob. I'm not in favor of adding a similar method to
 ArrayBuffer at this time and therefore not to Transferable. There is a
 high-level goal to keep the typed array specification as minimal as
 possible, and having Transferable support leak in to the public
 methods of the interfaces contradicts that goal.

 I think there's broad enough consensus amongst vendors to table the 
 discussion about adding close to Transferable.

 Would you please let me know why ypu believe ArrayBuffer should not have a 
 close method?

 I would like some clarity here. The Typed Array spec would not be cluttered 
 by the addition of a simple close method.

It's certainly a matter of opinion -- but while it's only the addition
of one method, it changes typed arrays' semantics to be much closer to
manual memory allocation than they currently are. It would be a
further divergence in behavior from ordinary ECMAScript arrays.

The TC39 working group, I have heard, is incorporating typed arrays
into the language specification, and for this reason I believe extreme
care is warranted when adding more functionality to the typed array
spec. The spec can certainly move forward, but personally I'd like to
check with TC39 on semantic changes like this one. That's the
rationale behind my statement above about preferring not to add this
method at this time.

-Ken


 I work much more with ArrayBuffer than Blob. I suspect others will too as 
 they progress with more advanced and resource intensive applications.

 What is the use-case distinction between close of immutable blob and close of 
 a mutable buffer?

 -Charles

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Charles Pritchard]

On 3/7/12 12:34 PM, Kenneth Russell wrote:

On Wed, Mar 7, 2012 at 12:02 PM, Charles Pritchardch...@jumis.com  wrote:

On Mar 7, 2012, at 11:38 AM, Kenneth Russellk...@google.com  wrote:


I believe that we should fix the immediate problem and add a close()
method to Blob. I'm not in favor of adding a similar method to
ArrayBuffer at this time and therefore not to Transferable. There is a
high-level goal to keep the typed array specification as minimal as
possible, and having Transferable support leak in to the public
methods of the interfaces contradicts that goal.

I think there's broad enough consensus amongst vendors to table the discussion 
about adding close to Transferable.

Would you please let me know why ypu believe ArrayBuffer should not have a 
close method?

I would like some clarity here. The Typed Array spec would not be cluttered by 
the addition of a simple close method.

It's certainly a matter of opinion -- but while it's only the addition
of one method, it changes typed arrays' semantics to be much closer to
manual memory allocation than they currently are. It would be a
further divergence in behavior from ordinary ECMAScript arrays.

The TC39 working group, I have heard, is incorporating typed arrays
into the language specification, and for this reason I believe extreme
care is warranted when adding more functionality to the typed array
spec. The spec can certainly move forward, but personally I'd like to
check with TC39 on semantic changes like this one. That's the
rationale behind my statement above about preferring not to add this
method at this time.


Searching through the net tells me that this has been a rumor for years.

I agree with taking extreme care -- so let's isolate one more bit of 
information:


Is ArrayBuffer being proposed for TC39 incorporation, or is it only the 
Typed Arrays? The idea here is to alter ArrayBuffer, an object which can 
be neutered via transfer map. It seems a waste to have to create a 
Worker to close down buffer views.


Will TC39 have anything to say about the neuter concept and/or Web 
Messaging?



Again, I'm bringing this up for the same practical experience that 
Blob.close() was brought up. I do appreciate that read/write allocation 
is a separate semantic from write-once/read-many allocation.


I certainly don't want to derail the introduction of Typed Array into 
TC39. I don't want to sit back for two years either, while the 
ArrayBuffer object is in limbo.


If necessary, I'll do some of the nasty test work of creating a worker 
simply to destroy buffers, and report back on it.

var worker = new Worker('trash.js');
worker.postMessage(null,[bufferToClose]);
worker.close();
vs.
bufferToClose.close();



-Charles

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Kenneth Russell]

On Wed, Mar 7, 2012 at 1:00 PM, Charles Pritchard ch...@jumis.com wrote:
 On 3/7/12 12:34 PM, Kenneth Russell wrote:

 On Wed, Mar 7, 2012 at 12:02 PM, Charles Pritchardch...@jumis.com
  wrote:

 On Mar 7, 2012, at 11:38 AM, Kenneth Russellk...@google.com  wrote:

 I believe that we should fix the immediate problem and add a close()
 method to Blob. I'm not in favor of adding a similar method to
 ArrayBuffer at this time and therefore not to Transferable. There is a
 high-level goal to keep the typed array specification as minimal as
 possible, and having Transferable support leak in to the public
 methods of the interfaces contradicts that goal.

 I think there's broad enough consensus amongst vendors to table the
 discussion about adding close to Transferable.

 Would you please let me know why ypu believe ArrayBuffer should not have
 a close method?

 I would like some clarity here. The Typed Array spec would not be
 cluttered by the addition of a simple close method.

 It's certainly a matter of opinion -- but while it's only the addition
 of one method, it changes typed arrays' semantics to be much closer to
 manual memory allocation than they currently are. It would be a
 further divergence in behavior from ordinary ECMAScript arrays.

 The TC39 working group, I have heard, is incorporating typed arrays
 into the language specification, and for this reason I believe extreme
 care is warranted when adding more functionality to the typed array
 spec. The spec can certainly move forward, but personally I'd like to
 check with TC39 on semantic changes like this one. That's the
 rationale behind my statement above about preferring not to add this
 method at this time.


 Searching through the net tells me that this has been a rumor for years.

Regardless of rumors I have talked to multiple members of TC39 who
have clearly stated it is being incorporated into ES6 Harmony.

 I agree with taking extreme care -- so let's isolate one more bit of
 information:

 Is ArrayBuffer being proposed for TC39 incorporation, or is it only the
 Typed Arrays? The idea here is to alter ArrayBuffer, an object which can be
 neutered via transfer map. It seems a waste to have to create a Worker to
 close down buffer views.

Both ArrayBuffer and the typed array views will be incorporated.

 Will TC39 have anything to say about the neuter concept and/or Web
 Messaging?

This is an excellent question and one which I've also posed to TC39. I
don't see how the language spec could reference these concepts. I'm
guessing that this is an area that TC39 hasn't yet figured out,
either.

 Again, I'm bringing this up for the same practical experience that
 Blob.close() was brought up. I do appreciate that read/write allocation is a
 separate semantic from write-once/read-many allocation.

 I certainly don't want to derail the introduction of Typed Array into TC39.
 I don't want to sit back for two years either, while the ArrayBuffer object
 is in limbo.

Understood and appreciated.

 If necessary, I'll do some of the nasty test work of creating a worker
 simply to destroy buffers, and report back on it.
 var worker = new Worker('trash.js');
 worker.postMessage(null,[bufferToClose]);
 worker.close();
 vs.
 bufferToClose.close();

I doubt that that will work. Garbage collection will still need to run
in the worker's JavaScript context in order for the transferred
ArrayBuffer to be cleaned up, and I doubt that happens eagerly upon
shutdown of the worker. Would be happy to be proven wrong.

If you prototype adding ArrayBuffer.close() in your open source
browser of choice and report back on significant efficiency
improvements in a real-world use case, that would be valuable
feedback.

-Ken

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Greg Billock]

On Mon, Mar 5, 2012 at 6:46 PM, Charles Pritchard ch...@jumis.com wrote:
 On 3/5/2012 5:56 PM, Glenn Maynard wrote:

 On Mon, Mar 5, 2012 at 7:04 PM, Charles Pritchard ch...@jumis.com wrote:

 Do you see old behavior working something like the following?


 var blob = new Blob(my new big blob);
 var keepBlob = blob.slice(); destination.postMessage(blob, '*', [blob]);
 // is try/catch needed here?


 You don't need to do that.  If you don't want postMessage to transfer the
 blob, then simply don't include it in the transfer parameter, and it'll
 perform a normal structured clone.  postMessage behaves this way in part for
 backwards-compatibility: so exactly in cases like this, we can make Blob
 implement Transferable without breaking existing code.

 See http://dev.w3.org/html5/postmsg/#posting-messages and similar
 postMessage APIs.


 Web Intents won't have a transfer map argument.
 http://dvcs.w3.org/hg/web-intents/raw-file/tip/spec/Overview.html#widl-Intent-data

 For the Web Intents structured cloning algorithm, Web Intents would be
 inserting into step 3:
     If input is a Transferable object, add it to the transfer map.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#internal-structured-cloning-algorithm

 Then Web Intents would move the first section of the structured cloning
 algorithm to follow the internal cloning algorithm section, swapping their
 order.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#safe-passing-of-structured-data

 That's my understanding.

We've been discussing the merits of this approach vs using a transfer
array argument. There's a lot to like about this alternative -- it
conserves arguments and looks simpler than the transfer map, as well
as not having the headaches of whether you can do (null, [port]) or
(port, [port]) and concerns like that.

The advantage of using the transfer map param is that it is more
contiguous with existing practice. We'd kind of hoped that this
particular debate was finalized before we got to the point of needing
to make a decision, so we bluffed and left it out of the web intents
spec draft. :-) At this point, I'm leaning toward needing to add a
transfer map parameter, and then dealing with that alongside other
uses, given the state of thinking on Transferables support and the
need to make this pretty consistent across structure clone
invocations.

I do think that complexity might be better solved by the type system
(i.e. a new Transferable(ArrayBuffer)), which would require a
different developer mechanic to set up clone vs transfer, but would
relieve complexity in the invocation of structured clone itself:
transferables could just always transfer transparently. I don't know
if, given current practice with MessagePort, that kind of solution is
available.

 Something like this may be necessary if Blob were a Transferable:
 var keepBlob = blob.slice();
 var intent = new Intent(-x-my-intent, blob);
 navigator.startActivity(intent, callback);

 And we might have an error on postMessage stashing it in the transfer
 array if it's not a Transferable on an older browser.




 Example of how easy the neutered concept applies to Transferrable:

 var blob = new Blob(my big blob);
 blob.close();


 I like the idea of having Blob implement Transferrable and adding close to
 the Transferrable interface.
 File.close could have a better relationship with the cache and/or locks on
 data.


 Some history on Transferrable and structured clones:

 Note: MessagePort does have a close method and is currently the only
 Transferrable mentioned in WHATWG:
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#transferable-objects

 ArrayBuffer is widely implemented. It was the second item to implement
 Transferrable:
 http://www.khronos.org/registry/typedarray/specs/latest/#9

 Subsequently, ImageData adopted Uint8ClampedArray for one of its properties,
 adopting TypedArrays:
 http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata

 This has lead to some instability in the structured clone algorithm for
 ImageData as the typed array object for ImageData is read-only.
 https://www.w3.org/Bugs/Public/show_bug.cgi?id=13800

 ArrayBuffer is still in a strawman state.

 -Charles

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Kenneth Russell]

On Tue, Mar 6, 2012 at 12:04 PM, Greg Billock gbill...@google.com wrote:
 On Mon, Mar 5, 2012 at 6:46 PM, Charles Pritchard ch...@jumis.com wrote:
 On 3/5/2012 5:56 PM, Glenn Maynard wrote:

 On Mon, Mar 5, 2012 at 7:04 PM, Charles Pritchard ch...@jumis.com wrote:

 Do you see old behavior working something like the following?


 var blob = new Blob(my new big blob);
 var keepBlob = blob.slice(); destination.postMessage(blob, '*', [blob]);
 // is try/catch needed here?


 You don't need to do that.  If you don't want postMessage to transfer the
 blob, then simply don't include it in the transfer parameter, and it'll
 perform a normal structured clone.  postMessage behaves this way in part for
 backwards-compatibility: so exactly in cases like this, we can make Blob
 implement Transferable without breaking existing code.

 See http://dev.w3.org/html5/postmsg/#posting-messages and similar
 postMessage APIs.


 Web Intents won't have a transfer map argument.
 http://dvcs.w3.org/hg/web-intents/raw-file/tip/spec/Overview.html#widl-Intent-data

 For the Web Intents structured cloning algorithm, Web Intents would be
 inserting into step 3:
     If input is a Transferable object, add it to the transfer map.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#internal-structured-cloning-algorithm

 Then Web Intents would move the first section of the structured cloning
 algorithm to follow the internal cloning algorithm section, swapping their
 order.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#safe-passing-of-structured-data

 That's my understanding.

 We've been discussing the merits of this approach vs using a transfer
 array argument. There's a lot to like about this alternative -- it
 conserves arguments and looks simpler than the transfer map, as well
 as not having the headaches of whether you can do (null, [port]) or
 (port, [port]) and concerns like that.

 The advantage of using the transfer map param is that it is more
 contiguous with existing practice. We'd kind of hoped that this
 particular debate was finalized before we got to the point of needing
 to make a decision, so we bluffed and left it out of the web intents
 spec draft. :-) At this point, I'm leaning toward needing to add a
 transfer map parameter, and then dealing with that alongside other
 uses, given the state of thinking on Transferables support and the
 need to make this pretty consistent across structure clone
 invocations.

 I do think that complexity might be better solved by the type system
 (i.e. a new Transferable(ArrayBuffer)), which would require a
 different developer mechanic to set up clone vs transfer, but would
 relieve complexity in the invocation of structured clone itself:
 transferables could just always transfer transparently. I don't know
 if, given current practice with MessagePort, that kind of solution is
 available.

A change like this would be feasible as long as it doesn't break
compatibility. In other words, the current Transferable array would
still need to be supported, but Transferable instances (or perhaps
instances of some other type) wrapping another Transferable object
would also express the intent.

The current API for Transferable and postMessage was informed by the
realization that the previous sequenceMessagePort argument to
postMessage was essentially already expressing the Transferable
concept.

I'm not familiar with the Web Intents API, but at first glance it
seems feasible to overload the constructor, postResult and postFailure
methods to support passing a sequenceTransferable as the last
argument. This would make the API look more like postMessage and avoid
adding more transfer semantics. Is that possible?


 Something like this may be necessary if Blob were a Transferable:
 var keepBlob = blob.slice();
 var intent = new Intent(-x-my-intent, blob);
 navigator.startActivity(intent, callback);

 And we might have an error on postMessage stashing it in the transfer
 array if it's not a Transferable on an older browser.




 Example of how easy the neutered concept applies to Transferrable:

 var blob = new Blob(my big blob);
 blob.close();


 I like the idea of having Blob implement Transferrable and adding close to
 the Transferrable interface.
 File.close could have a better relationship with the cache and/or locks on
 data.

I'm not sure that adding close() to Transferable is a good idea. Not
all Transferable types may want to support that explicit operation.
What about adding close() to Blob, and having the neutering operation
on Blob be defined to call close() on it?

-Ken


 Some history on Transferrable and structured clones:

 Note: MessagePort does have a close method and is currently the only
 Transferrable mentioned in WHATWG:
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#transferable-objects

 ArrayBuffer is widely implemented. It was the second item to implement
 

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Arun Ranganathan]

Ken,

 I'm not sure that adding close() to Transferable is a good idea. Not
 all Transferable types may want to support that explicit operation.
 What about adding close() to Blob, and having the neutering operation
 on Blob be defined to call close() on it?


Specifically, you think this is not something ArrayBuffer should inherit?  If 
it's also a bad idea for MessagePort, then those are really our only two use 
cases of Transferable right now.  I'm happy to create something like a close() 
on Blob.

-- A*

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Michael Nordman]

Sounds like there's a good case for an explicit blob.close() method
independent of 'transferable'. Separately defining blobs to be
transferrable feels like an unneeded complexity. A caller wishing to
neuter after sending can explicit call .close() rather than relying on
more obscure artifacts of having also put the 'blob' in a
'transferrable' array.

On Tue, Mar 6, 2012 at 1:18 PM, Kenneth Russell k...@google.com wrote:
 On Tue, Mar 6, 2012 at 12:04 PM, Greg Billock gbill...@google.com wrote:
 On Mon, Mar 5, 2012 at 6:46 PM, Charles Pritchard ch...@jumis.com wrote:
 On 3/5/2012 5:56 PM, Glenn Maynard wrote:

 On Mon, Mar 5, 2012 at 7:04 PM, Charles Pritchard ch...@jumis.com wrote:

 Do you see old behavior working something like the following?


 var blob = new Blob(my new big blob);
 var keepBlob = blob.slice(); destination.postMessage(blob, '*', [blob]);
 // is try/catch needed here?


 You don't need to do that.  If you don't want postMessage to transfer the
 blob, then simply don't include it in the transfer parameter, and it'll
 perform a normal structured clone.  postMessage behaves this way in part for
 backwards-compatibility: so exactly in cases like this, we can make Blob
 implement Transferable without breaking existing code.

 See http://dev.w3.org/html5/postmsg/#posting-messages and similar
 postMessage APIs.


 Web Intents won't have a transfer map argument.
 http://dvcs.w3.org/hg/web-intents/raw-file/tip/spec/Overview.html#widl-Intent-data

 For the Web Intents structured cloning algorithm, Web Intents would be
 inserting into step 3:
     If input is a Transferable object, add it to the transfer map.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#internal-structured-cloning-algorithm

 Then Web Intents would move the first section of the structured cloning
 algorithm to follow the internal cloning algorithm section, swapping their
 order.
 http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#safe-passing-of-structured-data

 That's my understanding.

 We've been discussing the merits of this approach vs using a transfer
 array argument. There's a lot to like about this alternative -- it
 conserves arguments and looks simpler than the transfer map, as well
 as not having the headaches of whether you can do (null, [port]) or
 (port, [port]) and concerns like that.

 The advantage of using the transfer map param is that it is more
 contiguous with existing practice. We'd kind of hoped that this
 particular debate was finalized before we got to the point of needing
 to make a decision, so we bluffed and left it out of the web intents
 spec draft. :-) At this point, I'm leaning toward needing to add a
 transfer map parameter, and then dealing with that alongside other
 uses, given the state of thinking on Transferables support and the
 need to make this pretty consistent across structure clone
 invocations.

 I do think that complexity might be better solved by the type system
 (i.e. a new Transferable(ArrayBuffer)), which would require a
 different developer mechanic to set up clone vs transfer, but would
 relieve complexity in the invocation of structured clone itself:
 transferables could just always transfer transparently. I don't know
 if, given current practice with MessagePort, that kind of solution is
 available.

 A change like this would be feasible as long as it doesn't break
 compatibility. In other words, the current Transferable array would
 still need to be supported, but Transferable instances (or perhaps
 instances of some other type) wrapping another Transferable object
 would also express the intent.

 The current API for Transferable and postMessage was informed by the
 realization that the previous sequenceMessagePort argument to
 postMessage was essentially already expressing the Transferable
 concept.

 I'm not familiar with the Web Intents API, but at first glance it
 seems feasible to overload the constructor, postResult and postFailure
 methods to support passing a sequenceTransferable as the last
 argument. This would make the API look more like postMessage and avoid
 adding more transfer semantics. Is that possible?


 Something like this may be necessary if Blob were a Transferable:
 var keepBlob = blob.slice();
 var intent = new Intent(-x-my-intent, blob);
 navigator.startActivity(intent, callback);

 And we might have an error on postMessage stashing it in the transfer
 array if it's not a Transferable on an older browser.




 Example of how easy the neutered concept applies to Transferrable:

 var blob = new Blob(my big blob);
 blob.close();


 I like the idea of having Blob implement Transferrable and adding close to
 the Transferrable interface.
 File.close could have a better relationship with the cache and/or locks on
 data.

 I'm not sure that adding close() to Transferable is a good idea. Not
 all Transferable types may want to support that explicit operation.
 What about adding 

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Glenn Maynard]

On Tue, Mar 6, 2012 at 3:18 PM, Kenneth Russell k...@google.com wrote:
 A change like this would be feasible as long as it doesn't break
 compatibility. In other words, the current Transferable array would
 still need to be supported, but Transferable instances (or perhaps
 instances of some other type) wrapping another Transferable object
 would also express the intent.

I don't like this, because it loses the invariant that structured clone is
a const operation by default.  That is, if you don't explicitly specify
objects in the transfer map, structured clone guarantees that the object
you pass in won't be modified, and that (as a corollary) you can create as
many structured clones as you want.  This always works:

function(obj)
{
thread1.postMessage(obj);
thread2.postMessage(obj);
thread3.postMessage(obj);
}

With the wrapper approach, it would no longer be guaranteed to work.

I'm not sure that adding close() to Transferable is a good idea. Not
 all Transferable types may want to support that explicit operation.
 What about adding close() to Blob, and having the neutering operation
 on Blob be defined to call close() on it?


When would you explicitly want to disallow manually neutering a
Transferable object?

The only case I can think where it's not obviously useful is MessagePort,
but I think that will be an exceptional case and that most Transferables
will want this method.  ArrayBuffer (currently the only other Transferable)
will, of course.

On Tue, Mar 6, 2012 at 3:34 PM, Michael Nordman micha...@google.com
 wrote:

 Separately defining blobs to be transferrable feels like an unneeded
 complexity. A caller wishing to
 neuter after sending can explicit call .close() rather than relying on
 more obscure artifacts of having also put the 'blob' in a
 'transferrable' array.


You can always call close() yourself, but Blob.close() should use the
neuter mechanism already there, not make up a new one.

-- 
Glenn Maynard

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Michael Nordman]

 Separately defining blobs to be transferrable feels like an unneeded
 complexity. A caller wishing to
 neuter after sending can explicit call .close() rather than relying on
 more obscure artifacts of having also put the 'blob' in a
 'transferrable' array.


 You can always call close() yourself, but Blob.close() should use the
 neuter mechanism already there, not make up a new one.

Blobs aren't transferable, there is no existing mechanism that applies
to them. Adding a blob.close() method is independent of making blob's
transferable, the former is not prerequisite on the latter.

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Kenneth Russell]

On Tue, Mar 6, 2012 at 1:31 PM, Arun Ranganathan
aranganat...@mozilla.com wrote:
 Ken,

 I'm not sure that adding close() to Transferable is a good idea. Not
 all Transferable types may want to support that explicit operation.
 What about adding close() to Blob, and having the neutering operation
 on Blob be defined to call close() on it?


 Specifically, you think this is not something ArrayBuffer should inherit?  If 
 it's also a bad idea for MessagePort, then those are really our only two use 
 cases of Transferable right now.  I'm happy to create something like a 
 close() on Blob.

MessagePort already defines a close() operation, so there's really no
question of whether its presence is a good or bad idea there. A
close() operation seems necessary in networking style APIs.

I would be hesitant to impose a close() method on all future
Transferable types. I don't think adding one to ArrayBuffer would be a
bad idea but I think that ideally it wouldn't be necessary. On memory
constrained devices, it would still be more efficient to re-use large
ArrayBuffers rather than close them and allocate new ones.


On Tue, Mar 6, 2012 at 1:34 PM, Michael Nordman micha...@google.com wrote:
 Sounds like there's a good case for an explicit blob.close() method
 independent of 'transferable'. Separately defining blobs to be
 transferrable feels like an unneeded complexity. A caller wishing to
 neuter after sending can explicit call .close() rather than relying on
 more obscure artifacts of having also put the 'blob' in a
 'transferrable' array.

This sounds like a good idea. As you pointed out offline, a key
difference between Blobs and ArrayBuffers is that Blobs are always
immutable. It isn't necessary to define Transferable semantics for
Blobs in order to post them efficiently, but it was essential for
ArrayBuffers.

-Ken

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Charles Pritchard]

On Mar 6, 2012, at 2:25 PM, Kenneth Russell k...@google.com wrote:

 On Tue, Mar 6, 2012 at 1:31 PM, Arun Ranganathan
 aranganat...@mozilla.com wrote:
 Ken,
 
 I'm not sure that adding close() to Transferable is a good idea. Not
 all Transferable types may want to support that explicit operation.
 What about adding close() to Blob, and having the neutering operation
 on Blob be defined to call close() on it?
 
 
 Specifically, you think this is not something ArrayBuffer should inherit?  
 If it's also a bad idea for MessagePort, then those are really our only two 
 use cases of Transferable right now.  I'm happy to create something like a 
 close() on Blob.
 
 MessagePort already defines a close() operation, so there's really no
 question of whether its presence is a good or bad idea there. A
 close() operation seems necessary in networking style APIs.
 
 I would be hesitant to impose a close() method on all future
 Transferable types. I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.


By definition, Transferable objects can be neutered; we're talking about an 
explicit method for it. After that, it's up to implementers.

I prefer .close to .release. 

An ArrayBuffer using 12megs of ram is something I want to release ASAP on 
mobile.

.close would still allow for the optimization you're implying in memory mapping.





 
 
 On Tue, Mar 6, 2012 at 1:34 PM, Michael Nordman micha...@google.com wrote:
 Sounds like there's a good case for an explicit blob.close() method
 independent of 'transferable'. Separately defining blobs to be
 transferrable feels like an unneeded complexity. A caller wishing to
 neuter after sending can explicit call .close() rather than relying on
 more obscure artifacts of having also put the 'blob' in a
 'transferrable' array.
 
 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.
 

Making Blob a Transferable may simplify the structured clone algorithm; Blob an 
File would no longer be explicitly listed. Adding close to Transferable would 
simplify three different objects -- ArrayBuffer, MessagePort and Blob. In 
theory anyway.

While Blob doesn't need the transfer map optimization, it may be helpful in the 
context of web intents postMessage as it would release the Blob references from 
one window, possibly making GC a little easier. That's just a guess... But this 
thread is about enhancing the process. Seems reasonable that this would be a 
side effect.

File.close() may have implementation side effects, such as releasing read locks 
on underlying files.

-Charles

RE: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Feras Moussa]

 -Original Message-
 From: Arun Ranganathan [mailto:aranganat...@mozilla.com]
 Sent: Tuesday, March 06, 2012 1:32 PM
 To: Kenneth Russell
 Cc: public-webapps@w3.org; Charles Pritchard; Glenn Maynard; Feras
 Moussa; Adrian Bateman; Greg Billock
 Subject: Re: Transferable and structured clones, was: Re: [FileAPI]
 Deterministic release of Blob proposal
 
 Ken,
 
  I'm not sure that adding close() to Transferable is a good idea. Not
  all Transferable types may want to support that explicit operation.
  What about adding close() to Blob, and having the neutering operation
  on Blob be defined to call close() on it?
 
 
 Specifically, you think this is not something ArrayBuffer should inherit?  If 
 it's
 also a bad idea for MessagePort, then those are really our only two use cases
 of Transferable right now.  I'm happy to create something like a close() on
 Blob.
 
 -- A*
We agree Blobs do not need to be transferrable, and thus it makes sense to have 
close directly on Blob, independent of being transferable.

Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal

[Glenn Maynard]

On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote:

  You can always call close() yourself, but Blob.close() should use the
  neuter mechanism already there, not make up a new one.

 Blobs aren't transferable, there is no existing mechanism that applies
 to them. Adding a blob.close() method is independent of making blob's
 transferable, the former is not prerequisite on the latter.


There is an existing mechanism for closing objects.  It's called
neutering.  Blob.close should use the same terminology, whether or not
the object is a Transferable.

On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote:

 I would be hesitant to impose a close() method on all future
 Transferable types.


Why?  All Transferable types must define how to neuter objects; all close()
does is trigger it.

I don't think adding one to ArrayBuffer would be a
 bad idea but I think that ideally it wouldn't be necessary. On memory
 constrained devices, it would still be more efficient to re-use large
 ArrayBuffers rather than close them and allocate new ones.


That's often not possible, when the ArrayBuffer is returned to you from an
API (eg. XHR2).

 This sounds like a good idea. As you pointed out offline, a key
 difference between Blobs and ArrayBuffers is that Blobs are always
 immutable. It isn't necessary to define Transferable semantics for
 Blobs in order to post them efficiently, but it was essential for
 ArrayBuffers.


No new semantics need to be defined; the semantics of Transferable are
defined by postMessage and are the same for all transferable objects.
That's already done.  The only thing that needs to be defined is how to
neuter an object, which is what Blob.close() has to define anyway.

Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future
large, structured clonable objects to all be released with the same
mechanisms: either pass them in the transfer argument to a postMessage
call, or use the consistent, identical close() method inherited from
Transferable.  This allows developers to think of the transfer list as a
list of objects which won't be needed after the postMessage call.  It
doesn't matter that the underlying optimizations are different; the visible
side-effects are identical (the object can no longer be accessed).

-- 
Glenn Maynard