Re: Async Image - ImageData conversion
On 6/26/15 4:07 AM, Ashley Gullen wrote: I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. Maybe I wasn't clear. I was suggesting that we have the methods on both HTMLImageElement and ImageBitmap (and possibly on any other things we feel should have the methods directly). I like the suggestion that ImageBitmap be the hub of image conversion, I agree that it sounds appealing, but it means ImageBitmap now has to serve two masters: it has to be something that you can paint from quickly (premultiplied, probably lives on the GPU) _and_ it needs to be something you can transferToImageData efficiently (better to not live on the GPU for this). Maybe that's OK; it's just a bit of a warning flag from my point of view when a single object is meant to do multiple quite different things; it makes it harder to have it be good at all of them... -Boris
Re: Async Image - ImageData conversion
Gosh, I keep putting the wrong terms in and it really confuses things. I need to get better at writing clearly! I mentioned transferToImageBitmap when I meant transferToImageData. The parargraph from my second-last email should read: I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageData() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. I understand that ImageBitmaps were not intended to be mutated, but the OffscreenCanvas proposal is already suggesting that there be a close() method (to immediately release it), and that they be transferrable (which mutates them on the thread they are transferred from). These appear to be essential requirements for the OffscreenCanvas proposal to work efficiently. If we are going to have this anyway, then it seems reasonable to use the same mechanism for ImageBitmap.transferToImageData(). Also there is still no direct access to the ImageBitmap pixel data, it's only mutable in the sense it can be reset to empty. The same thing could happen for ImageData.transferToImageBitmap(), where the ImageData is neutered and it returns an ImageBitmap which can share the same storage, but I guess this is less controversial since ImageData is already mutable. On 26 June 2015 at 14:10, Justin Novosad ju...@google.com wrote: On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I'm also wondering that if we have ImageBitmap.transferToImageData(), should we not also have ImageData.transferToImageBitmap()? Currently createImageBitmap(imageData) appears to require making a copy. On 26 June 2015 at 12:07, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Thu, Jun 25, 2015 at 4:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 6/24/15 1:28 PM, Ashley Gullen wrote: Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. Indeed. This is another situation where going directly from an HTMLImageElement to an ImageData or Blob would make some sense, because it would allow the UA to avoid the GPU upload and readback, right? I'm not convinced the spec should worry about GPU uploads and readbacks. I think the term undue latency is conveniently vague, which gives implementers an interesting level of freedom to use smart heuristics and make implementation-specific or platform-specific or decisions. For example, I would argue that on many platforms uploading an image to the GPU probably does not constitute undue latency, but a readback probably does. Under those assumptions, an implementation may decide that ImageBitmaps should be lazily uploaded to the GPU the first time they're drawn. That way the GPU can be short-circuited when appropriate. Also, the spec requires that ImageBitmaps be *drawn* without undue latency, it does not require that converting to a Blob or an ImageData be lightning fast. Anyways, my point is that implementers have enough wiggle room to do the right thing even without direct methods on img. -Boris
Re: Async Image - ImageData conversion
I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Clipboard API: remove dangerous formats from mandatory data types
Florian, as I pointed out earlier, this proposal is to remove the requirement that user agents allow content to set the local system clipboard directly with certain pre-existing clipboard formats, because doing so safely is not possible in general. Removing the requirement from the spec will simply mean that the spec more accurately reflects what is actually implemented in existing user agents. Defining how user agents can/should support setting of arbitrary formats is a separate discussion; user agents don't generally support that - including not supporting application/octet-stream, which the spec doesn't actually define the behaviour of! - so it would in effect be a new feature of the API the behaviour of which would need to be properly specified. Please feel free to fork this thread if that's something you'd like to propose ideas for. :) Thanks! On Thu, 25 Jun 2015 at 19:13 Florian Bösch pya...@gmail.com wrote: I think you underestimate the integrative need that web-apps will acquire and the lengths they will go to faced with a business need to make it work once clipboard API becomes common developer knowledge.
