Re: Async Image - ImageData conversion
On Mon, Jul 6, 2015 at 2:38 PM, Ashley Gullen ash...@scirra.com wrote: I thought it would be more confusing to have separate competing proposals in one document? They are more like three revisions of the same document anyway, the latest one including all feedback and rationale so far: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Sorry, my bad. It seems the other comments still apply though. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I'm new to writing specs and when I first looked ReSpec had templates and a guide - is there something similar for the HTML standard style spec? I changed toBlob to use a dictionary (still at https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html). Ashley On 7 July 2015 at 10:48, Anne van Kesteren ann...@annevk.nl wrote: On Mon, Jul 6, 2015 at 2:38 PM, Ashley Gullen ash...@scirra.com wrote: I thought it would be more confusing to have separate competing proposals in one document? They are more like three revisions of the same document anyway, the latest one including all feedback and rationale so far: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Sorry, my bad. It seems the other comments still apply though. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Tue, Jul 7, 2015 at 12:24 PM, Ashley Gullen ash...@scirra.com wrote: I'm new to writing specs and when I first looked ReSpec had templates and a guide - is there something similar for the HTML standard style spec? I think you can tweak ReSpec to do the right thing, though most new specifications are written using https://github.com/tabatkins/bikeshed/ these days. I changed toBlob to use a dictionary (still at https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html). Cool, though you need to actually define the dictionary. can*Type should probably return an enum. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I thought it would be more confusing to have separate competing proposals in one document? They are more like three revisions of the same document anyway, the latest one including all feedback and rationale so far: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html On 3 July 2015 at 13:04, Anne van Kesteren ann...@annevk.nl wrote: On Fri, Jul 3, 2015 at 1:51 PM, Ashley Gullen ash...@scirra.com wrote: These conversions are covered by what I proposed in my previous email, so I drafted another spec with them: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Could you please put them all in a single document? That would make it a bit easier to follow what is being proposed. And perhaps not use Respec or figure out some way to get them in a style similar to the original HTML Standard. This non-algorithmic way of describing methods is error prone. A new toBlob() should probably also take a dictionary as second argument rather than the any approach. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On 2 July 2015 at 21:36, Jeff Muizelaar jmuizel...@mozilla.com wrote:
The important ones of these are:
Blob - ImageData
HTMLImageElement - ImageData
ImageData - Blob
These conversions are covered by what I proposed in my previous email, so I
drafted another spec with them:
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html
I also mentioned canEncodeType and canDecodeType in another email thread,
but I think they are important to the same use cases, so I've put them in
the draft as well to provoke discussion.
Ashley
There are a bunch of conversions that aren't covered directly by these but
I'm
not sure it makes us much sense to short circut them beyond what
functionality
is currently exposed. e.g.
Going HTMLVideoElement - ImageBitmap - Blob
isn't really much worse than doing:
HTMLVideoElement - CanvasRenderContext2D - Blob
-Jeff
On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad ju...@google.com wrote:
Making ImageData the hub would imply more copies in memory in many
cases.
Because ImageData is mutable (not to mention the alpha multiplication
issues
which are also a factor), it cannot share its image buffer with the
source
it was created from, unlike ImageBitmap. Immutability is a significant
advantage of ImageBitmap, which allows for zero-copy code paths in many
cases, which helps with both performance and memory consumption.
On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen ash...@scirra.com
wrote:
If it's difficult to make ImageBitmap both an efficient drawing source
and
a conversion intermediary, then we could add conversion functions to
each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to
an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData,
using
Blob.toImageData(). It also has a larger surface area, requiring
changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like
this to
ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob,
ImageBitmap via createImageBitmap, or a canvas via putImageData (which
is
still synchronous, but no longer needs to be done purely for conversion
reasons, so probably means you really want it to appear in the canvas
and
therefore should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's
no
more transferring going on either, which I think is now unnecessary
since
you can get from HTMLImageElement to ImageData with one call. I think
it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image
data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on
both
HTMLImageElement and ImageBitmap (and possibly on any other things we
feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image
conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from
quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be
something
you can transferToImageData efficiently (better to not live on the GPU
for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of
view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
On Fri, Jul 3, 2015 at 1:51 PM, Ashley Gullen ash...@scirra.com wrote: These conversions are covered by what I proposed in my previous email, so I drafted another spec with them: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Could you please put them all in a single document? That would make it a bit easier to follow what is being proposed. And perhaps not use Respec or figure out some way to get them in a style similar to the original HTML Standard. This non-algorithmic way of describing methods is error prone. A new toBlob() should probably also take a dictionary as second argument rather than the any approach. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I'm not sure having a hub is worth the implementation pain.
It looks like there are three pretty different things here:
ImageBitmap - Immutable images for drawing on canvas. Typically
premultiplied data located on the GPU
ImageData - Mutable script accessible array of data. Unpremultiplied
data located in main memory
ImageElement/Blob - A pointer to some compressed image data.
Unpremultiplied - premultiplied is a lossy conversion step which can
cause additional implementor/user pain.
i.e. if you have an ImageData and you want to make it into a blob,
it's not clear if the browser will
do unpremul - premul - unpremul encoded images or unpremul -
unpremul encoded.
It's seems like the easiest thing for users and implementors is just
to add the necessary functions in all
of the important places:
The important ones of these are:
Blob - ImageData
HTMLImageElement - ImageData
ImageData - Blob
There are a bunch of conversions that aren't covered directly by these but I'm
not sure it makes us much sense to short circut them beyond what functionality
is currently exposed. e.g.
Going HTMLVideoElement - ImageBitmap - Blob
isn't really much worse than doing:
HTMLVideoElement - CanvasRenderContext2D - Blob
-Jeff
On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad ju...@google.com wrote:
Making ImageData the hub would imply more copies in memory in many cases.
Because ImageData is mutable (not to mention the alpha multiplication issues
which are also a factor), it cannot share its image buffer with the source
it was created from, unlike ImageBitmap. Immutability is a significant
advantage of ImageBitmap, which allows for zero-copy code paths in many
cases, which helps with both performance and memory consumption.
On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen ash...@scirra.com wrote:
If it's difficult to make ImageBitmap both an efficient drawing source and
a conversion intermediary, then we could add conversion functions to each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData, using
Blob.toImageData(). It also has a larger surface area, requiring changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like this to
ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob,
ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
still synchronous, but no longer needs to be done purely for conversion
reasons, so probably means you really want it to appear in the canvas and
therefore should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's no
more transferring going on either, which I think is now unnecessary since
you can get from HTMLImageElement to ImageData with one call. I think it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on both
HTMLImageElement and ImageBitmap (and possibly on any other things we feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be something
you can transferToImageData efficiently (better to not live on the GPU for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
Making ImageData the hub would imply more copies in memory in many cases.
Because ImageData is mutable (not to mention the alpha multiplication
issues which are also a factor), it cannot share its image buffer with the
source it was created from, unlike ImageBitmap. Immutability is a
significant advantage of ImageBitmap, which allows for zero-copy code paths
in many cases, which helps with both performance and memory consumption.
On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen ash...@scirra.com wrote:
If it's difficult to make ImageBitmap both an efficient drawing source and
a conversion intermediary, then we could add conversion functions to each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData, using
Blob.toImageData(). It also has a larger surface area, requiring changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like this
to ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob,
ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
still synchronous, but no longer needs to be done purely for conversion
reasons, so probably means you really want it to appear in the canvas and
therefore should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's no
more transferring going on either, which I think is now unnecessary since
you can get from HTMLImageElement to ImageData with one call. I think it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on both
HTMLImageElement and ImageBitmap (and possibly on any other things we feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be something
you can transferToImageData efficiently (better to not live on the GPU for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
If it's difficult to make ImageBitmap both an efficient drawing source and
a conversion intermediary, then we could add conversion functions to each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData, using
Blob.toImageData(). It also has a larger surface area, requiring changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like this
to ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob, ImageBitmap
via createImageBitmap, or a canvas via putImageData (which is still
synchronous, but no longer needs to be done purely for conversion reasons,
so probably means you really want it to appear in the canvas and therefore
should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's no
more transferring going on either, which I think is now unnecessary since
you can get from HTMLImageElement to ImageData with one call. I think it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on both
HTMLImageElement and ImageBitmap (and possibly on any other things we feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be something
you can transferToImageData efficiently (better to not live on the GPU for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
On 6/26/15 4:07 AM, Ashley Gullen wrote: I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. Maybe I wasn't clear. I was suggesting that we have the methods on both HTMLImageElement and ImageBitmap (and possibly on any other things we feel should have the methods directly). I like the suggestion that ImageBitmap be the hub of image conversion, I agree that it sounds appealing, but it means ImageBitmap now has to serve two masters: it has to be something that you can paint from quickly (premultiplied, probably lives on the GPU) _and_ it needs to be something you can transferToImageData efficiently (better to not live on the GPU for this). Maybe that's OK; it's just a bit of a warning flag from my point of view when a single object is meant to do multiple quite different things; it makes it harder to have it be good at all of them... -Boris
Re: Async Image - ImageData conversion
Gosh, I keep putting the wrong terms in and it really confuses things. I need to get better at writing clearly! I mentioned transferToImageBitmap when I meant transferToImageData. The parargraph from my second-last email should read: I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageData() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. I understand that ImageBitmaps were not intended to be mutated, but the OffscreenCanvas proposal is already suggesting that there be a close() method (to immediately release it), and that they be transferrable (which mutates them on the thread they are transferred from). These appear to be essential requirements for the OffscreenCanvas proposal to work efficiently. If we are going to have this anyway, then it seems reasonable to use the same mechanism for ImageBitmap.transferToImageData(). Also there is still no direct access to the ImageBitmap pixel data, it's only mutable in the sense it can be reset to empty. The same thing could happen for ImageData.transferToImageBitmap(), where the ImageData is neutered and it returns an ImageBitmap which can share the same storage, but I guess this is less controversial since ImageData is already mutable. On 26 June 2015 at 14:10, Justin Novosad ju...@google.com wrote: On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I'm also wondering that if we have ImageBitmap.transferToImageData(), should we not also have ImageData.transferToImageBitmap()? Currently createImageBitmap(imageData) appears to require making a copy. On 26 June 2015 at 12:07, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Thu, Jun 25, 2015 at 4:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 6/24/15 1:28 PM, Ashley Gullen wrote: Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. Indeed. This is another situation where going directly from an HTMLImageElement to an ImageData or Blob would make some sense, because it would allow the UA to avoid the GPU upload and readback, right? I'm not convinced the spec should worry about GPU uploads and readbacks. I think the term undue latency is conveniently vague, which gives implementers an interesting level of freedom to use smart heuristics and make implementation-specific or platform-specific or decisions. For example, I would argue that on many platforms uploading an image to the GPU probably does not constitute undue latency, but a readback probably does. Under those assumptions, an implementation may decide that ImageBitmaps should be lazily uploaded to the GPU the first time they're drawn. That way the GPU can be short-circuited when appropriate. Also, the spec requires that ImageBitmaps be *drawn* without undue latency, it does not require that converting to a Blob or an ImageData be lightning fast. Anyways, my point is that implementers have enough wiggle room to do the right thing even without direct methods on img. -Boris
Re: Async Image - ImageData conversion
I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On Thu, Jun 25, 2015 at 9:40 AM, Justin Novosad ju...@google.com wrote: On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen ash...@scirra.com wrote: I see that OffscreenCanvas also specifies a close() method for ImageBitmap. Perhaps browsers could use copy-on-write with the toImageData() method? Then you can efficiently either transfer or copy depending on if you close the ImageBitmap you transferred to an ImageData. Transfer: createImageBitmap() - imageBitmap.toImageData() - imageBitmap.close() No copy made, contents transferred to the imageData createImageBitmap() - imageBitmap.toImageData() - access resulting imageData.data Copy made on first access Copy on write is probably not a good idea because it would require injecting an observer pattern (or some kind of notification mechanism) into Uint8ClampedArray data assignment, which would realistically degrade performance of Image processing pixel loops. Having a one step transfer API (transferToImageData) avoids adding overhead to what is otherwise just a simple memory access. Something more: Blink, and probably other implementations as well, stores decoded images in alpha pre-multiplied form. ImageData is not pre-multiplied. For this reason sharing a buffer between ImageBitmap and ImageData would be somewhat inconvenient. On 24 June 2015 at 21:54, Kenneth Russell k...@google.com wrote: On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices.
Re: Async Image - ImageData conversion
On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen ash...@scirra.com wrote: I see that OffscreenCanvas also specifies a close() method for ImageBitmap. Perhaps browsers could use copy-on-write with the toImageData() method? Then you can efficiently either transfer or copy depending on if you close the ImageBitmap you transferred to an ImageData. Transfer: createImageBitmap() - imageBitmap.toImageData() - imageBitmap.close() No copy made, contents transferred to the imageData createImageBitmap() - imageBitmap.toImageData() - access resulting imageData.data Copy made on first access Copy on write is probably not a good idea because it would require injecting an observer pattern (or some kind of notification mechanism) into Uint8ClampedArray data assignment, which would realistically degrade performance of Image processing pixel loops. Having a one step transfer API (transferToImageData) avoids adding overhead to what is otherwise just a simple memory access. On 24 June 2015 at 21:54, Kenneth Russell k...@google.com wrote: On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
I see that OffscreenCanvas also specifies a close() method for ImageBitmap. Perhaps browsers could use copy-on-write with the toImageData() method? Then you can efficiently either transfer or copy depending on if you close the ImageBitmap you transferred to an ImageData. Transfer: createImageBitmap() - imageBitmap.toImageData() - imageBitmap.close() No copy made, contents transferred to the imageData createImageBitmap() - imageBitmap.toImageData() - access resulting imageData.data Copy made on first access On 24 June 2015 at 21:54, Kenneth Russell k...@google.com wrote: On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
On 6/24/15 1:28 PM, Ashley Gullen wrote: I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? Yes, it is. The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. Decoding can easily take hundreds of milliseconds. It definitely constitutes undue latency from our point of view. - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. This has the premultiplied vs non-premultiplied problem Justin points out. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. This would be a viable option. Another option is to add toBlob/toImageData directly to HTMLImageElement and the other things we want it on, in addition to having them on ImageBitmap. Then you wouldn't need to go through an ImageBitmap at all. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. Indeed. This is another situation where going directly from an HTMLImageElement to an ImageData or Blob would make some sense, because it would allow the UA to avoid the GPU upload and readback, right? The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. -Boris
Re: Async Image - ImageData conversion
On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Subject=Re: Async Image - ImageData conversion
Good point - it makes sense to have ImageBitmap as the hub of all conversion. I've drafted a new spec which instead proposes ImageBitmap.toBlob() and ImageBitmap.toImageData(): https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html This should cover all conversion cases asynchronously. On 23 June 2015 at 20:34, Justin Novosad ju...@google.com wrote: Based on feedback received from web developers, new APIs that move image data around should also strive to eliminate intermediate copies of the image data to avoid memory bloat. I think your proposal achieves that, but I think memory footprint reduction could be a stated objective of the proposal. For example, the current solution of using a canvas forces the creation of an intermediate copy (the canvas). Also, to avoid the multiplication of APIs for moving image data between various representations, I would like to suggest an alternative: using the ImageBitmap interface as a hub. The creation of ImageBitmaps is already asynchronous (createImageBitmap returns a promise), and it has overloads for acquiring images from img, video, canvas, url, blob, imagedata. All that is missing are a few methods for getting data directly out of an ImageBitmap. So I think adding toBlob and toImageData (both async) to ImageBitmap is a more succinct proposal that would address your use cases, and many additional ones at the same time.
Re: Async Image - ImageData conversion
Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Let me know if you have any feedback on this. Ashley, I assume the relevant link is now https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html, right? I'm trying to understand where the methods are being added. The IDL says partial interface ImageBitmapConversion but there is no current ImageBitmapConversion interface I'm aware of. Is this supposed to be partial interface ImageBitmap? -Boris
Subject=Re: Async Image - ImageData conversion
Based on feedback received from web developers, new APIs that move image data around should also strive to eliminate intermediate copies of the image data to avoid memory bloat. I think your proposal achieves that, but I think memory footprint reduction could be a stated objective of the proposal. For example, the current solution of using a canvas forces the creation of an intermediate copy (the canvas). Also, to avoid the multiplication of APIs for moving image data between various representations, I would like to suggest an alternative: using the ImageBitmap interface as a hub. The creation of ImageBitmaps is already asynchronous (createImageBitmap returns a promise), and it has overloads for acquiring images from img, video, canvas, url, blob, imagedata. All that is missing are a few methods for getting data directly out of an ImageBitmap. So I think adding toBlob and toImageData (both async) to ImageBitmap is a more succinct proposal that would address your use cases, and many additional ones at the same time.
Re: Async Image - ImageData conversion
I was advised to mark this as an Editor's draft, and have done so. I also
added a new proposed ImageData.fromImage method, with rationale included in
the document. It's at the same URL:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
On 19 June 2015 at 13:43, Ashley Gullen ash...@scirra.com wrote:
I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
Let me know if you have any feedback on this.
On 18 June 2015 at 22:27, Travis Leithead travis.leith...@microsoft.com
wrote:
Cool. Want to write it up as a small extension spec? We have a new
Incubator Community Group coming up soon that would be an ideal place to
drop a small spec into and let implementers experiment with it and get
broader community feedback on the idea.
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 2:06 PM
*To:* Travis Leithead
*Cc:* public-webapps@w3.org
*Subject:* Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a
putImageDataAsync counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular
Blob - ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is
all async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data
from the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or
similar. It would also be useful if we could decode directly from Blob to
ImageData without having to go via an Image as well (perhaps
ImageData.fromBlob(blob) ?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D
Re: Async Image - ImageData conversion
I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
Let me know if you have any feedback on this.
On 18 June 2015 at 22:27, Travis Leithead travis.leith...@microsoft.com
wrote:
Cool. Want to write it up as a small extension spec? We have a new
Incubator Community Group coming up soon that would be an ideal place to
drop a small spec into and let implementers experiment with it and get
broader community feedback on the idea.
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 2:06 PM
*To:* Travis Leithead
*Cc:* public-webapps@w3.org
*Subject:* Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a
putImageDataAsync counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob
- ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is
all async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data from
the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.
Ashley Gullen
Scirra.com
RE: Async Image - ImageData conversion
Cool. Want to write it up as a small extension spec? We have a new Incubator
Community Group coming up soon that would be an ideal place to drop a small
spec into and let implementers experiment with it and get broader community
feedback on the idea.
From: a...@scirra.com [mailto:a...@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 2:06 PM
To: Travis Leithead
Cc: public-webapps@w3.org
Subject: Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a putImageDataAsync
counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage - getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load blob
URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but what
happens if you make other synchronous draw calls while putImageDataAsync is
processing? What actually ends up in the canvas and how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob -
ImageData and ImageData - Image. If implementors think they can optimise these
to be as efficient as direct conversion that's fine, but is that actually
doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob - ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is all
async.
I was thinking about toImage and fromImage, but it seems to be a bigger step to
add a new way to make images, and there's the question of what the src for the
image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead
travis.leith...@microsoft.commailto:travis.leith...@microsoft.com wrote:
I think solving at least the first-order problem of extracting data from the
Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add ☺
From: a...@scirra.commailto:a...@scirra.com
[mailto:a...@scirra.commailto:a...@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 10:00 AM
To: public-webapps@w3.orgmailto:public-webapps@w3.org
Subject: Async Image - ImageData conversion
I was wondering if there is anything on the standards track to asynchronously
get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and at
some point we need to get an ImageData for this. Currently the only way I know
of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop machine.
Maybe this can be faster but I really doubt this whole function is likely to be
done in 16ms on low-end devices, so I believe there ought to be an async
alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar. It
would also be useful if we could decode directly from Blob to ImageData without
having to go via an Image as well (perhaps ImageData.fromBlob(blob) ?).
More broadly, it would be good to have a comprehensive set of async conversion
functions between Blob, Image, and ImageData. Blob - Image can already be done
by setting the image src to a blob URL, but basically everything else requires
going synchronously through a 2D canvas, which janks apps which have to process
lots of large images. Running a canvas in a worker could possibly help, but
that would require both Blob and Image to be transferable to be useful.
Ashley Gullen
Scirra.com
RE: Async Image - ImageData conversion
I think solving at least the first-order problem of extracting data from the
Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add ☺
From: a...@scirra.com [mailto:a...@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 10:00 AM
To: public-webapps@w3.org
Subject: Async Image - ImageData conversion
I was wondering if there is anything on the standards track to asynchronously
get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and at
some point we need to get an ImageData for this. Currently the only way I know
of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop machine.
Maybe this can be faster but I really doubt this whole function is likely to be
done in 16ms on low-end devices, so I believe there ought to be an async
alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar. It
would also be useful if we could decode directly from Blob to ImageData without
having to go via an Image as well (perhaps ImageData.fromBlob(blob) ?).
More broadly, it would be good to have a comprehensive set of async conversion
functions between Blob, Image, and ImageData. Blob - Image can already be done
by setting the image src to a blob URL, but basically everything else requires
going synchronously through a 2D canvas, which janks apps which have to process
lots of large images. Running a canvas in a worker could possibly help, but
that would require both Blob and Image to be transferable to be useful.
Ashley Gullen
Scirra.com
Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a putImageDataAsync
counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob
- ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is all
async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data from
the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.
Ashley Gullen
Scirra.com
