Re: Async Image -> ImageData conversion

[Anne van Kesteren]

On Tue, Jul 7, 2015 at 12:24 PM, Ashley Gullen  wrote:
> I'm new to writing specs and when I first looked ReSpec had templates and a
> guide - is there something similar for the HTML standard style spec?

I think you can tweak ReSpec to do the right thing, though most new
specifications are written using
https://github.com/tabatkins/bikeshed/ these days.


> I changed toBlob to use a dictionary (still at
> https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html).

Cool, though you need to actually define the dictionary. can*Type
should probably return an enum.


-- 
https://annevankesteren.nl/

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I'm new to writing specs and when I first looked ReSpec had templates and a
guide - is there something similar for the HTML standard style spec?

I changed toBlob to use a dictionary (still at
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html).

Ashley


On 7 July 2015 at 10:48, Anne van Kesteren  wrote:

> On Mon, Jul 6, 2015 at 2:38 PM, Ashley Gullen  wrote:
> > I thought it would be more confusing to have separate competing
> proposals in
> > one document? They are more like three revisions of the same document
> > anyway, the latest one including all feedback and rationale so far:
> > https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html
>
> Sorry, my bad. It seems the other comments still apply though.
>
>
> --
> https://annevankesteren.nl/
>

Re: Async Image -> ImageData conversion

[Anne van Kesteren]

On Mon, Jul 6, 2015 at 2:38 PM, Ashley Gullen  wrote:
> I thought it would be more confusing to have separate competing proposals in
> one document? They are more like three revisions of the same document
> anyway, the latest one including all feedback and rationale so far:
> https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html

Sorry, my bad. It seems the other comments still apply though.


-- 
https://annevankesteren.nl/

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I thought it would be more confusing to have separate competing proposals
in one document? They are more like three revisions of the same document
anyway, the latest one including all feedback and rationale so far:
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html



On 3 July 2015 at 13:04, Anne van Kesteren  wrote:

> On Fri, Jul 3, 2015 at 1:51 PM, Ashley Gullen  wrote:
> > These conversions are covered by what I proposed in my previous email,
> so I
> > drafted another spec with them:
> > https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html
>
> Could you please put them all in a single document? That would make it
> a bit easier to follow what is being proposed. And perhaps not use
> Respec or figure out some way to get them in a style similar to the
> original HTML Standard. This non-algorithmic way of describing methods
> is error prone.
>
> A new toBlob() should probably also take a dictionary as second
> argument rather than the "any" approach.
>
>
> --
> https://annevankesteren.nl/
>

Re: Async Image -> ImageData conversion

[Anne van Kesteren]

On Fri, Jul 3, 2015 at 1:51 PM, Ashley Gullen  wrote:
> These conversions are covered by what I proposed in my previous email, so I
> drafted another spec with them:
> https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html

Could you please put them all in a single document? That would make it
a bit easier to follow what is being proposed. And perhaps not use
Respec or figure out some way to get them in a style similar to the
original HTML Standard. This non-algorithmic way of describing methods
is error prone.

A new toBlob() should probably also take a dictionary as second
argument rather than the "any" approach.


-- 
https://annevankesteren.nl/

Re: Async Image -> ImageData conversion

[Ashley Gullen]

On 2 July 2015 at 21:36, Jeff Muizelaar  wrote:
>
> The important ones of these are:
> Blob -> ImageData
> HTMLImageElement -> ImageData
> ImageData -> Blob
>

These conversions are covered by what I proposed in my previous email, so I
drafted another spec with them:
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html

I also mentioned canEncodeType and canDecodeType in another email thread,
but I think they are important to the same use cases, so I've put them in
the draft as well to provoke discussion.

Ashley








>
> There are a bunch of conversions that aren't covered directly by these but
> I'm
> not sure it makes us much sense to short circut them beyond what
> functionality
> is currently exposed. e.g.
> Going HTMLVideoElement -> ImageBitmap -> Blob
> isn't really much worse than doing:
> HTMLVideoElement -> CanvasRenderContext2D -> Blob
>
> -Jeff
>
> On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad  wrote:
> > Making ImageData the "hub" would imply more copies in memory in many
> cases.
> > Because ImageData is mutable (not to mention the alpha multiplication
> issues
> > which are also a factor), it cannot share its image buffer with the
> source
> > it was created from, unlike ImageBitmap.  Immutability is a significant
> > advantage of ImageBitmap, which allows for zero-copy code paths in many
> > cases, which helps with both performance and memory consumption.
> >
> > On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen 
> wrote:
> >>
> >> If it's difficult to make ImageBitmap both an efficient drawing source
> and
> >> a conversion intermediary, then we could add conversion functions to
> each
> >> object that can be converted, e.g.:
> >>
> >> HTMLImageElement.toBlob()
> >> HTMLImageElement.toImageData()
> >> Blob.toImageData()
> >> ImageData.toBlob()
> >>
> >> This seems inconsistent. For example to convert a Blob to an Image, you
> >> should create a URL and set an Image's src to it; to convert a Blob to
> an
> >> ImageBitmap, you should use createImageBitmap(blob); and then under this
> >> proposal there's a third approach to convert a Blob to an ImageData,
> using
> >> Blob.toImageData(). It also has a larger surface area, requiring
> changes to
> >> several interfaces.
> >>
> >> So perhaps it would be better to make ImageData the "hub" of conversion,
> >> more like the first proposal I made. If ImageBitmap is a GPU-hosted
> >> premultiplied resource, then ImageData is an expressly not-on-GPU
> >> not-premultiplied alternative. That would mean adding something like
> this to
> >> ImageData:
> >>
> >> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
> >>
> >> partial interface ImageData {
> >> static Promise create(ImageDataSource source)
> >> Promise toBlob()
> >> }
> >>
> >> ImageData can also be converted to HTMLImageElement via toBlob,
> >> ImageBitmap via createImageBitmap, or a canvas via putImageData (which
> is
> >> still synchronous, but no longer needs to be done purely for conversion
> >> reasons, so probably means you really want it to appear in the canvas
> and
> >> therefore should remain synchronous).
> >>
> >> This covers all the conversion use cases I can think of without
> >> complicating ImageBitmap's "without undue latency" requirement. There's
> no
> >> more transferring going on either, which I think is now unnecessary
> since
> >> you can get from HTMLImageElement to ImageData with one call. I think
> it's
> >> more future-proof too since future types can be added to ImageDataSource
> >> allowing new types to be converted without a new method being added.
> >>
> >> Does this approach sound better?
> >>
> >> Ashley
> >>
> >>
> >> On 26 June 2015 at 16:37, Boris Zbarsky  wrote:
> >>>
> >>> On 6/26/15 4:07 AM, Ashley Gullen wrote:
> 
>  I don't think we should extend HTMLImageElement because it is not
>  available in workers. Adding the conversion methods to ImageBitmap
>  allows workers to perform conversions using Blob (compressed image
> data)
>  in the place of HTMLImageElement.
> >>>
> >>>
> >>> Maybe I wasn't clear.  I was suggesting that we have the methods on
> both
> >>> HTMLImageElement and ImageBitmap (and possibly on any other things we
> feel
> >>> should have the methods directly).
> >>>
>  I like the suggestion that "ImageBitmap be the hub of image
> conversion",
> >>>
> >>>
> >>> I agree that it sounds appealing, but it means ImageBitmap now has to
> >>> serve two masters: it has to be something that you can paint from
> quickly
> >>> (premultiplied, probably lives on the GPU) _and_ it needs to be
> something
> >>> you can transferToImageData efficiently (better to not live on the GPU
> for
> >>> this).
> >>>
> >>> Maybe that's OK; it's just a bit of a warning flag from my point of
> view
> >>> when a single object is meant to do multiple quite different things; it
> >>> makes it harder to have it be good at all of them...
> >>>
> >>> -Boris
> >>
> >>
> >
>

Re: Async Image -> ImageData conversion

[Jeff Muizelaar]

I'm not sure having a hub is worth the implementation pain.

It looks like there are three pretty different things here:

ImageBitmap - Immutable images for drawing on canvas. Typically
premultiplied data located on the GPU
ImageData - Mutable script accessible array of data. Unpremultiplied
data located in main memory
ImageElement/Blob - A pointer to some compressed image data.

Unpremultiplied -> premultiplied is a lossy conversion step which can
cause additional implementor/user pain.
i.e. if you have an ImageData and you want to make it into a blob,
it's not clear if the browser will
do unpremul -> premul -> unpremul encoded images or unpremul ->
unpremul encoded.

It's seems like the easiest thing for users and implementors is just
to add the necessary functions in all
of the important places:

The important ones of these are:
Blob -> ImageData
HTMLImageElement -> ImageData
ImageData -> Blob

There are a bunch of conversions that aren't covered directly by these but I'm
not sure it makes us much sense to short circut them beyond what functionality
is currently exposed. e.g.
Going HTMLVideoElement -> ImageBitmap -> Blob
isn't really much worse than doing:
HTMLVideoElement -> CanvasRenderContext2D -> Blob

-Jeff

On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad  wrote:
> Making ImageData the "hub" would imply more copies in memory in many cases.
> Because ImageData is mutable (not to mention the alpha multiplication issues
> which are also a factor), it cannot share its image buffer with the source
> it was created from, unlike ImageBitmap.  Immutability is a significant
> advantage of ImageBitmap, which allows for zero-copy code paths in many
> cases, which helps with both performance and memory consumption.
>
> On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen  wrote:
>>
>> If it's difficult to make ImageBitmap both an efficient drawing source and
>> a conversion intermediary, then we could add conversion functions to each
>> object that can be converted, e.g.:
>>
>> HTMLImageElement.toBlob()
>> HTMLImageElement.toImageData()
>> Blob.toImageData()
>> ImageData.toBlob()
>>
>> This seems inconsistent. For example to convert a Blob to an Image, you
>> should create a URL and set an Image's src to it; to convert a Blob to an
>> ImageBitmap, you should use createImageBitmap(blob); and then under this
>> proposal there's a third approach to convert a Blob to an ImageData, using
>> Blob.toImageData(). It also has a larger surface area, requiring changes to
>> several interfaces.
>>
>> So perhaps it would be better to make ImageData the "hub" of conversion,
>> more like the first proposal I made. If ImageBitmap is a GPU-hosted
>> premultiplied resource, then ImageData is an expressly not-on-GPU
>> not-premultiplied alternative. That would mean adding something like this to
>> ImageData:
>>
>> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
>>
>> partial interface ImageData {
>> static Promise create(ImageDataSource source)
>> Promise toBlob()
>> }
>>
>> ImageData can also be converted to HTMLImageElement via toBlob,
>> ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
>> still synchronous, but no longer needs to be done purely for conversion
>> reasons, so probably means you really want it to appear in the canvas and
>> therefore should remain synchronous).
>>
>> This covers all the conversion use cases I can think of without
>> complicating ImageBitmap's "without undue latency" requirement. There's no
>> more transferring going on either, which I think is now unnecessary since
>> you can get from HTMLImageElement to ImageData with one call. I think it's
>> more future-proof too since future types can be added to ImageDataSource
>> allowing new types to be converted without a new method being added.
>>
>> Does this approach sound better?
>>
>> Ashley
>>
>>
>> On 26 June 2015 at 16:37, Boris Zbarsky  wrote:
>>>
>>> On 6/26/15 4:07 AM, Ashley Gullen wrote:

 I don't think we should extend HTMLImageElement because it is not
 available in workers. Adding the conversion methods to ImageBitmap
 allows workers to perform conversions using Blob (compressed image data)
 in the place of HTMLImageElement.
>>>
>>>
>>> Maybe I wasn't clear.  I was suggesting that we have the methods on both
>>> HTMLImageElement and ImageBitmap (and possibly on any other things we feel
>>> should have the methods directly).
>>>
 I like the suggestion that "ImageBitmap be the hub of image conversion",
>>>
>>>
>>> I agree that it sounds appealing, but it means ImageBitmap now has to
>>> serve two masters: it has to be something that you can paint from quickly
>>> (premultiplied, probably lives on the GPU) _and_ it needs to be something
>>> you can transferToImageData efficiently (better to not live on the GPU for
>>> this).
>>>
>>> Maybe that's OK; it's just a bit of a warning flag from my point of view
>>> when a single object is meant to do multiple quite different thing

Re: Async Image -> ImageData conversion

[Justin Novosad]

Making ImageData the "hub" would imply more copies in memory in many cases.
Because ImageData is mutable (not to mention the alpha multiplication
issues which are also a factor), it cannot share its image buffer with the
source it was created from, unlike ImageBitmap.  Immutability is a
significant advantage of ImageBitmap, which allows for zero-copy code paths
in many cases, which helps with both performance and memory consumption.

On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen  wrote:

> If it's difficult to make ImageBitmap both an efficient drawing source and
> a conversion intermediary, then we could add conversion functions to each
> object that can be converted, e.g.:
>
> HTMLImageElement.toBlob()
> HTMLImageElement.toImageData()
> Blob.toImageData()
> ImageData.toBlob()
>
> This seems inconsistent. For example to convert a Blob to an Image, you
> should create a URL and set an Image's src to it; to convert a Blob to an
> ImageBitmap, you should use createImageBitmap(blob); and then under this
> proposal there's a third approach to convert a Blob to an ImageData, using
> Blob.toImageData(). It also has a larger surface area, requiring changes to
> several interfaces.
>
> So perhaps it would be better to make ImageData the "hub" of conversion,
> more like the first proposal I made. If ImageBitmap is a GPU-hosted
> premultiplied resource, then ImageData is an expressly not-on-GPU
> not-premultiplied alternative. That would mean adding something like this
> to ImageData:
>
> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
>
> partial interface ImageData {
> static Promise create(ImageDataSource source)
> Promise toBlob()
> }
>
> ImageData can also be converted to HTMLImageElement via toBlob,
> ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
> still synchronous, but no longer needs to be done purely for conversion
> reasons, so probably means you really want it to appear in the canvas and
> therefore should remain synchronous).
>
> This covers all the conversion use cases I can think of without
> complicating ImageBitmap's "without undue latency" requirement. There's no
> more transferring going on either, which I think is now unnecessary since
> you can get from HTMLImageElement to ImageData with one call. I think it's
> more future-proof too since future types can be added to ImageDataSource
> allowing new types to be converted without a new method being added.
>
> Does this approach sound better?
>
> Ashley
>
>
> On 26 June 2015 at 16:37, Boris Zbarsky  wrote:
>
>> On 6/26/15 4:07 AM, Ashley Gullen wrote:
>>
>>> I don't think we should extend HTMLImageElement because it is not
>>> available in workers. Adding the conversion methods to ImageBitmap
>>> allows workers to perform conversions using Blob (compressed image data)
>>> in the place of HTMLImageElement.
>>>
>>
>> Maybe I wasn't clear.  I was suggesting that we have the methods on both
>> HTMLImageElement and ImageBitmap (and possibly on any other things we feel
>> should have the methods directly).
>>
>>  I like the suggestion that "ImageBitmap be the hub of image conversion",
>>>
>>
>> I agree that it sounds appealing, but it means ImageBitmap now has to
>> serve two masters: it has to be something that you can paint from quickly
>> (premultiplied, probably lives on the GPU) _and_ it needs to be something
>> you can transferToImageData efficiently (better to not live on the GPU for
>> this).
>>
>> Maybe that's OK; it's just a bit of a warning flag from my point of view
>> when a single object is meant to do multiple quite different things; it
>> makes it harder to have it be good at all of them...
>>
>> -Boris
>>
>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

If it's difficult to make ImageBitmap both an efficient drawing source and
a conversion intermediary, then we could add conversion functions to each
object that can be converted, e.g.:

HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()

This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData, using
Blob.toImageData(). It also has a larger surface area, requiring changes to
several interfaces.

So perhaps it would be better to make ImageData the "hub" of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like this
to ImageData:

typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;

partial interface ImageData {
static Promise create(ImageDataSource source)
Promise toBlob()
}

ImageData can also be converted to HTMLImageElement via toBlob, ImageBitmap
via createImageBitmap, or a canvas via putImageData (which is still
synchronous, but no longer needs to be done purely for conversion reasons,
so probably means you really want it to appear in the canvas and therefore
should remain synchronous).

This covers all the conversion use cases I can think of without
complicating ImageBitmap's "without undue latency" requirement. There's no
more transferring going on either, which I think is now unnecessary since
you can get from HTMLImageElement to ImageData with one call. I think it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.

Does this approach sound better?

Ashley


On 26 June 2015 at 16:37, Boris Zbarsky  wrote:

> On 6/26/15 4:07 AM, Ashley Gullen wrote:
>
>> I don't think we should extend HTMLImageElement because it is not
>> available in workers. Adding the conversion methods to ImageBitmap
>> allows workers to perform conversions using Blob (compressed image data)
>> in the place of HTMLImageElement.
>>
>
> Maybe I wasn't clear.  I was suggesting that we have the methods on both
> HTMLImageElement and ImageBitmap (and possibly on any other things we feel
> should have the methods directly).
>
>  I like the suggestion that "ImageBitmap be the hub of image conversion",
>>
>
> I agree that it sounds appealing, but it means ImageBitmap now has to
> serve two masters: it has to be something that you can paint from quickly
> (premultiplied, probably lives on the GPU) _and_ it needs to be something
> you can transferToImageData efficiently (better to not live on the GPU for
> this).
>
> Maybe that's OK; it's just a bit of a warning flag from my point of view
> when a single object is meant to do multiple quite different things; it
> makes it harder to have it be good at all of them...
>
> -Boris
>

Re: Async Image -> ImageData conversion

[Boris Zbarsky]

On 6/26/15 4:07 AM, Ashley Gullen wrote:

I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image data)
in the place of HTMLImageElement.


Maybe I wasn't clear.  I was suggesting that we have the methods on both 
HTMLImageElement and ImageBitmap (and possibly on any other things we 
feel should have the methods directly).



I like the suggestion that "ImageBitmap be the hub of image conversion",


I agree that it sounds appealing, but it means ImageBitmap now has to 
serve two masters: it has to be something that you can paint from 
quickly (premultiplied, probably lives on the GPU) _and_ it needs to be 
something you can transferToImageData efficiently (better to not live on 
the GPU for this).


Maybe that's OK; it's just a bit of a warning flag from my point of view 
when a single object is meant to do multiple quite different things; it 
makes it harder to have it be good at all of them...


-Boris

Re: Async Image -> ImageData conversion

[Ashley Gullen]

Gosh, I keep putting the wrong terms in and it really confuses things. I
need to get better at writing clearly! I mentioned "transferToImageBitmap"
when I meant "transferToImageData". The parargraph from my second-last
email should read:


I like the suggestion that "ImageBitmap be the hub of image conversion",
which is why I think it makes sense to add these methods to ImageBitmap and
not create() methods on other objects. In particular transferToImageData()
seems like it ought to be an ImageBitmap method since it mutates
the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap)
seems unintuitive if it mutates its parameter. If transferToImageData()
belongs to ImageBitmap, I think toImageData() logically should belong to
ImageBitmap too. ImageBitmap.toBlob() is also consistent with
HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the
object which represents the data source.


I understand that ImageBitmaps were not intended to be mutated, but the
OffscreenCanvas proposal is already suggesting that there be a close()
method (to immediately release it), and that they be transferrable (which
mutates them on the thread they are transferred from). These appear to be
essential requirements for the OffscreenCanvas proposal to work
efficiently. If we are going to have this anyway, then it seems reasonable
to use the same mechanism for ImageBitmap.transferToImageData(). Also there
is still no direct access to the ImageBitmap pixel data, it's only mutable
in the sense it can be reset to empty.

The same thing could happen for ImageData.transferToImageBitmap(), where
the ImageData is neutered and it returns an ImageBitmap which can share the
same storage, but I guess this is less controversial since ImageData is
already mutable.




On 26 June 2015 at 14:10, Justin Novosad  wrote:

>
>
> On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen  wrote:
>
>> I've updated the draft with an ImageBitmap.transferToImageData() method:
>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
>> It's also used in Example 2 to demonstrate converting a Blob to an
>> ImageData without redundantly copying the pixel data.
>>
>> I don't think we should extend HTMLImageElement because it is not
>> available in workers. Adding the conversion methods to ImageBitmap allows
>> workers to perform conversions using Blob (compressed image data) in the
>> place of HTMLImageElement.
>>
>> I like the suggestion that "ImageBitmap be the hub of image conversion",
>> which is why I think it makes sense to add these methods to ImageBitmap and
>> not create() methods on other objects. In particular
>> transferToImageBitmap() seems like it ought to be an ImageBitmap method
>> since it mutates the ImageBitmap, whereas something like
>> ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its
>> parameter.
>>
>
> If I am not mistaken, ImageBitmaps were not intended to be mutated. So far
> I always assumed the intended behavior of transferToImageBitmap() was that
> it created a new ImageBitmap. The difference with createImageBitmap is that
> the transfer semantic guarantees zero copying, which implies that the
> source object has to be neutered. transferToImageBitmap should only be
> necessary for transferring from objects that have mutable pixels data
> (ImageData, canvas).
>
>
>> If transferToImageData() belongs to ImageBitmap, I think toImageData()
>> logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also
>> consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method
>> belongs to the object which represents the data source.
>>
>> Ashley
>>
>>
>> On 25 June 2015 at 22:10, Anne van Kesteren  wrote:
>>
>>> On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky  wrote:
>>> > The drawback of adding toBlob/toImageData to the various things
>>> ImageData
>>> > can be constructed from is that it's a bit more spec complexity, but I
>>> don't
>>> > see that as a showstopper, necessarily.
>>>
>>> We should probably stick to the pattern of either having factory
>>> methods or putting methods directly on various objects. Or maybe,
>>> change them all to be static methods on the various classes:
>>>
>>>   ImageBitmap.create(...)
>>>   ImageData.create(...)
>>>   ...
>>>
>>> I would personally prefer this last pattern for creating instances
>>> where we need to go through a promise.
>>>
>>>
>>> --
>>> https://annevankesteren.nl/
>>>
>>
>>
>

Re: Async Image -> ImageData conversion

[Justin Novosad]

On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen  wrote:

> I've updated the draft with an ImageBitmap.transferToImageData() method:
> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
> It's also used in Example 2 to demonstrate converting a Blob to an
> ImageData without redundantly copying the pixel data.
>
> I don't think we should extend HTMLImageElement because it is not
> available in workers. Adding the conversion methods to ImageBitmap allows
> workers to perform conversions using Blob (compressed image data) in the
> place of HTMLImageElement.
>
> I like the suggestion that "ImageBitmap be the hub of image conversion",
> which is why I think it makes sense to add these methods to ImageBitmap and
> not create() methods on other objects. In particular
> transferToImageBitmap() seems like it ought to be an ImageBitmap method
> since it mutates the ImageBitmap, whereas something like
> ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its
> parameter.
>

If I am not mistaken, ImageBitmaps were not intended to be mutated. So far
I always assumed the intended behavior of transferToImageBitmap() was that
it created a new ImageBitmap. The difference with createImageBitmap is that
the transfer semantic guarantees zero copying, which implies that the
source object has to be neutered. transferToImageBitmap should only be
necessary for transferring from objects that have mutable pixels data
(ImageData, canvas).


> If transferToImageData() belongs to ImageBitmap, I think toImageData()
> logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also
> consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method
> belongs to the object which represents the data source.
>
> Ashley
>
>
> On 25 June 2015 at 22:10, Anne van Kesteren  wrote:
>
>> On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky  wrote:
>> > The drawback of adding toBlob/toImageData to the various things
>> ImageData
>> > can be constructed from is that it's a bit more spec complexity, but I
>> don't
>> > see that as a showstopper, necessarily.
>>
>> We should probably stick to the pattern of either having factory
>> methods or putting methods directly on various objects. Or maybe,
>> change them all to be static methods on the various classes:
>>
>>   ImageBitmap.create(...)
>>   ImageData.create(...)
>>   ...
>>
>> I would personally prefer this last pattern for creating instances
>> where we need to go through a promise.
>>
>>
>> --
>> https://annevankesteren.nl/
>>
>
>

Re: Async Image -> ImageData conversion

[Justin Novosad]

On Thu, Jun 25, 2015 at 4:57 PM, Boris Zbarsky  wrote:

> On 6/24/15 1:28 PM, Ashley Gullen wrote:
>
>>
>>  Note for devices with discrete GPUs, it's possible that ImageBitmap
>> stores the decompressed image data in a GPU texture but does not have it
>> in system RAM.
>>
>
> Indeed.  This is another situation where going directly from an
> HTMLImageElement to an ImageData or Blob would make some sense, because it
> would allow the UA to avoid the GPU upload and readback, right?
>
> I'm not convinced the spec should worry about GPU uploads and readbacks.
I think the term "undue latency" is conveniently vague, which gives
implementers an interesting level of freedom to use smart heuristics and
make  implementation-specific or platform-specific or decisions. For
example, I would argue that on many platforms uploading an image to the GPU
probably does not constitute undue latency, but a readback probably does.
Under those assumptions, an implementation may decide that ImageBitmaps
should be lazily uploaded to the GPU the first time they're drawn. That way
the GPU can be short-circuited when appropriate. Also, the spec requires
that ImageBitmaps be *drawn* without undue latency, it does not require
that converting to a Blob or an ImageData be lightning fast.  Anyways, my
point is that implementers have enough wiggle room to do the right thing
even without direct methods on .


>
> -Boris
>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I'm also wondering that if we have ImageBitmap.transferToImageData(),
should we not also have ImageData.transferToImageBitmap()? Currently
createImageBitmap(imageData) appears to require making a copy.


On 26 June 2015 at 12:07, Ashley Gullen  wrote:

> I've updated the draft with an ImageBitmap.transferToImageData() method:
> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
> It's also used in Example 2 to demonstrate converting a Blob to an
> ImageData without redundantly copying the pixel data.
>
> I don't think we should extend HTMLImageElement because it is not
> available in workers. Adding the conversion methods to ImageBitmap allows
> workers to perform conversions using Blob (compressed image data) in the
> place of HTMLImageElement.
>
> I like the suggestion that "ImageBitmap be the hub of image conversion",
> which is why I think it makes sense to add these methods to ImageBitmap and
> not create() methods on other objects. In particular
> transferToImageBitmap() seems like it ought to be an ImageBitmap method
> since it mutates the ImageBitmap, whereas something like
> ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its
> parameter. If transferToImageData() belongs to ImageBitmap, I think
> toImageData() logically should belong to ImageBitmap too.
> ImageBitmap.toBlob() is also consistent with
> HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the
> object which represents the data source.
>
> Ashley
>
>
> On 25 June 2015 at 22:10, Anne van Kesteren  wrote:
>
>> On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky  wrote:
>> > The drawback of adding toBlob/toImageData to the various things
>> ImageData
>> > can be constructed from is that it's a bit more spec complexity, but I
>> don't
>> > see that as a showstopper, necessarily.
>>
>> We should probably stick to the pattern of either having factory
>> methods or putting methods directly on various objects. Or maybe,
>> change them all to be static methods on the various classes:
>>
>>   ImageBitmap.create(...)
>>   ImageData.create(...)
>>   ...
>>
>> I would personally prefer this last pattern for creating instances
>> where we need to go through a promise.
>>
>>
>> --
>> https://annevankesteren.nl/
>>
>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I've updated the draft with an ImageBitmap.transferToImageData() method:
https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
It's also used in Example 2 to demonstrate converting a Blob to an
ImageData without redundantly copying the pixel data.

I don't think we should extend HTMLImageElement because it is not available
in workers. Adding the conversion methods to ImageBitmap allows workers to
perform conversions using Blob (compressed image data) in the place of
HTMLImageElement.

I like the suggestion that "ImageBitmap be the hub of image conversion",
which is why I think it makes sense to add these methods to ImageBitmap and
not create() methods on other objects. In particular
transferToImageBitmap() seems like it ought to be an ImageBitmap method
since it mutates the ImageBitmap, whereas something like
ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its
parameter. If transferToImageData() belongs to ImageBitmap, I think
toImageData() logically should belong to ImageBitmap too.
ImageBitmap.toBlob() is also consistent with
HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the
object which represents the data source.

Ashley


On 25 June 2015 at 22:10, Anne van Kesteren  wrote:

> On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky  wrote:
> > The drawback of adding toBlob/toImageData to the various things ImageData
> > can be constructed from is that it's a bit more spec complexity, but I
> don't
> > see that as a showstopper, necessarily.
>
> We should probably stick to the pattern of either having factory
> methods or putting methods directly on various objects. Or maybe,
> change them all to be static methods on the various classes:
>
>   ImageBitmap.create(...)
>   ImageData.create(...)
>   ...
>
> I would personally prefer this last pattern for creating instances
> where we need to go through a promise.
>
>
> --
> https://annevankesteren.nl/
>

Re: Async Image -> ImageData conversion

[Anne van Kesteren]

On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky  wrote:
> The drawback of adding toBlob/toImageData to the various things ImageData
> can be constructed from is that it's a bit more spec complexity, but I don't
> see that as a showstopper, necessarily.

We should probably stick to the pattern of either having factory
methods or putting methods directly on various objects. Or maybe,
change them all to be static methods on the various classes:

  ImageBitmap.create(...)
  ImageData.create(...)
  ...

I would personally prefer this last pattern for creating instances
where we need to go through a promise.


-- 
https://annevankesteren.nl/

Re: Async Image -> ImageData conversion

[Boris Zbarsky]

On 6/24/15 1:28 PM, Ashley Gullen wrote:

I'm new to specs and WebIDL, my intent was to say those are new methods
on ImageBitmap. Is "partial interface ImageBitmap" the correct way to
say that?


Yes, it is.


The wording of "undue latency" in the ImageBitmap spec does not appear
to rule out ImageBitmap storing its encoded image data, and lazily
decoding it.


Decoding can easily take hundreds of milliseconds.  It definitely 
constitutes undue latency from our point of view.



- require ImageBitmap to decompress its contents, and make it available
through a read-only Uint8ClampedArray like ImageData has.


This has the premultiplied vs non-premultiplied problem Justin points out.


- provide a way to "neuter" the ImageBitmap when converting it to
ImageData, transferring its contents and avoiding any copy, which is
useful if the ImageBitmap is a temporary object only being created for
the purposes of getting the ImageData. I guess this would be similar to
transferrable objects with workers.


This would be a viable option.

Another option is to add toBlob/toImageData directly to HTMLImageElement 
and the other things we want it on, in addition to having them on 
ImageBitmap.  Then you wouldn't need to go through an ImageBitmap at all.



Note for devices with discrete GPUs, it's possible that ImageBitmap
stores the decompressed image data in a GPU texture but does not have it
in system RAM.


Indeed.  This is another situation where going directly from an 
HTMLImageElement to an ImageData or Blob would make some sense, because 
it would allow the UA to avoid the GPU upload and readback, right?


The drawback of adding toBlob/toImageData to the various things 
ImageData can be constructed from is that it's a bit more spec 
complexity, but I don't see that as a showstopper, necessarily.


-Boris

Re: Async Image -> ImageData conversion

[Justin Novosad]

On Thu, Jun 25, 2015 at 9:40 AM, Justin Novosad  wrote:

>
>
> On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen  wrote:
>
>> I see that OffscreenCanvas also specifies a close() method for
>> ImageBitmap. Perhaps browsers could use copy-on-write with the
>> toImageData() method? Then you can efficiently either transfer or copy
>> depending on if you close the ImageBitmap you transferred to an ImageData.
>>
>
>> Transfer:
>> createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close()
>> No copy made, contents transferred to the imageData
>>
>> createImageBitmap() -> imageBitmap.toImageData() -> access resulting
>> imageData.data
>> Copy made on first access
>>
>
> Copy on write is probably not a good idea because it would require
> injecting an observer pattern (or some kind of notification mechanism) into
> Uint8ClampedArray data assignment, which would realistically degrade
> performance of Image processing pixel loops.  Having a one step transfer
> API (transferToImageData) avoids adding overhead to what is otherwise just
> a simple memory access.
>

Something more: Blink, and probably other implementations as well, stores
decoded images in alpha pre-multiplied form.  ImageData is not
pre-multiplied.  For this reason sharing a buffer between ImageBitmap and
ImageData would be somewhat inconvenient.


>>
>> On 24 June 2015 at 21:54, Kenneth Russell  wrote:
>>
>>> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen 
>>> wrote:
>>> > Sorry for the confusion. Yes, the latest URL is:
>>> >
>>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
>>> > I'm new to specs and WebIDL, my intent was to say those are new
>>> methods on
>>> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
>>> that?
>>> > (I updated the URL to say that instead)
>>> >
>>> > The wording of "undue latency" in the ImageBitmap spec does not appear
>>> to
>>> > rule out ImageBitmap storing its encoded image data, and lazily
>>> decoding it.
>>> > However I'm not sure what the point of that would be, since it seems
>>> to be
>>> > how drawing HTMLImageElements to a canvas already works. I see a
>>> couple of
>>> > options:
>>> > - allow lazy decoding in ImageBitmap, and therefore converting to
>>> ImageData
>>> > is an explicit request to decompress
>>> > - require ImageBitmap to decompress its contents, and make it available
>>> > through a read-only Uint8ClampedArray like ImageData has. Therefore,
>>> > converting to ImageData indicates the intent to modify this content,
>>> > therefore a copy is warranted.
>>> > - provide a way to "neuter" the ImageBitmap when converting it to
>>> ImageData,
>>> > transferring its contents and avoiding any copy, which is useful if the
>>> > ImageBitmap is a temporary object only being created for the purposes
>>> of
>>> > getting the ImageData. I guess this would be similar to transferrable
>>> > objects with workers.
>>>
>>> Perhaps this could be done by specifying a "transferToImageData" method.
>>>
>>> This proposal makes ImageBitmap neuterable, and there's intent to
>>> implement it in order to allow rendering to canvas contexts from
>>> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .
>>>
>>> -Ken
>>>
>>>
>>> > - add some kind of "load" method on ImageBitmap. It may store its
>>> contents
>>> > in compressed form, but calling "load" causes it to be decompressed (or
>>> > loaded from somewhere else). The ImageBitmap is only guaranteed to be
>>> drawn
>>> > with "undue latency" after the "load" call. Therefore for the use case
>>> of
>>> > low-latency rendering "load" can always be called immediately after
>>> > creation, but for conversion purposes not calling "load" avoids
>>> duplicating
>>> > memory.
>>> >
>>> > Note for devices with discrete GPUs, it's possible that ImageBitmap
>>> stores
>>> > the decompressed image data in a GPU texture but does not have it in
>>> system
>>> > RAM. In this case making a copy for the ImageData is also warranted.
>>> >
>>> > I'm a web developer proposing this because it would be useful for my
>>> > purposes, I've no idea which of these would be favoured by
>>> implementors or
>>> > the spec process. I'm happy to get involved in spec work though so
>>> please
>>> > let me know if you have any feedback/suggestions on anything I'm doing
>>> here.
>>> >
>>> > Ashley
>>> >
>>> >
>>> >
>>> > On 24 June 2015 at 19:12, Boris Zbarsky  wrote:
>>> >>
>>> >> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>>> >>>
>>> >>> I've not done this before, so I've no idea if this is the
>>> right/useful
>>> >>> approach, but I drafted a spec for it here:
>>> >>>
>>> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>>> >>
>>> >>
>>> >> Ashley,
>>> >>
>>> >> We at Mozilla were just discussing this proposal, and we have a
>>> concern.
>>> >>
>>> >> With this proposal, going from an  to an ImageData requires
>>> >> conversion of an intermediate ImageBitmap.  Unfortunately, an
>>> ImageBitmap is
>>> >> specified to 

Re: Async Image -> ImageData conversion

[Justin Novosad]

On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen  wrote:

> I see that OffscreenCanvas also specifies a close() method for
> ImageBitmap. Perhaps browsers could use copy-on-write with the
> toImageData() method? Then you can efficiently either transfer or copy
> depending on if you close the ImageBitmap you transferred to an ImageData.
>

> Transfer:
> createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close()
> No copy made, contents transferred to the imageData
>
> createImageBitmap() -> imageBitmap.toImageData() -> access resulting
> imageData.data
> Copy made on first access
>

Copy on write is probably not a good idea because it would require
injecting an observer pattern (or some kind of notification mechanism) into
Uint8ClampedArray data assignment, which would realistically degrade
performance of Image processing pixel loops.  Having a one step transfer
API (transferToImageData) avoids adding overhead to what is otherwise just
a simple memory access.


>
> On 24 June 2015 at 21:54, Kenneth Russell  wrote:
>
>> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen  wrote:
>> > Sorry for the confusion. Yes, the latest URL is:
>> >
>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
>> > I'm new to specs and WebIDL, my intent was to say those are new methods
>> on
>> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
>> that?
>> > (I updated the URL to say that instead)
>> >
>> > The wording of "undue latency" in the ImageBitmap spec does not appear
>> to
>> > rule out ImageBitmap storing its encoded image data, and lazily
>> decoding it.
>> > However I'm not sure what the point of that would be, since it seems to
>> be
>> > how drawing HTMLImageElements to a canvas already works. I see a couple
>> of
>> > options:
>> > - allow lazy decoding in ImageBitmap, and therefore converting to
>> ImageData
>> > is an explicit request to decompress
>> > - require ImageBitmap to decompress its contents, and make it available
>> > through a read-only Uint8ClampedArray like ImageData has. Therefore,
>> > converting to ImageData indicates the intent to modify this content,
>> > therefore a copy is warranted.
>> > - provide a way to "neuter" the ImageBitmap when converting it to
>> ImageData,
>> > transferring its contents and avoiding any copy, which is useful if the
>> > ImageBitmap is a temporary object only being created for the purposes of
>> > getting the ImageData. I guess this would be similar to transferrable
>> > objects with workers.
>>
>> Perhaps this could be done by specifying a "transferToImageData" method.
>>
>> This proposal makes ImageBitmap neuterable, and there's intent to
>> implement it in order to allow rendering to canvas contexts from
>> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .
>>
>> -Ken
>>
>>
>> > - add some kind of "load" method on ImageBitmap. It may store its
>> contents
>> > in compressed form, but calling "load" causes it to be decompressed (or
>> > loaded from somewhere else). The ImageBitmap is only guaranteed to be
>> drawn
>> > with "undue latency" after the "load" call. Therefore for the use case
>> of
>> > low-latency rendering "load" can always be called immediately after
>> > creation, but for conversion purposes not calling "load" avoids
>> duplicating
>> > memory.
>> >
>> > Note for devices with discrete GPUs, it's possible that ImageBitmap
>> stores
>> > the decompressed image data in a GPU texture but does not have it in
>> system
>> > RAM. In this case making a copy for the ImageData is also warranted.
>> >
>> > I'm a web developer proposing this because it would be useful for my
>> > purposes, I've no idea which of these would be favoured by implementors
>> or
>> > the spec process. I'm happy to get involved in spec work though so
>> please
>> > let me know if you have any feedback/suggestions on anything I'm doing
>> here.
>> >
>> > Ashley
>> >
>> >
>> >
>> > On 24 June 2015 at 19:12, Boris Zbarsky  wrote:
>> >>
>> >> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>> >>>
>> >>> I've not done this before, so I've no idea if this is the right/useful
>> >>> approach, but I drafted a spec for it here:
>> >>>
>> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>> >>
>> >>
>> >> Ashley,
>> >>
>> >> We at Mozilla were just discussing this proposal, and we have a
>> concern.
>> >>
>> >> With this proposal, going from an  to an ImageData requires
>> >> conversion of an intermediate ImageBitmap.  Unfortunately, an
>> ImageBitmap is
>> >> specified to be paintable "without undue latency", which we interpret
>> to
>> >> mean it needs to have decoded image data, and the ImageData will end up
>> >> needing to copy said data in practice (because it needs an editable
>> copy).
>> >>
>> >> That creates two copies of the decoded image data in memory, which
>> seems
>> >> fairly undesirable on memory-constrained devices.
>> >>
>> >> -Boris
>> >>
>> >>
>> >
>>
>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I see that OffscreenCanvas also specifies a close() method for ImageBitmap.
Perhaps browsers could use copy-on-write with the toImageData() method?
Then you can efficiently either transfer or copy depending on if you close
the ImageBitmap you transferred to an ImageData.

Transfer:
createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close()
No copy made, contents transferred to the imageData

createImageBitmap() -> imageBitmap.toImageData() -> access resulting
imageData.data
Copy made on first access



On 24 June 2015 at 21:54, Kenneth Russell  wrote:

> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen  wrote:
> > Sorry for the confusion. Yes, the latest URL is:
> > https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
> > I'm new to specs and WebIDL, my intent was to say those are new methods
> on
> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
> that?
> > (I updated the URL to say that instead)
> >
> > The wording of "undue latency" in the ImageBitmap spec does not appear to
> > rule out ImageBitmap storing its encoded image data, and lazily decoding
> it.
> > However I'm not sure what the point of that would be, since it seems to
> be
> > how drawing HTMLImageElements to a canvas already works. I see a couple
> of
> > options:
> > - allow lazy decoding in ImageBitmap, and therefore converting to
> ImageData
> > is an explicit request to decompress
> > - require ImageBitmap to decompress its contents, and make it available
> > through a read-only Uint8ClampedArray like ImageData has. Therefore,
> > converting to ImageData indicates the intent to modify this content,
> > therefore a copy is warranted.
> > - provide a way to "neuter" the ImageBitmap when converting it to
> ImageData,
> > transferring its contents and avoiding any copy, which is useful if the
> > ImageBitmap is a temporary object only being created for the purposes of
> > getting the ImageData. I guess this would be similar to transferrable
> > objects with workers.
>
> Perhaps this could be done by specifying a "transferToImageData" method.
>
> This proposal makes ImageBitmap neuterable, and there's intent to
> implement it in order to allow rendering to canvas contexts from
> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .
>
> -Ken
>
>
> > - add some kind of "load" method on ImageBitmap. It may store its
> contents
> > in compressed form, but calling "load" causes it to be decompressed (or
> > loaded from somewhere else). The ImageBitmap is only guaranteed to be
> drawn
> > with "undue latency" after the "load" call. Therefore for the use case of
> > low-latency rendering "load" can always be called immediately after
> > creation, but for conversion purposes not calling "load" avoids
> duplicating
> > memory.
> >
> > Note for devices with discrete GPUs, it's possible that ImageBitmap
> stores
> > the decompressed image data in a GPU texture but does not have it in
> system
> > RAM. In this case making a copy for the ImageData is also warranted.
> >
> > I'm a web developer proposing this because it would be useful for my
> > purposes, I've no idea which of these would be favoured by implementors
> or
> > the spec process. I'm happy to get involved in spec work though so please
> > let me know if you have any feedback/suggestions on anything I'm doing
> here.
> >
> > Ashley
> >
> >
> >
> > On 24 June 2015 at 19:12, Boris Zbarsky  wrote:
> >>
> >> On 6/19/15 5:43 AM, Ashley Gullen wrote:
> >>>
> >>> I've not done this before, so I've no idea if this is the right/useful
> >>> approach, but I drafted a spec for it here:
> >>>
> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
> >>
> >>
> >> Ashley,
> >>
> >> We at Mozilla were just discussing this proposal, and we have a concern.
> >>
> >> With this proposal, going from an  to an ImageData requires
> >> conversion of an intermediate ImageBitmap.  Unfortunately, an
> ImageBitmap is
> >> specified to be paintable "without undue latency", which we interpret to
> >> mean it needs to have decoded image data, and the ImageData will end up
> >> needing to copy said data in practice (because it needs an editable
> copy).
> >>
> >> That creates two copies of the decoded image data in memory, which seems
> >> fairly undesirable on memory-constrained devices.
> >>
> >> -Boris
> >>
> >>
> >
>

Re: Async Image -> ImageData conversion

[Kenneth Russell]

On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen  wrote:
> Sorry for the confusion. Yes, the latest URL is:
> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
> I'm new to specs and WebIDL, my intent was to say those are new methods on
> ImageBitmap. Is "partial interface ImageBitmap" the correct way to say that?
> (I updated the URL to say that instead)
>
> The wording of "undue latency" in the ImageBitmap spec does not appear to
> rule out ImageBitmap storing its encoded image data, and lazily decoding it.
> However I'm not sure what the point of that would be, since it seems to be
> how drawing HTMLImageElements to a canvas already works. I see a couple of
> options:
> - allow lazy decoding in ImageBitmap, and therefore converting to ImageData
> is an explicit request to decompress
> - require ImageBitmap to decompress its contents, and make it available
> through a read-only Uint8ClampedArray like ImageData has. Therefore,
> converting to ImageData indicates the intent to modify this content,
> therefore a copy is warranted.
> - provide a way to "neuter" the ImageBitmap when converting it to ImageData,
> transferring its contents and avoiding any copy, which is useful if the
> ImageBitmap is a temporary object only being created for the purposes of
> getting the ImageData. I guess this would be similar to transferrable
> objects with workers.

Perhaps this could be done by specifying a "transferToImageData" method.

This proposal makes ImageBitmap neuterable, and there's intent to
implement it in order to allow rendering to canvas contexts from
worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas .

-Ken


> - add some kind of "load" method on ImageBitmap. It may store its contents
> in compressed form, but calling "load" causes it to be decompressed (or
> loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn
> with "undue latency" after the "load" call. Therefore for the use case of
> low-latency rendering "load" can always be called immediately after
> creation, but for conversion purposes not calling "load" avoids duplicating
> memory.
>
> Note for devices with discrete GPUs, it's possible that ImageBitmap stores
> the decompressed image data in a GPU texture but does not have it in system
> RAM. In this case making a copy for the ImageData is also warranted.
>
> I'm a web developer proposing this because it would be useful for my
> purposes, I've no idea which of these would be favoured by implementors or
> the spec process. I'm happy to get involved in spec work though so please
> let me know if you have any feedback/suggestions on anything I'm doing here.
>
> Ashley
>
>
>
> On 24 June 2015 at 19:12, Boris Zbarsky  wrote:
>>
>> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>>>
>>> I've not done this before, so I've no idea if this is the right/useful
>>> approach, but I drafted a spec for it here:
>>>
>>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>>
>>
>> Ashley,
>>
>> We at Mozilla were just discussing this proposal, and we have a concern.
>>
>> With this proposal, going from an  to an ImageData requires
>> conversion of an intermediate ImageBitmap.  Unfortunately, an ImageBitmap is
>> specified to be paintable "without undue latency", which we interpret to
>> mean it needs to have decoded image data, and the ImageData will end up
>> needing to copy said data in practice (because it needs an editable copy).
>>
>> That creates two copies of the decoded image data in memory, which seems
>> fairly undesirable on memory-constrained devices.
>>
>> -Boris
>>
>>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

Sorry for the confusion. Yes, the latest URL is:
https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html
I'm new to specs and WebIDL, my intent was to say those are new methods on
ImageBitmap. Is "partial interface ImageBitmap" the correct way to say
that? (I updated the URL to say that instead)

The wording of "undue latency" in the ImageBitmap spec does not appear to
rule out ImageBitmap storing its encoded image data, and lazily decoding
it. However I'm not sure what the point of that would be, since it seems to
be how drawing HTMLImageElements to a canvas already works. I see a couple
of options:
- allow lazy decoding in ImageBitmap, and therefore converting to ImageData
is an explicit request to decompress
- require ImageBitmap to decompress its contents, and make it available
through a read-only Uint8ClampedArray like ImageData has. Therefore,
converting to ImageData indicates the intent to modify this content,
therefore a copy is warranted.
- provide a way to "neuter" the ImageBitmap when converting it to
ImageData, transferring its contents and avoiding any copy, which is useful
if the ImageBitmap is a temporary object only being created for the
purposes of getting the ImageData. I guess this would be similar to
transferrable objects with workers.
- add some kind of "load" method on ImageBitmap. It may store its contents
in compressed form, but calling "load" causes it to be decompressed (or
loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn
with "undue latency" after the "load" call. Therefore for the use case of
low-latency rendering "load" can always be called immediately after
creation, but for conversion purposes not calling "load" avoids duplicating
memory.

Note for devices with discrete GPUs, it's possible that ImageBitmap stores
the decompressed image data in a GPU texture but does not have it in system
RAM. In this case making a copy for the ImageData is also warranted.

I'm a web developer proposing this because it would be useful for my
purposes, I've no idea which of these would be favoured by implementors or
the spec process. I'm happy to get involved in spec work though so please
let me know if you have any feedback/suggestions on anything I'm doing here.

Ashley



On 24 June 2015 at 19:12, Boris Zbarsky  wrote:

> On 6/19/15 5:43 AM, Ashley Gullen wrote:
>
>> I've not done this before, so I've no idea if this is the right/useful
>> approach, but I drafted a spec for it here:
>>
>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>>
>
> Ashley,
>
> We at Mozilla were just discussing this proposal, and we have a concern.
>
> With this proposal, going from an  to an ImageData requires
> conversion of an intermediate ImageBitmap.  Unfortunately, an ImageBitmap
> is specified to be paintable "without undue latency", which we interpret to
> mean it needs to have decoded image data, and the ImageData will end up
> needing to copy said data in practice (because it needs an editable copy).
>
> That creates two copies of the decoded image data in memory, which seems
> fairly undesirable on memory-constrained devices.
>
> -Boris
>
>
>

Re: Async Image -> ImageData conversion

[Boris Zbarsky]

On 6/19/15 5:43 AM, Ashley Gullen wrote:

I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:

https://www.scirra.com/labs/specs/imagedata-blob-extensions.html


Ashley,

We at Mozilla were just discussing this proposal, and we have a concern.

With this proposal, going from an  to an ImageData requires 
conversion of an intermediate ImageBitmap.  Unfortunately, an 
ImageBitmap is specified to be paintable "without undue latency", which 
we interpret to mean it needs to have decoded image data, and the 
ImageData will end up needing to copy said data in practice (because it 
needs an editable copy).


That creates two copies of the decoded image data in memory, which seems 
fairly undesirable on memory-constrained devices.


-Boris

Re: Async Image -> ImageData conversion

[Boris Zbarsky]

On 6/19/15 5:43 AM, Ashley Gullen wrote:

I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:

https://www.scirra.com/labs/specs/imagedata-blob-extensions.html

Let me know if you have any feedback on this.


Ashley,

I assume the relevant link is now 
, 
right?


I'm trying to understand where the methods are being added.  The IDL 
says "partial interface ImageBitmapConversion" but there is no current 
ImageBitmapConversion interface I'm aware of.  Is this supposed to be 
"partial interface ImageBitmap"?


-Boris

Re: Subject=Re: Async Image -> ImageData conversion

[Ashley Gullen]

Good point - it makes sense to have ImageBitmap as the "hub" of all
conversion.

I've drafted a new spec which instead proposes ImageBitmap.toBlob() and
ImageBitmap.toImageData():
https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html

This should cover all conversion cases asynchronously.



On 23 June 2015 at 20:34, Justin Novosad  wrote:

> Based on feedback received from web developers, new APIs that move image
> data around should also strive to eliminate intermediate copies of the
> image data to avoid memory bloat. I think your proposal achieves that, but
> I think memory footprint reduction could be a stated objective of the
> proposal. For example, the current solution of using a canvas forces the
> creation of an intermediate copy (the canvas).
>
> Also, to avoid the multiplication of APIs for moving image data between
> various representations, I would like to suggest an alternative: using the
> ImageBitmap interface as a hub. The creation of ImageBitmaps is already
> asynchronous (createImageBitmap returns a promise), and it has overloads
> for acquiring images from img, video, canvas, url, blob, imagedata. All
> that is missing are a few methods for getting data directly out of an
> ImageBitmap. So I think adding toBlob and toImageData (both async) to
> ImageBitmap is a more succinct proposal that would address your use cases,
> and many additional ones at the same time.
>

Subject=Re: Async Image -> ImageData conversion

[Justin Novosad]

Based on feedback received from web developers, new APIs that move image
data around should also strive to eliminate intermediate copies of the
image data to avoid memory bloat. I think your proposal achieves that, but
I think memory footprint reduction could be a stated objective of the
proposal. For example, the current solution of using a canvas forces the
creation of an intermediate copy (the canvas).

Also, to avoid the multiplication of APIs for moving image data between
various representations, I would like to suggest an alternative: using the
ImageBitmap interface as a hub. The creation of ImageBitmaps is already
asynchronous (createImageBitmap returns a promise), and it has overloads
for acquiring images from img, video, canvas, url, blob, imagedata. All
that is missing are a few methods for getting data directly out of an
ImageBitmap. So I think adding toBlob and toImageData (both async) to
ImageBitmap is a more succinct proposal that would address your use cases,
and many additional ones at the same time.

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I was advised to mark this as an "Editor's draft", and have done so. I also
added a new proposed ImageData.fromImage method, with rationale included in
the document. It's at the same URL:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html


On 19 June 2015 at 13:43, Ashley Gullen  wrote:

> I've not done this before, so I've no idea if this is the right/useful
> approach, but I drafted a spec for it here:
>
> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
>
> Let me know if you have any feedback on this.
>
>
>
> On 18 June 2015 at 22:27, Travis Leithead 
> wrote:
>
>>  Cool. Want to write it up as a small extension spec? We have a new
>> Incubator Community Group coming up soon that would be an ideal place to
>> drop a small spec into and let implementers experiment with it and get
>> broader community feedback on the idea.
>>
>>
>>
>> *From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
>> Gullen
>> *Sent:* Wednesday, June 17, 2015 2:06 PM
>> *To:* Travis Leithead
>> *Cc:* public-webapps@w3.org
>> *Subject:* Re: Async Image -> ImageData conversion
>>
>>
>>
>> That seems like a good start. I suppose there should be a
>> putImageDataAsync counterpart too? Then we can do:
>>
>>
>>
>> Blob -> Image via: load blob URL
>>
>> Blob -> ImageData via: load blob URL -> Canvas drawImage
>> -> getImageDataAsync
>>
>> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>>
>> Image -> ImageData via: Canvas drawImage -> getImageDataAsync
>>
>> ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
>>
>> ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load
>> blob URL
>>
>>
>>
>> I think the potential problems with this are:
>>
>> - putImageDataAsync is the logical counterpart to getImageDataAsync, but
>> what happens if you make other synchronous draw calls while
>> putImageDataAsync is processing? What actually ends up in the canvas and
>> how is this defined?
>>
>> - some of the conversion steps seem pretty long winded, in particular
>> Blob -> ImageData and ImageData -> Image. If implementors think they can
>> optimise these to be as efficient as direct conversion that's fine, but is
>> that actually doable with such a roundabout API?
>>
>>
>>
>> If ImageData has:
>>
>> Promise toBlob();
>>
>> Promise fromBlob(blob);
>>
>>
>>
>> then we can use the following conversions without new canvas functions:
>>
>>
>>
>> Blob -> Image via: load blob URL
>>
>> Blob -> ImageData via: ImageData.fromBlob
>>
>> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>>
>> Image -> ImageData via: Canvas drawImage -> Canvas toBlob ->
>> ImageData.fromBlob
>>
>> ImageData -> Blob via: ImageData.toBlob
>>
>> ImageData -> Image via: ImageData.toBlob -> load blob URL
>>
>>
>>
>> That looks like a much more reasonable set of conversions to me, and is
>> all async.
>>
>>
>>
>> I was thinking about toImage and fromImage, but it seems to be a bigger
>> step to add a new way to make images, and there's the question of what the
>> src for the image returned by toImage should be.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 17 June 2015 at 18:59, Travis Leithead 
>> wrote:
>>
>>  I think solving at least the first-order problem of extracting data
>> from the Canvas async is do-able.
>>
>>
>>
>> Something like:
>>
>>
>>
>> Promise getImageDataAsync(x,y,w,h);
>>
>>
>>
>> seems sensible to add J
>>
>>
>>
>> *From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
>> Gullen
>> *Sent:* Wednesday, June 17, 2015 10:00 AM
>> *To:* public-webapps@w3.org
>> *Subject:* Async Image -> ImageData conversion
>>
>>
>>
>> I was wondering if there is anything on the standards track to
>> asynchronously get an ImageData object from an Image?
>>
>>
>>
>> We have a web app dealing with large sprite sheets (up to 2048x2048), and
>> at some point we need to get an ImageData for this. Currently the only way
>> I know of doing this is via a canvas 2d context like so:
>>
>>
>>
>> function ImageToImageData(img)
>>
>

Re: Async Image -> ImageData conversion

[Ashley Gullen]

I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:

https://www.scirra.com/labs/specs/imagedata-blob-extensions.html

Let me know if you have any feedback on this.



On 18 June 2015 at 22:27, Travis Leithead 
wrote:

>  Cool. Want to write it up as a small extension spec? We have a new
> Incubator Community Group coming up soon that would be an ideal place to
> drop a small spec into and let implementers experiment with it and get
> broader community feedback on the idea.
>
>
>
> *From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
> Gullen
> *Sent:* Wednesday, June 17, 2015 2:06 PM
> *To:* Travis Leithead
> *Cc:* public-webapps@w3.org
> *Subject:* Re: Async Image -> ImageData conversion
>
>
>
> That seems like a good start. I suppose there should be a
> putImageDataAsync counterpart too? Then we can do:
>
>
>
> Blob -> Image via: load blob URL
>
> Blob -> ImageData via: load blob URL -> Canvas drawImage
> -> getImageDataAsync
>
> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>
> Image -> ImageData via: Canvas drawImage -> getImageDataAsync
>
> ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
>
> ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load
> blob URL
>
>
>
> I think the potential problems with this are:
>
> - putImageDataAsync is the logical counterpart to getImageDataAsync, but
> what happens if you make other synchronous draw calls while
> putImageDataAsync is processing? What actually ends up in the canvas and
> how is this defined?
>
> - some of the conversion steps seem pretty long winded, in particular Blob
> -> ImageData and ImageData -> Image. If implementors think they can
> optimise these to be as efficient as direct conversion that's fine, but is
> that actually doable with such a roundabout API?
>
>
>
> If ImageData has:
>
> Promise toBlob();
>
> Promise fromBlob(blob);
>
>
>
> then we can use the following conversions without new canvas functions:
>
>
>
> Blob -> Image via: load blob URL
>
> Blob -> ImageData via: ImageData.fromBlob
>
> Image -> Blob via: Canvas drawImage -> Canvas toBlob
>
> Image -> ImageData via: Canvas drawImage -> Canvas toBlob ->
> ImageData.fromBlob
>
> ImageData -> Blob via: ImageData.toBlob
>
> ImageData -> Image via: ImageData.toBlob -> load blob URL
>
>
>
> That looks like a much more reasonable set of conversions to me, and is
> all async.
>
>
>
> I was thinking about toImage and fromImage, but it seems to be a bigger
> step to add a new way to make images, and there's the question of what the
> src for the image returned by toImage should be.
>
>
>
>
>
>
>
>
>
> On 17 June 2015 at 18:59, Travis Leithead 
> wrote:
>
>  I think solving at least the first-order problem of extracting data from
> the Canvas async is do-able.
>
>
>
> Something like:
>
>
>
> Promise getImageDataAsync(x,y,w,h);
>
>
>
> seems sensible to add J
>
>
>
> *From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
> Gullen
> *Sent:* Wednesday, June 17, 2015 10:00 AM
> *To:* public-webapps@w3.org
> *Subject:* Async Image -> ImageData conversion
>
>
>
> I was wondering if there is anything on the standards track to
> asynchronously get an ImageData object from an Image?
>
>
>
> We have a web app dealing with large sprite sheets (up to 2048x2048), and
> at some point we need to get an ImageData for this. Currently the only way
> I know of doing this is via a canvas 2d context like so:
>
>
>
> function ImageToImageData(img)
>
> {
>
> let w = img.width;
>
> let h = img.height;
>
>
>
> let canvas = document.createElement("canvas");
>
> canvas.width = w;
>
> canvas.height = h;
>
> let ctx = canvas.getContext("2d");
>
> ctx.drawImage(img, 0, 0);
>
> return ctx.getImageData(0, 0, w, h);
>
> };
>
>
>
> This whole function is synchronous. With large images, in Chrome the
> getImageData call can jank for well over 100ms on a high-end desktop
> machine. Maybe this can be faster but I really doubt this whole function is
> likely to be done in <16ms on low-end devices, so I believe there ought to
> be an async alternative. As far as I am aware there is none.
>
>
>
> Ideally we could have an async ImageData.fromImage(img) method or similar.
> It

RE: Async Image -> ImageData conversion

[Travis Leithead]

Cool. Want to write it up as a small extension spec? We have a new Incubator 
Community Group coming up soon that would be an ideal place to drop a small 
spec into and let implementers experiment with it and get broader community 
feedback on the idea.

From: a...@scirra.com [mailto:a...@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 2:06 PM
To: Travis Leithead
Cc: public-webapps@w3.org
Subject: Re: Async Image -> ImageData conversion

That seems like a good start. I suppose there should be a putImageDataAsync 
counterpart too? Then we can do:

Blob -> Image via: load blob URL
Blob -> ImageData via: load blob URL -> Canvas drawImage -> getImageDataAsync
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> getImageDataAsync
ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load blob 
URL

I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but what 
happens if you make other synchronous draw calls while putImageDataAsync is 
processing? What actually ends up in the canvas and how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob -> 
ImageData and ImageData -> Image. If implementors think they can optimise these 
to be as efficient as direct conversion that's fine, but is that actually 
doable with such a roundabout API?

If ImageData has:
Promise toBlob();
Promise fromBlob(blob);

then we can use the following conversions without new canvas functions:

Blob -> Image via: load blob URL
Blob -> ImageData via: ImageData.fromBlob
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> Canvas toBlob -> ImageData.fromBlob
ImageData -> Blob via: ImageData.toBlob
ImageData -> Image via: ImageData.toBlob -> load blob URL

That looks like a much more reasonable set of conversions to me, and is all 
async.

I was thinking about toImage and fromImage, but it seems to be a bigger step to 
add a new way to make images, and there's the question of what the src for the 
image returned by toImage should be.




On 17 June 2015 at 18:59, Travis Leithead 
mailto:travis.leith...@microsoft.com>> wrote:
I think solving at least the first-order problem of extracting data from the 
Canvas async is do-able.

Something like:

Promise getImageDataAsync(x,y,w,h);

seems sensible to add ☺

From: a...@scirra.com<mailto:a...@scirra.com> 
[mailto:a...@scirra.com<mailto:a...@scirra.com>] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 10:00 AM
To: public-webapps@w3.org<mailto:public-webapps@w3.org>
Subject: Async Image -> ImageData conversion

I was wondering if there is anything on the standards track to asynchronously 
get an ImageData object from an Image?

We have a web app dealing with large sprite sheets (up to 2048x2048), and at 
some point we need to get an ImageData for this. Currently the only way I know 
of doing this is via a canvas 2d context like so:

function ImageToImageData(img)
{
let w = img.width;
let h = img.height;

let canvas = document.createElement("canvas");
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext("2d");
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};

This whole function is synchronous. With large images, in Chrome the 
getImageData call can jank for well over 100ms on a high-end desktop machine. 
Maybe this can be faster but I really doubt this whole function is likely to be 
done in <16ms on low-end devices, so I believe there ought to be an async 
alternative. As far as I am aware there is none.

Ideally we could have an async ImageData.fromImage(img) method or similar. It 
would also be useful if we could decode directly from Blob to ImageData without 
having to go via an Image as well (perhaps ImageData.fromBlob(blob) ?).

More broadly, it would be good to have a comprehensive set of async conversion 
functions between Blob, Image, and ImageData. Blob -> Image can already be done 
by setting the image src to a blob URL, but basically everything else requires 
going synchronously through a 2D canvas, which janks apps which have to process 
lots of large images. Running a canvas in a worker could possibly help, but 
that would require both Blob and Image to be transferable to be useful.

Ashley Gullen
Scirra.com

Re: Async Image -> ImageData conversion

[Ashley Gullen]

That seems like a good start. I suppose there should be a putImageDataAsync
counterpart too? Then we can do:

Blob -> Image via: load blob URL
Blob -> ImageData via: load blob URL -> Canvas drawImage
-> getImageDataAsync
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> getImageDataAsync
ImageData -> Blob via: Canvas putImageDataAsync -> Canvas toBlob
ImageData -> Image via: Canvas putImageDataAsync -> Canvas toBlob -> load
blob URL

I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob
-> ImageData and ImageData -> Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?

If ImageData has:
Promise toBlob();
Promise fromBlob(blob);

then we can use the following conversions without new canvas functions:

Blob -> Image via: load blob URL
Blob -> ImageData via: ImageData.fromBlob
Image -> Blob via: Canvas drawImage -> Canvas toBlob
Image -> ImageData via: Canvas drawImage -> Canvas toBlob ->
ImageData.fromBlob
ImageData -> Blob via: ImageData.toBlob
ImageData -> Image via: ImageData.toBlob -> load blob URL

That looks like a much more reasonable set of conversions to me, and is all
async.

I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.




On 17 June 2015 at 18:59, Travis Leithead 
wrote:

>  I think solving at least the first-order problem of extracting data from
> the Canvas async is do-able.
>
>
>
> Something like:
>
>
>
> Promise getImageDataAsync(x,y,w,h);
>
>
>
> seems sensible to add J
>
>
>
> *From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
> Gullen
> *Sent:* Wednesday, June 17, 2015 10:00 AM
> *To:* public-webapps@w3.org
> *Subject:* Async Image -> ImageData conversion
>
>
>
> I was wondering if there is anything on the standards track to
> asynchronously get an ImageData object from an Image?
>
>
>
> We have a web app dealing with large sprite sheets (up to 2048x2048), and
> at some point we need to get an ImageData for this. Currently the only way
> I know of doing this is via a canvas 2d context like so:
>
>
>
> function ImageToImageData(img)
>
> {
>
> let w = img.width;
>
> let h = img.height;
>
>
>
> let canvas = document.createElement("canvas");
>
> canvas.width = w;
>
> canvas.height = h;
>
> let ctx = canvas.getContext("2d");
>
> ctx.drawImage(img, 0, 0);
>
> return ctx.getImageData(0, 0, w, h);
>
> };
>
>
>
> This whole function is synchronous. With large images, in Chrome the
> getImageData call can jank for well over 100ms on a high-end desktop
> machine. Maybe this can be faster but I really doubt this whole function is
> likely to be done in <16ms on low-end devices, so I believe there ought to
> be an async alternative. As far as I am aware there is none.
>
>
>
> Ideally we could have an async ImageData.fromImage(img) method or similar.
> It would also be useful if we could decode directly from Blob to ImageData
> without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
> ?).
>
>
>
> More broadly, it would be good to have a comprehensive set of async
> conversion functions between Blob, Image, and ImageData. Blob -> Image can
> already be done by setting the image src to a blob URL, but basically
> everything else requires going synchronously through a 2D canvas, which
> janks apps which have to process lots of large images. Running a canvas in
> a worker could possibly help, but that would require both Blob and Image to
> be transferable to be useful.
>
>
>
> Ashley Gullen
>
> Scirra.com
>
>
>

RE: Async Image -> ImageData conversion

[Travis Leithead]

I think solving at least the first-order problem of extracting data from the 
Canvas async is do-able.

Something like:

Promise getImageDataAsync(x,y,w,h);

seems sensible to add ☺

From: a...@scirra.com [mailto:a...@scirra.com] On Behalf Of Ashley Gullen
Sent: Wednesday, June 17, 2015 10:00 AM
To: public-webapps@w3.org
Subject: Async Image -> ImageData conversion

I was wondering if there is anything on the standards track to asynchronously 
get an ImageData object from an Image?

We have a web app dealing with large sprite sheets (up to 2048x2048), and at 
some point we need to get an ImageData for this. Currently the only way I know 
of doing this is via a canvas 2d context like so:

function ImageToImageData(img)
{
let w = img.width;
let h = img.height;

let canvas = document.createElement("canvas");
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext("2d");
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};

This whole function is synchronous. With large images, in Chrome the 
getImageData call can jank for well over 100ms on a high-end desktop machine. 
Maybe this can be faster but I really doubt this whole function is likely to be 
done in <16ms on low-end devices, so I believe there ought to be an async 
alternative. As far as I am aware there is none.

Ideally we could have an async ImageData.fromImage(img) method or similar. It 
would also be useful if we could decode directly from Blob to ImageData without 
having to go via an Image as well (perhaps ImageData.fromBlob(blob) ?).

More broadly, it would be good to have a comprehensive set of async conversion 
functions between Blob, Image, and ImageData. Blob -> Image can already be done 
by setting the image src to a blob URL, but basically everything else requires 
going synchronously through a 2D canvas, which janks apps which have to process 
lots of large images. Running a canvas in a worker could possibly help, but 
that would require both Blob and Image to be transferable to be useful.

Ashley Gullen
Scirra.com

Async Image -> ImageData conversion

[Ashley Gullen]

I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?

We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:

function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
 let canvas = document.createElement("canvas");
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext("2d");
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};

This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in <16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.

Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).

More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob -> Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.

Ashley Gullen
Scirra.com