Re: Async Image -> ImageData conversion
Making ImageData the "hub" would imply more copies in memory in many cases.
Because ImageData is mutable (not to mention the alpha multiplication
issues which are also a factor), it cannot share its image buffer with the
source it was created from, unlike ImageBitmap. Immutability is a
significant advantage of ImageBitmap, which allows for zero-copy code paths
in many cases, which helps with both performance and memory consumption.
On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen wrote:
> If it's difficult to make ImageBitmap both an efficient drawing source and
> a conversion intermediary, then we could add conversion functions to each
> object that can be converted, e.g.:
>
> HTMLImageElement.toBlob()
> HTMLImageElement.toImageData()
> Blob.toImageData()
> ImageData.toBlob()
>
> This seems inconsistent. For example to convert a Blob to an Image, you
> should create a URL and set an Image's src to it; to convert a Blob to an
> ImageBitmap, you should use createImageBitmap(blob); and then under this
> proposal there's a third approach to convert a Blob to an ImageData, using
> Blob.toImageData(). It also has a larger surface area, requiring changes to
> several interfaces.
>
> So perhaps it would be better to make ImageData the "hub" of conversion,
> more like the first proposal I made. If ImageBitmap is a GPU-hosted
> premultiplied resource, then ImageData is an expressly not-on-GPU
> not-premultiplied alternative. That would mean adding something like this
> to ImageData:
>
> typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
>
> partial interface ImageData {
> static Promise create(ImageDataSource source)
> Promise toBlob()
> }
>
> ImageData can also be converted to HTMLImageElement via toBlob,
> ImageBitmap via createImageBitmap, or a canvas via putImageData (which is
> still synchronous, but no longer needs to be done purely for conversion
> reasons, so probably means you really want it to appear in the canvas and
> therefore should remain synchronous).
>
> This covers all the conversion use cases I can think of without
> complicating ImageBitmap's "without undue latency" requirement. There's no
> more transferring going on either, which I think is now unnecessary since
> you can get from HTMLImageElement to ImageData with one call. I think it's
> more future-proof too since future types can be added to ImageDataSource
> allowing new types to be converted without a new method being added.
>
> Does this approach sound better?
>
> Ashley
>
>
> On 26 June 2015 at 16:37, Boris Zbarsky wrote:
>
>> On 6/26/15 4:07 AM, Ashley Gullen wrote:
>>
>>> I don't think we should extend HTMLImageElement because it is not
>>> available in workers. Adding the conversion methods to ImageBitmap
>>> allows workers to perform conversions using Blob (compressed image data)
>>> in the place of HTMLImageElement.
>>>
>>
>> Maybe I wasn't clear. I was suggesting that we have the methods on both
>> HTMLImageElement and ImageBitmap (and possibly on any other things we feel
>> should have the methods directly).
>>
>> I like the suggestion that "ImageBitmap be the hub of image conversion",
>>>
>>
>> I agree that it sounds appealing, but it means ImageBitmap now has to
>> serve two masters: it has to be something that you can paint from quickly
>> (premultiplied, probably lives on the GPU) _and_ it needs to be something
>> you can transferToImageData efficiently (better to not live on the GPU for
>> this).
>>
>> Maybe that's OK; it's just a bit of a warning flag from my point of view
>> when a single object is meant to do multiple quite different things; it
>> makes it harder to have it be good at all of them...
>>
>> -Boris
>>
>
>
Re: Async Image -> ImageData conversion
On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen wrote: > I've updated the draft with an ImageBitmap.transferToImageData() method: > https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html > It's also used in Example 2 to demonstrate converting a Blob to an > ImageData without redundantly copying the pixel data. > > I don't think we should extend HTMLImageElement because it is not > available in workers. Adding the conversion methods to ImageBitmap allows > workers to perform conversions using Blob (compressed image data) in the > place of HTMLImageElement. > > I like the suggestion that "ImageBitmap be the hub of image conversion", > which is why I think it makes sense to add these methods to ImageBitmap and > not create() methods on other objects. In particular > transferToImageBitmap() seems like it ought to be an ImageBitmap method > since it mutates the ImageBitmap, whereas something like > ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its > parameter. > If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). > If transferToImageData() belongs to ImageBitmap, I think toImageData() > logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also > consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method > belongs to the object which represents the data source. > > Ashley > > > On 25 June 2015 at 22:10, Anne van Kesteren wrote: > >> On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky wrote: >> > The drawback of adding toBlob/toImageData to the various things >> ImageData >> > can be constructed from is that it's a bit more spec complexity, but I >> don't >> > see that as a showstopper, necessarily. >> >> We should probably stick to the pattern of either having factory >> methods or putting methods directly on various objects. Or maybe, >> change them all to be static methods on the various classes: >> >> ImageBitmap.create(...) >> ImageData.create(...) >> ... >> >> I would personally prefer this last pattern for creating instances >> where we need to go through a promise. >> >> >> -- >> https://annevankesteren.nl/ >> > >
Re: Async Image -> ImageData conversion
On Thu, Jun 25, 2015 at 4:57 PM, Boris Zbarsky wrote: > On 6/24/15 1:28 PM, Ashley Gullen wrote: > >> >> Note for devices with discrete GPUs, it's possible that ImageBitmap >> stores the decompressed image data in a GPU texture but does not have it >> in system RAM. >> > > Indeed. This is another situation where going directly from an > HTMLImageElement to an ImageData or Blob would make some sense, because it > would allow the UA to avoid the GPU upload and readback, right? > > I'm not convinced the spec should worry about GPU uploads and readbacks. I think the term "undue latency" is conveniently vague, which gives implementers an interesting level of freedom to use smart heuristics and make implementation-specific or platform-specific or decisions. For example, I would argue that on many platforms uploading an image to the GPU probably does not constitute undue latency, but a readback probably does. Under those assumptions, an implementation may decide that ImageBitmaps should be lazily uploaded to the GPU the first time they're drawn. That way the GPU can be short-circuited when appropriate. Also, the spec requires that ImageBitmaps be *drawn* without undue latency, it does not require that converting to a Blob or an ImageData be lightning fast. Anyways, my point is that implementers have enough wiggle room to do the right thing even without direct methods on . > > -Boris > >
Re: Async Image -> ImageData conversion
On Thu, Jun 25, 2015 at 9:40 AM, Justin Novosad wrote: > > > On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen wrote: > >> I see that OffscreenCanvas also specifies a close() method for >> ImageBitmap. Perhaps browsers could use copy-on-write with the >> toImageData() method? Then you can efficiently either transfer or copy >> depending on if you close the ImageBitmap you transferred to an ImageData. >> > >> Transfer: >> createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close() >> No copy made, contents transferred to the imageData >> >> createImageBitmap() -> imageBitmap.toImageData() -> access resulting >> imageData.data >> Copy made on first access >> > > Copy on write is probably not a good idea because it would require > injecting an observer pattern (or some kind of notification mechanism) into > Uint8ClampedArray data assignment, which would realistically degrade > performance of Image processing pixel loops. Having a one step transfer > API (transferToImageData) avoids adding overhead to what is otherwise just > a simple memory access. > Something more: Blink, and probably other implementations as well, stores decoded images in alpha pre-multiplied form. ImageData is not pre-multiplied. For this reason sharing a buffer between ImageBitmap and ImageData would be somewhat inconvenient. >> >> On 24 June 2015 at 21:54, Kenneth Russell wrote: >> >>> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen >>> wrote: >>> > Sorry for the confusion. Yes, the latest URL is: >>> > >>> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html >>> > I'm new to specs and WebIDL, my intent was to say those are new >>> methods on >>> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say >>> that? >>> > (I updated the URL to say that instead) >>> > >>> > The wording of "undue latency" in the ImageBitmap spec does not appear >>> to >>> > rule out ImageBitmap storing its encoded image data, and lazily >>> decoding it. >>> > However I'm not sure what the point of that would be, since it seems >>> to be >>> > how drawing HTMLImageElements to a canvas already works. I see a >>> couple of >>> > options: >>> > - allow lazy decoding in ImageBitmap, and therefore converting to >>> ImageData >>> > is an explicit request to decompress >>> > - require ImageBitmap to decompress its contents, and make it available >>> > through a read-only Uint8ClampedArray like ImageData has. Therefore, >>> > converting to ImageData indicates the intent to modify this content, >>> > therefore a copy is warranted. >>> > - provide a way to "neuter" the ImageBitmap when converting it to >>> ImageData, >>> > transferring its contents and avoiding any copy, which is useful if the >>> > ImageBitmap is a temporary object only being created for the purposes >>> of >>> > getting the ImageData. I guess this would be similar to transferrable >>> > objects with workers. >>> >>> Perhaps this could be done by specifying a "transferToImageData" method. >>> >>> This proposal makes ImageBitmap neuterable, and there's intent to >>> implement it in order to allow rendering to canvas contexts from >>> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . >>> >>> -Ken >>> >>> >>> > - add some kind of "load" method on ImageBitmap. It may store its >>> contents >>> > in compressed form, but calling "load" causes it to be decompressed (or >>> > loaded from somewhere else). The ImageBitmap is only guaranteed to be >>> drawn >>> > with "undue latency" after the "load" call. Therefore for the use case >>> of >>> > low-latency rendering "load" can always be called immediately after >>> > creation, but for conversion purposes not calling "load" avoids >>> duplicating >>> > memory. >>> > >>> > Note for devices with discrete GPUs, it's possible that ImageBitmap >>> stores >>> > the decompressed image data in a GPU texture but does not have it in >>> system >>> > RAM. In this case making a copy for the ImageData is also warranted. >>> > >>> > I'm a web developer proposing this because it would
Re: Async Image -> ImageData conversion
On Thu, Jun 25, 2015 at 5:56 AM, Ashley Gullen wrote: > I see that OffscreenCanvas also specifies a close() method for > ImageBitmap. Perhaps browsers could use copy-on-write with the > toImageData() method? Then you can efficiently either transfer or copy > depending on if you close the ImageBitmap you transferred to an ImageData. > > Transfer: > createImageBitmap() -> imageBitmap.toImageData() -> imageBitmap.close() > No copy made, contents transferred to the imageData > > createImageBitmap() -> imageBitmap.toImageData() -> access resulting > imageData.data > Copy made on first access > Copy on write is probably not a good idea because it would require injecting an observer pattern (or some kind of notification mechanism) into Uint8ClampedArray data assignment, which would realistically degrade performance of Image processing pixel loops. Having a one step transfer API (transferToImageData) avoids adding overhead to what is otherwise just a simple memory access. > > On 24 June 2015 at 21:54, Kenneth Russell wrote: > >> On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen wrote: >> > Sorry for the confusion. Yes, the latest URL is: >> > >> https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html >> > I'm new to specs and WebIDL, my intent was to say those are new methods >> on >> > ImageBitmap. Is "partial interface ImageBitmap" the correct way to say >> that? >> > (I updated the URL to say that instead) >> > >> > The wording of "undue latency" in the ImageBitmap spec does not appear >> to >> > rule out ImageBitmap storing its encoded image data, and lazily >> decoding it. >> > However I'm not sure what the point of that would be, since it seems to >> be >> > how drawing HTMLImageElements to a canvas already works. I see a couple >> of >> > options: >> > - allow lazy decoding in ImageBitmap, and therefore converting to >> ImageData >> > is an explicit request to decompress >> > - require ImageBitmap to decompress its contents, and make it available >> > through a read-only Uint8ClampedArray like ImageData has. Therefore, >> > converting to ImageData indicates the intent to modify this content, >> > therefore a copy is warranted. >> > - provide a way to "neuter" the ImageBitmap when converting it to >> ImageData, >> > transferring its contents and avoiding any copy, which is useful if the >> > ImageBitmap is a temporary object only being created for the purposes of >> > getting the ImageData. I guess this would be similar to transferrable >> > objects with workers. >> >> Perhaps this could be done by specifying a "transferToImageData" method. >> >> This proposal makes ImageBitmap neuterable, and there's intent to >> implement it in order to allow rendering to canvas contexts from >> worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . >> >> -Ken >> >> >> > - add some kind of "load" method on ImageBitmap. It may store its >> contents >> > in compressed form, but calling "load" causes it to be decompressed (or >> > loaded from somewhere else). The ImageBitmap is only guaranteed to be >> drawn >> > with "undue latency" after the "load" call. Therefore for the use case >> of >> > low-latency rendering "load" can always be called immediately after >> > creation, but for conversion purposes not calling "load" avoids >> duplicating >> > memory. >> > >> > Note for devices with discrete GPUs, it's possible that ImageBitmap >> stores >> > the decompressed image data in a GPU texture but does not have it in >> system >> > RAM. In this case making a copy for the ImageData is also warranted. >> > >> > I'm a web developer proposing this because it would be useful for my >> > purposes, I've no idea which of these would be favoured by implementors >> or >> > the spec process. I'm happy to get involved in spec work though so >> please >> > let me know if you have any feedback/suggestions on anything I'm doing >> here. >> > >> > Ashley >> > >> > >> > >> > On 24 June 2015 at 19:12, Boris Zbarsky wrote: >> >> >> >> On 6/19/15 5:43 AM, Ashley Gullen wrote: >> >>> >> >>> I've not done this before, so I've no idea if this is the right/useful >> >>> approach, but I drafted a spec for it here: >> >>> >> >>> https://www.scirra.com/labs/specs/imagedata-blob-extensions.html >> >> >> >> >> >> Ashley, >> >> >> >> We at Mozilla were just discussing this proposal, and we have a >> concern. >> >> >> >> With this proposal, going from an to an ImageData requires >> >> conversion of an intermediate ImageBitmap. Unfortunately, an >> ImageBitmap is >> >> specified to be paintable "without undue latency", which we interpret >> to >> >> mean it needs to have decoded image data, and the ImageData will end up >> >> needing to copy said data in practice (because it needs an editable >> copy). >> >> >> >> That creates two copies of the decoded image data in memory, which >> seems >> >> fairly undesirable on memory-constrained devices. >> >> >> >> -Boris >> >> >> >> >> > >> > >
Subject=Re: Async Image -> ImageData conversion
Based on feedback received from web developers, new APIs that move image data around should also strive to eliminate intermediate copies of the image data to avoid memory bloat. I think your proposal achieves that, but I think memory footprint reduction could be a stated objective of the proposal. For example, the current solution of using a canvas forces the creation of an intermediate copy (the canvas). Also, to avoid the multiplication of APIs for moving image data between various representations, I would like to suggest an alternative: using the ImageBitmap interface as a hub. The creation of ImageBitmaps is already asynchronous (createImageBitmap returns a promise), and it has overloads for acquiring images from img, video, canvas, url, blob, imagedata. All that is missing are a few methods for getting data directly out of an ImageBitmap. So I think adding toBlob and toImageData (both async) to ImageBitmap is a more succinct proposal that would address your use cases, and many additional ones at the same time.
