Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Rik Cabanier]

On Tue, Dec 17, 2013 at 11:59 PM, David Flanagan wrote:

>  On 12/17/13 10:55 PM, Rik Cabanier wrote:
>
>
>
>
> On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan wrote:
>
>>  On 12/17/13 8:36 PM, Rik Cabanier wrote:
>>
>> Hi David,
>>
>>  is there a reason why you are completely decoding the image when you
>> create the imageBitmap? [1]
>>
>>  I assume that that is the intent of calling createImageBitmap() on a
>> blob. Since JPEG decoding probably takes significantly longer than blocking
>> on memory access, I assume that lazy decoding is not really allowed.
>>
>
>  No, nothing in the spec says that you *must* decode the bits:
>
>  The exact judgement of what is undue latency of this is left up to the
> implementer, but in general if making use of the bitmap requires network
> I/O, or even local disk I/O, then the latency is probably undue; whereas if
> it only requires a blocking read from a GPU or system RAM, the latency is
> probably acceptable.
>
>
>  In your case, things are reversed. Allocating system ram will kill
> performance and cause undue latency. Reading the JPEG image on the fly from
> a Flash disk will be less disruptive and faster.
>
>
>>  But that misses my point. On the devices I'm concerned with I can never
>> completely decode the image whether it is deferred or not.  If I decode at
>> full size, apps running in the background are likely to be killed because
>> of low memory. I need the ability to do the downsampling during the
>> decoding process, so that there is never the memory impact of holding the
>> entire full-size image in memory.
>>
>>
>>   If you detect a situation where this operation causes excessive memory
>> consumption, you could hold on to the compressed data URL and defer
>> decoding until the point where it is actually needed.
>> Since exhausting VM will create "undue latency", this workaround follows
>> the spirit of the spec.
>>
>>  If you really want to have the downsampled bits in memory, you could
>> create a canvas and draw your image into it.
>>
>>  I can't do that because I don't have (and cannot have) a full-size
>> decoded image.  I've got a blob that is a JPEG encoded 5 megapixel image.
>> And I want to end up with a decoded 320x480 image.  And I want to get from
>> A to B without ever allocating 20mb and decoding the image at full size
>>
>
>  The downsampling happens *during* the drawimage of the imageBitmap into
> the canvas. At no point do you have to allocate 20mb.
>
>
>   Ah. I see what you're saying now.  My first reaction was "that's
> brilliant!".  Unfortunately, my second reaction was that drawImage() would
> then block on the image decoding, and unless this was being done in a
> Worker, I'm almost certain that would be an unacceptable performance hit.
> (One of my use cases is scanning an SD card for hundreds of images and
> generating thumbnails for each of them.  If doing this used drawImage()
> calls that blocked the main thread, my UI would be completely
> non-responsive.)
>
> I also suspect that adding an async version of drawImage() to the canvas
> API is a non-starter because that API is pretty fundamentally synchronous.
>

It doesn't have to be synchronous. (ie Chrome draws everything asynchronous
) However, presenting the data will still block until all the draw calls
are completed so it wouldn't help in this situation.
Is createImageBitmap going to do the decoding in another thread and fulfill
the promise on the main thread?

Your workflow sounds like it should run in a worker to avoid blocking
behavior.
Is it too much work to add my proposal and the needed JavaScript objects to
a worker? (The browser could be smart about when to decode or hold on to
the blob by looking at available system memory)
If so, your proposal sounds reasonable. Could you also add support so the
type of downsampling can be specified? [1]

So for this reason, I also want to propose that
> ImageBitmap have a transferToCanvas() method akin to the
> transferToImageBitmap() and transferToImage() methods proposed at
> http://wiki.whatwg.org/wiki/WorkerCanvas.  transferToCanvas would
> transfer the image data into a new Canvas object and would neuter the
> ImageBitmap so that it could no longer be used.


We will have to define the CORS issues here.
I assume the canvas gets the same origin as the imageBitmap? The current
spec for imageBitmap only seems to allow same-origin which seems
limiting... [2]

1:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-imagesmoothingenabled
2:
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#images

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Robert O'Callahan]

Yes. Especially on these devices, JPEG decoding is undue latency.

Also, Rik's suggestion makes an ImageBitmap just a thin wrapper around a
Blob, with decoding performed every time we do a drawImage call. That's bad
for performance in many cases. David's proposal lets the author be a little
bit more explicit and the UA doesn't have to guess what to do.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[David Flanagan]

On 12/17/13 10:55 PM, Rik Cabanier wrote:




On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan > wrote:


On 12/17/13 8:36 PM, Rik Cabanier wrote:

Hi David,

is there a reason why you are completely decoding the image when
you create the imageBitmap? [1]

I assume that that is the intent of calling createImageBitmap() on
a blob. Since JPEG decoding probably takes significantly longer
than blocking on memory access, I assume that lazy decoding is not
really allowed.


No, nothing in the spec says that you *must* decode the bits:

The exact judgement of what is undue latency of this is left up to
the implementer, but in general if making use of the bitmap
requires network I/O, or even local disk I/O, then the latency is
probably undue; whereas if it only requires a blocking read from a
GPU or system RAM, the latency is probably acceptable.


In your case, things are reversed. Allocating system ram will kill 
performance and cause undue latency. Reading the JPEG image on the fly 
from a Flash disk will be less disruptive and faster.


But that misses my point. On the devices I'm concerned with I can
never completely decode the image whether it is deferred or not. 
If I decode at full size, apps running in the background are

likely to be killed because of low memory. I need the ability to
do the downsampling during the decoding process, so that there is
never the memory impact of holding the entire full-size image in
memory.



If you detect a situation where this operation causes excessive
memory consumption, you could hold on to the compressed data URL
and defer decoding until the point where it is actually needed.
Since exhausting VM will create "undue latency", this workaround
follows the spirit of the spec.

If you really want to have the downsampled bits in memory, you
could create a canvas and draw your image into it.

I can't do that because I don't have (and cannot have) a full-size
decoded image.  I've got a blob that is a JPEG encoded 5 megapixel
image.  And I want to end up with a decoded 320x480 image.  And I
want to get from A to B without ever allocating 20mb and decoding
the image at full size


The downsampling happens *during* the drawimage of the imageBitmap 
into the canvas. At no point do you have to allocate 20mb.



Ah. I see what you're saying now.  My first reaction was "that's 
brilliant!".  Unfortunately, my second reaction was that drawImage() 
would then block on the image decoding, and unless this was being done 
in a Worker, I'm almost certain that would be an unacceptable 
performance hit. (One of my use cases is scanning an SD card for 
hundreds of images and generating thumbnails for each of them.  If doing 
this used drawImage() calls that blocked the main thread, my UI would be 
completely non-responsive.)


I also suspect that adding an async version of drawImage() to the canvas 
API is a non-starter because that API is pretty fundamentally synchronous.


  David

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Rik Cabanier]

On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan wrote:

>  On 12/17/13 8:36 PM, Rik Cabanier wrote:
>
> Hi David,
>
>  is there a reason why you are completely decoding the image when you
> create the imageBitmap? [1]
>
> I assume that that is the intent of calling createImageBitmap() on a blob.
> Since JPEG decoding probably takes significantly longer than blocking on
> memory access, I assume that lazy decoding is not really allowed.
>

No, nothing in the spec says that you *must* decode the bits:

The exact judgement of what is undue latency of this is left up to the
implementer, but in general if making use of the bitmap requires network
I/O, or even local disk I/O, then the latency is probably undue; whereas if
it only requires a blocking read from a GPU or system RAM, the latency is
probably acceptable.


In your case, things are reversed. Allocating system ram will kill
performance and cause undue latency. Reading the JPEG image on the fly from
a Flash disk will be less disruptive and faster.


> But that misses my point. On the devices I'm concerned with I can never
> completely decode the image whether it is deferred or not.  If I decode at
> full size, apps running in the background are likely to be killed because
> of low memory. I need the ability to do the downsampling during the
> decoding process, so that there is never the memory impact of holding the
> entire full-size image in memory.
>
>
>   If you detect a situation where this operation causes excessive memory
> consumption, you could hold on to the compressed data URL and defer
> decoding until the point where it is actually needed.
> Since exhausting VM will create "undue latency", this workaround follows
> the spirit of the spec.
>
>  If you really want to have the downsampled bits in memory, you could
> create a canvas and draw your image into it.
>
> I can't do that because I don't have (and cannot have) a full-size decoded
> image.  I've got a blob that is a JPEG encoded 5 megapixel image.  And I
> want to end up with a decoded 320x480 image.  And I want to get from A to B
> without ever allocating 20mb and decoding the image at full size
>

The downsampling happens *during* the drawimage of the imageBitmap into the
canvas. At no point do you have to allocate 20mb.


>
>
>  6) Finally, because image data can take up so much memory, I would
>> like to propose that ImageBitmap have a release() method to explicitly
>> free the memory that holds the decoded image when that decoded image
>> data is no longer needed. This gives web applications more precise
>> control over memory allocation without having to wait for garbage
>> collection.
>
>
>  There was an email thread on adding this to canvas [2], it seems
> reasonable to add it to imageBitmap as well.
>
>
>  1:
> http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap
> 2:
> http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html
>
>
> Thanks for this link.  It looks like the January message quoted in this
> July message is requesting exactly the same feature as I am for discarding
> or releasing ImageBitmaps.
>
>   David
>

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[David Flanagan]

On 12/17/13 8:36 PM, Rik Cabanier wrote:

Hi David,

is there a reason why you are completely decoding the image when you 
create the imageBitmap? [1]
I assume that that is the intent of calling createImageBitmap() on a 
blob. Since JPEG decoding probably takes significantly longer than 
blocking on memory access, I assume that lazy decoding is not really 
allowed.


But that misses my point. On the devices I'm concerned with I can never 
completely decode the image whether it is deferred or not.  If I decode 
at full size, apps running in the background are likely to be killed 
because of low memory. I need the ability to do the downsampling during 
the decoding process, so that there is never the memory impact of 
holding the entire full-size image in memory.


If you detect a situation where this operation causes excessive memory 
consumption, you could hold on to the compressed data URL and defer 
decoding until the point where it is actually needed.
Since exhausting VM will create "undue latency", this workaround 
follows the spirit of the spec.


If you really want to have the downsampled bits in memory, you could 
create a canvas and draw your image into it.
I can't do that because I don't have (and cannot have) a full-size 
decoded image.  I've got a blob that is a JPEG encoded 5 megapixel 
image.  And I want to end up with a decoded 320x480 image.  And I want 
to get from A to B without ever allocating 20mb and decoding the image 
at full size.





6) Finally, because image data can take up so much memory, I would
like to propose that ImageBitmap have a release() method to explicitly
free the memory that holds the decoded image when that decoded image
data is no longer needed. This gives web applications more precise
control over memory allocation without having to wait for garbage
collection.


There was an email thread on adding this to canvas [2], it seems 
reasonable to add it to imageBitmap as well.


1: 
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap
2: 
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html
Thanks for this link.  It looks like the January message quoted in this 
July message is requesting exactly the same feature as I am for 
discarding or releasing ImageBitmaps.


  David

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Rik Cabanier]

Hi David,

is there a reason why you are completely decoding the image when you create
the imageBitmap? [1]
If you detect a situation where this operation causes excessive memory
consumption, you could hold on to the compressed data URL and defer
decoding until the point where it is actually needed.
Since exhausting VM will create "undue latency", this workaround follows
the spirit of the spec.

If you really want to have the downsampled bits in memory, you could create
a canvas and draw your image into it.

On Tue, Dec 17, 2013 at 4:31 PM, David Flanagan wrote:

> This is a proposal for changes to ImageBitmap and createImageBitmap()
> to enable the memory-efficient display of large images on small screens.
>
> Background:
>
> The camera resolution on mobile devices has grown (and is continuing
> to grow) much faster than the screen size and memory of those
> devices. In my work with FirefoxOS, I work with devices that have
> camera sensors that can capture 5 megapixels images but have 320x480
> pixel (0.15 megapixel) screens. This means that photos from the
> camera are 33 times as large as the screen.
>
> An RGBA image format requires 4 bytes per pixels for decoded image
> data, so if I want to decode one of these 5mp images for display on my
> 0.15mp screen, I have to allocate 20mb of memory. This particular
> low-end device I'm talking about has 256mb of RAM, and less than 200mb
> available for apps, so displaying a single photo requires more than
> 10% of available memory.
>
> To make this work in the initial releases of FirefoxOS, we've limited
> the camera resolution to 2 or 3mp on low-memory devices and have
> ensured that our Camera app includes screen-sized EXIF preview images
> in the photos it captures. We use JavaScript to extract the EXIF
> preview from the photo and display that, when we can, instead of the
> actual image. So initial display of a photo does not actually require
> us to decode the full-size photo. But as soon as the user zooms in,
> we have do have to decode it and take the memory hit. The result is
> that on low-end FirefoxOS phones background apps (including the
> homescreen) are commonly killed while using the Gallery app.
>
> The web platform has two mechanisms for decoding images: the 
> element and the new window.createImageBitmap() function. Native
> libraries exist that can downsample an image while decoding it, but
> the web platform does not expose this feature. This is a fundamental
> shortcoming: Web apps will not be able to achieve parity with native
> photo display and processing apps until they are able to decode and
> downsample a large image into a smaller bitmap.
>
> With that as background, I'd like to propose changes to the
> createImageBitmap() factory method and also related changes to
> ImageBitmap itself.
>
> Proposal:
>
> (Note createImageBitmap() can take any ImageBitmapSource object as its
> first argument. I am primarily concerned with case where the first
> argument is a Blob and image decoding is involved, though I think that
> the changes I am proposing will also work for the various other
> ImageBitmapSource types, but obviously no image decoding will be
> required for those other types.)
>
> 1) My main proposal is to add dw and dh (destination width,
> destination height) arguments to createImageBitmap() to specify the
> desired size of the resulting ImageBitmap. dw and dh could be
> restricted to be less than or equal to sw and sh, so that image
> enlargement is never required. Note that it is not necessary to
> specify that an implementation be memory efficient. Desktop
> implementations that are not memory constrained might decode in one
> step and then downsample in a separate step. It is enough here to have
> an API that allows memory-efficient implementations on devices that
> need it.
>
> 2) There are times when we want to downsample and decode the entire
> image rather than a region of it, and having a string of 6 optional
> arguments to createImageBitmap() is awkward, so I'd like to suggest
> that we convert the method so that it takes an ImageBitmapSource as
> the first argument and an options Dictionary as the second argument
> to hold the various optional (sx, sy, sw, sh, dw, dh, etc.) arguments.
>
> 3) Sometimes we want to decode and downsample a Blob but do not know
> the pixel size or aspect ratio of the original image, so we cannot
> specify exact dw, dh values.  My main use case here is to obtain a
> decoded image that is no bigger than necessary but maintains the
> aspect ratio of the original.  One way to get this would be to allow
> maxWidth and maxHeight properties in the options dictionary.  If those
> properties were defined, then createImageBitmap() would maintain the
> aspect ratio and create an ImageBitmap that is no wider than maxWidth
> and no higher than maxHeight.  Another, more flexible, solution would
> be to allow a size property in the dictionary. If size was omitted,
> then the dw and dh properties would be t