Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

2013-12-18 Thread Robert O'Callahan
Yes. Especially on these devices, JPEG decoding is undue latency.

Also, Rik's suggestion makes an ImageBitmap just a thin wrapper around a
Blob, with decoding performed every time we do a drawImage call. That's bad
for performance in many cases. David's proposal lets the author be a little
bit more explicit and the UA doesn't have to guess what to do.

Rob
-- 
Jtehsauts  tshaei dS,o n Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w


Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

2013-12-17 Thread Rik Cabanier
Hi David,

is there a reason why you are completely decoding the image when you create
the imageBitmap? [1]
If you detect a situation where this operation causes excessive memory
consumption, you could hold on to the compressed data URL and defer
decoding until the point where it is actually needed.
Since exhausting VM will create undue latency, this workaround follows
the spirit of the spec.

If you really want to have the downsampled bits in memory, you could create
a canvas and draw your image into it.

On Tue, Dec 17, 2013 at 4:31 PM, David Flanagan dflana...@mozilla.comwrote:

 This is a proposal for changes to ImageBitmap and createImageBitmap()
 to enable the memory-efficient display of large images on small screens.

 Background:

 The camera resolution on mobile devices has grown (and is continuing
 to grow) much faster than the screen size and memory of those
 devices. In my work with FirefoxOS, I work with devices that have
 camera sensors that can capture 5 megapixels images but have 320x480
 pixel (0.15 megapixel) screens. This means that photos from the
 camera are 33 times as large as the screen.

 An RGBA image format requires 4 bytes per pixels for decoded image
 data, so if I want to decode one of these 5mp images for display on my
 0.15mp screen, I have to allocate 20mb of memory. This particular
 low-end device I'm talking about has 256mb of RAM, and less than 200mb
 available for apps, so displaying a single photo requires more than
 10% of available memory.

 To make this work in the initial releases of FirefoxOS, we've limited
 the camera resolution to 2 or 3mp on low-memory devices and have
 ensured that our Camera app includes screen-sized EXIF preview images
 in the photos it captures. We use JavaScript to extract the EXIF
 preview from the photo and display that, when we can, instead of the
 actual image. So initial display of a photo does not actually require
 us to decode the full-size photo. But as soon as the user zooms in,
 we have do have to decode it and take the memory hit. The result is
 that on low-end FirefoxOS phones background apps (including the
 homescreen) are commonly killed while using the Gallery app.

 The web platform has two mechanisms for decoding images: the img
 element and the new window.createImageBitmap() function. Native
 libraries exist that can downsample an image while decoding it, but
 the web platform does not expose this feature. This is a fundamental
 shortcoming: Web apps will not be able to achieve parity with native
 photo display and processing apps until they are able to decode and
 downsample a large image into a smaller bitmap.

 With that as background, I'd like to propose changes to the
 createImageBitmap() factory method and also related changes to
 ImageBitmap itself.

 Proposal:

 (Note createImageBitmap() can take any ImageBitmapSource object as its
 first argument. I am primarily concerned with case where the first
 argument is a Blob and image decoding is involved, though I think that
 the changes I am proposing will also work for the various other
 ImageBitmapSource types, but obviously no image decoding will be
 required for those other types.)

 1) My main proposal is to add dw and dh (destination width,
 destination height) arguments to createImageBitmap() to specify the
 desired size of the resulting ImageBitmap. dw and dh could be
 restricted to be less than or equal to sw and sh, so that image
 enlargement is never required. Note that it is not necessary to
 specify that an implementation be memory efficient. Desktop
 implementations that are not memory constrained might decode in one
 step and then downsample in a separate step. It is enough here to have
 an API that allows memory-efficient implementations on devices that
 need it.

 2) There are times when we want to downsample and decode the entire
 image rather than a region of it, and having a string of 6 optional
 arguments to createImageBitmap() is awkward, so I'd like to suggest
 that we convert the method so that it takes an ImageBitmapSource as
 the first argument and an options Dictionary as the second argument
 to hold the various optional (sx, sy, sw, sh, dw, dh, etc.) arguments.

 3) Sometimes we want to decode and downsample a Blob but do not know
 the pixel size or aspect ratio of the original image, so we cannot
 specify exact dw, dh values.  My main use case here is to obtain a
 decoded image that is no bigger than necessary but maintains the
 aspect ratio of the original.  One way to get this would be to allow
 maxWidth and maxHeight properties in the options dictionary.  If those
 properties were defined, then createImageBitmap() would maintain the
 aspect ratio and create an ImageBitmap that is no wider than maxWidth
 and no higher than maxHeight.  Another, more flexible, solution would
 be to allow a size property in the dictionary. If size was omitted,
 then the dw and dh properties would be the actual size of the
 ImageBitmap, even if that resulted 

Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

2013-12-17 Thread David Flanagan

On 12/17/13 8:36 PM, Rik Cabanier wrote:

Hi David,

is there a reason why you are completely decoding the image when you 
create the imageBitmap? [1]
I assume that that is the intent of calling createImageBitmap() on a 
blob. Since JPEG decoding probably takes significantly longer than 
blocking on memory access, I assume that lazy decoding is not really 
allowed.


But that misses my point. On the devices I'm concerned with I can never 
completely decode the image whether it is deferred or not.  If I decode 
at full size, apps running in the background are likely to be killed 
because of low memory. I need the ability to do the downsampling during 
the decoding process, so that there is never the memory impact of 
holding the entire full-size image in memory.


If you detect a situation where this operation causes excessive memory 
consumption, you could hold on to the compressed data URL and defer 
decoding until the point where it is actually needed.
Since exhausting VM will create undue latency, this workaround 
follows the spirit of the spec.


If you really want to have the downsampled bits in memory, you could 
create a canvas and draw your image into it.
I can't do that because I don't have (and cannot have) a full-size 
decoded image.  I've got a blob that is a JPEG encoded 5 megapixel 
image.  And I want to end up with a decoded 320x480 image.  And I want 
to get from A to B without ever allocating 20mb and decoding the image 
at full size.


snip


6) Finally, because image data can take up so much memory, I would
like to propose that ImageBitmap have a release() method to explicitly
free the memory that holds the decoded image when that decoded image
data is no longer needed. This gives web applications more precise
control over memory allocation without having to wait for garbage
collection.


There was an email thread on adding this to canvas [2], it seems 
reasonable to add it to imageBitmap as well.


1: 
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap
2: 
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html
Thanks for this link.  It looks like the January message quoted in this 
July message is requesting exactly the same feature as I am for 
discarding or releasing ImageBitmaps.


  David


Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

2013-12-17 Thread Rik Cabanier
On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan dflana...@mozilla.comwrote:

  On 12/17/13 8:36 PM, Rik Cabanier wrote:

 Hi David,

  is there a reason why you are completely decoding the image when you
 create the imageBitmap? [1]

 I assume that that is the intent of calling createImageBitmap() on a blob.
 Since JPEG decoding probably takes significantly longer than blocking on
 memory access, I assume that lazy decoding is not really allowed.


No, nothing in the spec says that you *must* decode the bits:

The exact judgement of what is undue latency of this is left up to the
implementer, but in general if making use of the bitmap requires network
I/O, or even local disk I/O, then the latency is probably undue; whereas if
it only requires a blocking read from a GPU or system RAM, the latency is
probably acceptable.


In your case, things are reversed. Allocating system ram will kill
performance and cause undue latency. Reading the JPEG image on the fly from
a Flash disk will be less disruptive and faster.


 But that misses my point. On the devices I'm concerned with I can never
 completely decode the image whether it is deferred or not.  If I decode at
 full size, apps running in the background are likely to be killed because
 of low memory. I need the ability to do the downsampling during the
 decoding process, so that there is never the memory impact of holding the
 entire full-size image in memory.


   If you detect a situation where this operation causes excessive memory
 consumption, you could hold on to the compressed data URL and defer
 decoding until the point where it is actually needed.
 Since exhausting VM will create undue latency, this workaround follows
 the spirit of the spec.

  If you really want to have the downsampled bits in memory, you could
 create a canvas and draw your image into it.

 I can't do that because I don't have (and cannot have) a full-size decoded
 image.  I've got a blob that is a JPEG encoded 5 megapixel image.  And I
 want to end up with a decoded 320x480 image.  And I want to get from A to B
 without ever allocating 20mb and decoding the image at full size


The downsampling happens *during* the drawimage of the imageBitmap into the
canvas. At no point do you have to allocate 20mb.

snip


  6) Finally, because image data can take up so much memory, I would
 like to propose that ImageBitmap have a release() method to explicitly
 free the memory that holds the decoded image when that decoded image
 data is no longer needed. This gives web applications more precise
 control over memory allocation without having to wait for garbage
 collection.


  There was an email thread on adding this to canvas [2], it seems
 reasonable to add it to imageBitmap as well.


  1:
 http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap
 2:
 http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html


 Thanks for this link.  It looks like the January message quoted in this
 July message is requesting exactly the same feature as I am for discarding
 or releasing ImageBitmaps.

   David



Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

2013-12-17 Thread David Flanagan

On 12/17/13 10:55 PM, Rik Cabanier wrote:




On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan dflana...@mozilla.com 
mailto:dflana...@mozilla.com wrote:


On 12/17/13 8:36 PM, Rik Cabanier wrote:

Hi David,

is there a reason why you are completely decoding the image when
you create the imageBitmap? [1]

I assume that that is the intent of calling createImageBitmap() on
a blob. Since JPEG decoding probably takes significantly longer
than blocking on memory access, I assume that lazy decoding is not
really allowed.


No, nothing in the spec says that you *must* decode the bits:

The exact judgement of what is undue latency of this is left up to
the implementer, but in general if making use of the bitmap
requires network I/O, or even local disk I/O, then the latency is
probably undue; whereas if it only requires a blocking read from a
GPU or system RAM, the latency is probably acceptable.


In your case, things are reversed. Allocating system ram will kill 
performance and cause undue latency. Reading the JPEG image on the fly 
from a Flash disk will be less disruptive and faster.


But that misses my point. On the devices I'm concerned with I can
never completely decode the image whether it is deferred or not. 
If I decode at full size, apps running in the background are

likely to be killed because of low memory. I need the ability to
do the downsampling during the decoding process, so that there is
never the memory impact of holding the entire full-size image in
memory.



If you detect a situation where this operation causes excessive
memory consumption, you could hold on to the compressed data URL
and defer decoding until the point where it is actually needed.
Since exhausting VM will create undue latency, this workaround
follows the spirit of the spec.

If you really want to have the downsampled bits in memory, you
could create a canvas and draw your image into it.

I can't do that because I don't have (and cannot have) a full-size
decoded image.  I've got a blob that is a JPEG encoded 5 megapixel
image.  And I want to end up with a decoded 320x480 image.  And I
want to get from A to B without ever allocating 20mb and decoding
the image at full size


The downsampling happens *during* the drawimage of the imageBitmap 
into the canvas. At no point do you have to allocate 20mb.



Ah. I see what you're saying now.  My first reaction was that's 
brilliant!.  Unfortunately, my second reaction was that drawImage() 
would then block on the image decoding, and unless this was being done 
in a Worker, I'm almost certain that would be an unacceptable 
performance hit. (One of my use cases is scanning an SD card for 
hundreds of images and generating thumbnails for each of them.  If doing 
this used drawImage() calls that blocked the main thread, my UI would be 
completely non-responsive.)


I also suspect that adding an async version of drawImage() to the canvas 
API is a non-starter because that API is pretty fundamentally synchronous.


  David