Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()
Yes. Especially on these devices, JPEG decoding is undue latency. Also, Rik's suggestion makes an ImageBitmap just a thin wrapper around a Blob, with decoding performed every time we do a drawImage call. That's bad for performance in many cases. David's proposal lets the author be a little bit more explicit and the UA doesn't have to guess what to do. Rob -- Jtehsauts tshaei dS,o n Wohfy Mdaon yhoaus eanuttehrotraiitny eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.rt sS?o Whhei csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r sGients uapr,e tfaokreg iyvoeunr, 'm aotr atnod sgaoy ,h o'mGee.t uTph eann dt hwea lmka'n? gBoutt uIp waanndt wyeonut thoo mken.o w
Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()
Hi David, is there a reason why you are completely decoding the image when you create the imageBitmap? [1] If you detect a situation where this operation causes excessive memory consumption, you could hold on to the compressed data URL and defer decoding until the point where it is actually needed. Since exhausting VM will create undue latency, this workaround follows the spirit of the spec. If you really want to have the downsampled bits in memory, you could create a canvas and draw your image into it. On Tue, Dec 17, 2013 at 4:31 PM, David Flanagan dflana...@mozilla.comwrote: This is a proposal for changes to ImageBitmap and createImageBitmap() to enable the memory-efficient display of large images on small screens. Background: The camera resolution on mobile devices has grown (and is continuing to grow) much faster than the screen size and memory of those devices. In my work with FirefoxOS, I work with devices that have camera sensors that can capture 5 megapixels images but have 320x480 pixel (0.15 megapixel) screens. This means that photos from the camera are 33 times as large as the screen. An RGBA image format requires 4 bytes per pixels for decoded image data, so if I want to decode one of these 5mp images for display on my 0.15mp screen, I have to allocate 20mb of memory. This particular low-end device I'm talking about has 256mb of RAM, and less than 200mb available for apps, so displaying a single photo requires more than 10% of available memory. To make this work in the initial releases of FirefoxOS, we've limited the camera resolution to 2 or 3mp on low-memory devices and have ensured that our Camera app includes screen-sized EXIF preview images in the photos it captures. We use JavaScript to extract the EXIF preview from the photo and display that, when we can, instead of the actual image. So initial display of a photo does not actually require us to decode the full-size photo. But as soon as the user zooms in, we have do have to decode it and take the memory hit. The result is that on low-end FirefoxOS phones background apps (including the homescreen) are commonly killed while using the Gallery app. The web platform has two mechanisms for decoding images: the img element and the new window.createImageBitmap() function. Native libraries exist that can downsample an image while decoding it, but the web platform does not expose this feature. This is a fundamental shortcoming: Web apps will not be able to achieve parity with native photo display and processing apps until they are able to decode and downsample a large image into a smaller bitmap. With that as background, I'd like to propose changes to the createImageBitmap() factory method and also related changes to ImageBitmap itself. Proposal: (Note createImageBitmap() can take any ImageBitmapSource object as its first argument. I am primarily concerned with case where the first argument is a Blob and image decoding is involved, though I think that the changes I am proposing will also work for the various other ImageBitmapSource types, but obviously no image decoding will be required for those other types.) 1) My main proposal is to add dw and dh (destination width, destination height) arguments to createImageBitmap() to specify the desired size of the resulting ImageBitmap. dw and dh could be restricted to be less than or equal to sw and sh, so that image enlargement is never required. Note that it is not necessary to specify that an implementation be memory efficient. Desktop implementations that are not memory constrained might decode in one step and then downsample in a separate step. It is enough here to have an API that allows memory-efficient implementations on devices that need it. 2) There are times when we want to downsample and decode the entire image rather than a region of it, and having a string of 6 optional arguments to createImageBitmap() is awkward, so I'd like to suggest that we convert the method so that it takes an ImageBitmapSource as the first argument and an options Dictionary as the second argument to hold the various optional (sx, sy, sw, sh, dw, dh, etc.) arguments. 3) Sometimes we want to decode and downsample a Blob but do not know the pixel size or aspect ratio of the original image, so we cannot specify exact dw, dh values. My main use case here is to obtain a decoded image that is no bigger than necessary but maintains the aspect ratio of the original. One way to get this would be to allow maxWidth and maxHeight properties in the options dictionary. If those properties were defined, then createImageBitmap() would maintain the aspect ratio and create an ImageBitmap that is no wider than maxWidth and no higher than maxHeight. Another, more flexible, solution would be to allow a size property in the dictionary. If size was omitted, then the dw and dh properties would be the actual size of the ImageBitmap, even if that resulted
Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()
On 12/17/13 8:36 PM, Rik Cabanier wrote: Hi David, is there a reason why you are completely decoding the image when you create the imageBitmap? [1] I assume that that is the intent of calling createImageBitmap() on a blob. Since JPEG decoding probably takes significantly longer than blocking on memory access, I assume that lazy decoding is not really allowed. But that misses my point. On the devices I'm concerned with I can never completely decode the image whether it is deferred or not. If I decode at full size, apps running in the background are likely to be killed because of low memory. I need the ability to do the downsampling during the decoding process, so that there is never the memory impact of holding the entire full-size image in memory. If you detect a situation where this operation causes excessive memory consumption, you could hold on to the compressed data URL and defer decoding until the point where it is actually needed. Since exhausting VM will create undue latency, this workaround follows the spirit of the spec. If you really want to have the downsampled bits in memory, you could create a canvas and draw your image into it. I can't do that because I don't have (and cannot have) a full-size decoded image. I've got a blob that is a JPEG encoded 5 megapixel image. And I want to end up with a decoded 320x480 image. And I want to get from A to B without ever allocating 20mb and decoding the image at full size. snip 6) Finally, because image data can take up so much memory, I would like to propose that ImageBitmap have a release() method to explicitly free the memory that holds the decoded image when that decoded image data is no longer needed. This gives web applications more precise control over memory allocation without having to wait for garbage collection. There was an email thread on adding this to canvas [2], it seems reasonable to add it to imageBitmap as well. 1: http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap 2: http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html Thanks for this link. It looks like the January message quoted in this July message is requesting exactly the same feature as I am for discarding or releasing ImageBitmaps. David
Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()
On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan dflana...@mozilla.comwrote: On 12/17/13 8:36 PM, Rik Cabanier wrote: Hi David, is there a reason why you are completely decoding the image when you create the imageBitmap? [1] I assume that that is the intent of calling createImageBitmap() on a blob. Since JPEG decoding probably takes significantly longer than blocking on memory access, I assume that lazy decoding is not really allowed. No, nothing in the spec says that you *must* decode the bits: The exact judgement of what is undue latency of this is left up to the implementer, but in general if making use of the bitmap requires network I/O, or even local disk I/O, then the latency is probably undue; whereas if it only requires a blocking read from a GPU or system RAM, the latency is probably acceptable. In your case, things are reversed. Allocating system ram will kill performance and cause undue latency. Reading the JPEG image on the fly from a Flash disk will be less disruptive and faster. But that misses my point. On the devices I'm concerned with I can never completely decode the image whether it is deferred or not. If I decode at full size, apps running in the background are likely to be killed because of low memory. I need the ability to do the downsampling during the decoding process, so that there is never the memory impact of holding the entire full-size image in memory. If you detect a situation where this operation causes excessive memory consumption, you could hold on to the compressed data URL and defer decoding until the point where it is actually needed. Since exhausting VM will create undue latency, this workaround follows the spirit of the spec. If you really want to have the downsampled bits in memory, you could create a canvas and draw your image into it. I can't do that because I don't have (and cannot have) a full-size decoded image. I've got a blob that is a JPEG encoded 5 megapixel image. And I want to end up with a decoded 320x480 image. And I want to get from A to B without ever allocating 20mb and decoding the image at full size The downsampling happens *during* the drawimage of the imageBitmap into the canvas. At no point do you have to allocate 20mb. snip 6) Finally, because image data can take up so much memory, I would like to propose that ImageBitmap have a release() method to explicitly free the memory that holds the decoded image when that decoded image data is no longer needed. This gives web applications more precise control over memory allocation without having to wait for garbage collection. There was an email thread on adding this to canvas [2], it seems reasonable to add it to imageBitmap as well. 1: http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap 2: http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html Thanks for this link. It looks like the January message quoted in this July message is requesting exactly the same feature as I am for discarding or releasing ImageBitmaps. David
Re: [whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()
On 12/17/13 10:55 PM, Rik Cabanier wrote: On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan dflana...@mozilla.com mailto:dflana...@mozilla.com wrote: On 12/17/13 8:36 PM, Rik Cabanier wrote: Hi David, is there a reason why you are completely decoding the image when you create the imageBitmap? [1] I assume that that is the intent of calling createImageBitmap() on a blob. Since JPEG decoding probably takes significantly longer than blocking on memory access, I assume that lazy decoding is not really allowed. No, nothing in the spec says that you *must* decode the bits: The exact judgement of what is undue latency of this is left up to the implementer, but in general if making use of the bitmap requires network I/O, or even local disk I/O, then the latency is probably undue; whereas if it only requires a blocking read from a GPU or system RAM, the latency is probably acceptable. In your case, things are reversed. Allocating system ram will kill performance and cause undue latency. Reading the JPEG image on the fly from a Flash disk will be less disruptive and faster. But that misses my point. On the devices I'm concerned with I can never completely decode the image whether it is deferred or not. If I decode at full size, apps running in the background are likely to be killed because of low memory. I need the ability to do the downsampling during the decoding process, so that there is never the memory impact of holding the entire full-size image in memory. If you detect a situation where this operation causes excessive memory consumption, you could hold on to the compressed data URL and defer decoding until the point where it is actually needed. Since exhausting VM will create undue latency, this workaround follows the spirit of the spec. If you really want to have the downsampled bits in memory, you could create a canvas and draw your image into it. I can't do that because I don't have (and cannot have) a full-size decoded image. I've got a blob that is a JPEG encoded 5 megapixel image. And I want to end up with a decoded 320x480 image. And I want to get from A to B without ever allocating 20mb and decoding the image at full size The downsampling happens *during* the drawimage of the imageBitmap into the canvas. At no point do you have to allocate 20mb. Ah. I see what you're saying now. My first reaction was that's brilliant!. Unfortunately, my second reaction was that drawImage() would then block on the image decoding, and unless this was being done in a Worker, I'm almost certain that would be an unacceptable performance hit. (One of my use cases is scanning an SD card for hundreds of images and generating thumbnails for each of them. If doing this used drawImage() calls that blocked the main thread, my UI would be completely non-responsive.) I also suspect that adding an async version of drawImage() to the canvas API is a non-starter because that API is pretty fundamentally synchronous. David