Re: [whatwg] Canvas 2D memory management
FWIW, imageBitmap.discard() wouldn't be unprecedented - WebGL allows you to explicitly release memory with deleteTexture() rather than letting the GC collect unused textures. Ashley On 18 July 2013 17:50, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Canvas 2D memory management
On Fri, Jul 19, 2013 at 7:09 AM, Ashley Gullen ash...@scirra.com wrote: FWIW, imageBitmap.discard() wouldn't be unprecedented - WebGL allows you to explicitly release memory with deleteTexture() rather than letting the GC collect unused textures. A related issue we have now is with canvas backing stores. It is common for web apps to create temporary canvases to do some offscreen rendering. When the temporary canvas goes out of scope, it continues to consume RAM or GPU memory until it is garbage collected. Occasionally this results in memory-leak-like symptoms. The usual workaround is to use a single persistent global canvas for offscreen work instead of temporary ones (yuck). This could be handled in a cleaner way if there were a .discard() method on canvases elements too. Ashley On 18 July 2013 17:50, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Canvas 2D memory management
Some of my applications would definitely benefit from this as well. A port of one client's game managed to hit around 1GB of backing store/bitmap data combined when preloading all their image assets using img. Even though browsers then discard the bitmap data, it made it difficult to get things running without killing a tab due to hitting a memory limit temporarily. (The assets were not all in use at once, so the actual usage while playing is fine). Having explicit control over whether bitmaps are resident in memory would be great for this use case since I can preload the actual file over the network, then do the actual async forced decode by creating an ImageBitmap from a Blob, and discard it when the pixel data is no longer needed (the game already has this information since it uses the C# IDisposable pattern, where resources are disposed after use) On Fri, Jul 19, 2013 at 12:34 PM, Justin Novosad ju...@google.com wrote: On Fri, Jul 19, 2013 at 7:09 AM, Ashley Gullen ash...@scirra.com wrote: FWIW, imageBitmap.discard() wouldn't be unprecedented - WebGL allows you to explicitly release memory with deleteTexture() rather than letting the GC collect unused textures. A related issue we have now is with canvas backing stores. It is common for web apps to create temporary canvases to do some offscreen rendering. When the temporary canvas goes out of scope, it continues to consume RAM or GPU memory until it is garbage collected. Occasionally this results in memory-leak-like symptoms. The usual workaround is to use a single persistent global canvas for offscreen work instead of temporary ones (yuck). This could be handled in a cleaner way if there were a .discard() method on canvases elements too. Ashley On 18 July 2013 17:50, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. -- Ian Hickson U+1047E)\._.,--,'``. fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Canvas 2D memory management
On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Canvas 2D memory management
On Thu, Jul 18, 2013 at 12:50 PM, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. This is probably an area where most browsers could do a better job. Browsers should be able to handle the texture memory issues automatically without any new APIs, if they can't, then file bug reports. If garbage collection is not kicking-in at the right time, report it to the vendor. ImageBitmap should provide the same kind of pinning semantics as the suggested ctx.load/unload. However, one weakness of the current API is that upon construction of the ImageBitmap, the browser does not know whether the asset will be used with a GPU-accelerated rendering context or not. If this information were available, the asset could be pre-cached on the GPU when appropriate. Maybe something like ctx.prefetch(image) would be appropriate for warming up the caches. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Canvas 2D memory management
On Thu, Jul 18, 2013 at 2:03 PM, Justin Novosad ju...@google.com wrote: On Thu, Jul 18, 2013 at 12:50 PM, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. This is probably an area where most browsers could do a better job. Browsers should be able to handle the texture memory issues automatically without any new APIs, if they can't, then file bug reports. If garbage collection is not kicking-in at the right time, report it to the vendor. Does the JS VM know about the image bits? It seems not since they live on the C++ side so the imageBitmap could look like a small object that is GC'ed later. ImageBitmap should provide the same kind of pinning semantics as the suggested ctx.load/unload. However, one weakness of the current API is that upon construction of the ImageBitmap, the browser does not know whether the asset will be used with a GPU-accelerated rendering context or not. If this information were available, the asset could be pre-cached on the GPU when appropriate. Maybe something like ctx.prefetch(image) would be appropriate for warming up the caches. That seems too implementation specific.
Re: [whatwg] Canvas 2D memory management
On 7/18/13 5:18 PM, Rik Cabanier wrote: Does the JS VM know about the image bits? For what it's worth, at least in Firefox it would; we already tell the JS VM about all sort of other large C++-side allocations owned by JS objects. -Boris
Re: [whatwg] Canvas 2D memory management
On Thu, Jul 18, 2013 at 5:18 PM, Rik Cabanier caban...@gmail.com wrote: On Thu, Jul 18, 2013 at 2:03 PM, Justin Novosad ju...@google.com wrote: On Thu, Jul 18, 2013 at 12:50 PM, Ian Hickson i...@hixie.ch wrote: On Wed, 9 Jan 2013, Ashley Gullen wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. [...] Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory The Web API tries to use garbage collection for this; the idea being that you load the images you need when you need them, then discard then when you're done, and the memory gets reclaimed when possible. We could introduce a mechanism to flush ImageBitmap objects more forcibly, e.g. imageBitmap.discard(). This would be a pretty new thing, though. Are there any browser vendors who have opinions about this? We should probably wait to see if people are able to use ImageBitmap with garbage collection first. Note, though, that ImageBitmap doesn't really add anything you couldn't do with img before, in the non-Worker case. That is, you could just create img elements then lose references to them when you wanted them GC'ed; if that isn't working today, I don't see why it would start working with ImageBitmap. This is probably an area where most browsers could do a better job. Browsers should be able to handle the texture memory issues automatically without any new APIs, if they can't, then file bug reports. If garbage collection is not kicking-in at the right time, report it to the vendor. Does the JS VM know about the image bits? It seems not since they live on the C++ side so the imageBitmap could look like a small object that is GC'ed later. If there is any memory consumed by the ImageBitmap that is pinned for the life-time of the object, then that memory needs to be declared to the JS VM even if the data does not live in the JS heap. I know V8 and JavaScriptCore have APIs for that and other engines probably do too, so the problem is manageable.
Re: [whatwg] Canvas 2D memory management
On 11.01.13 18:41, Rik Cabanier caban...@gmail.com wrote: On Wed, Jan 9, 2013 at 8:00 AM, Ashley Gullen ash...@scirra.com wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. We have a WebGL renderer which solves this by explicitly creating and deleting textures as necessary when switching levels, which guarantees that memory is managed efficiently. It also has the additional benefit that all necessary textures are pre-loaded, so there's no janking during the game as the first drawImage() of a particular asset in the level uploads a texture. I would like to suggest memory management features for the canvas 2D rendering context. By explicitly pre-loading images and releasing them at the end of the level we can guarantee that devices will not run out of memory, as well as making gameplay smoother. Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used Is this what you're looking for: http://www.whatwg.org/specs/web-apps/current-work/#imagebitmap Very interesting! I wasn't aware of this part of the spec. Sounds like this could become extremely useful. Looking forward to play with an implementation to see if it fixes our issues. ctx.unload(image): release the image from memory Releasing all reference to ImageBitmap should release it from memory. Are you looking for a scheme that does not involve garbage collection? Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. 2) we can drawImage() every image on startup to force lazy-loading browsers to load everything that will be used, but there's still no way to indicate which images should be released at the end of a level. This could be left for the browser to determine (perhaps releasing by least-recently-used), but perhaps this should be required in the specification? 3) leave current behavior as it is and suggest WebGL for this type of application My preference is option 1, but I don't know if this works for all use cases and will work nicely with implementations. Any thoughts? Ashley Gullen Scirra.com
Re: [whatwg] Canvas 2D memory management
I'd be curious to learn more about how IE handles it. Does it use least-recently-used or some other heuristics? I think letting the browser guess could result in sub-optimal load times. Consider a game with two levels: Level 1: uses texture set A and B Level 2: uses texture set A and C When switching from level 1 to level 2, we want to release texture set B then load texture set C, leaving set A in memory. If the browser guesses about releasing textures, it may release textures from set A while loading set C, when it would be more appropriate to release textures from set B. Since level 2 still needs set A, it will load them again. This means either the game still janks during level 2, or when loading textures from set A are needlessly released and loaded again, increasing the load time. I hadn't heard of ImageBitmap before, and it looks like it could solve the problem, assuming the existence of an ImageBitmap implies the texture is already loaded in memory (which it looks like the intent is). If set A's ImageBitmap objects are still referenced, and set B are unreferenced and collected when loading set C if the device runs out of memory, that appears to be optimal behavior. What's the implementation status of ImageBitmap? Do any browsers support it yet? Ashley Gullen Scirra.com On 11 January 2013 22:56, Rik Cabanier caban...@gmail.com wrote: On Fri, Jan 11, 2013 at 2:36 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Sat, Jan 12, 2013 at 6:41 AM, Rik Cabanier caban...@gmail.com wrote: Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. WeakMaps don't work like that. You can only test whether something is in a WeakMap by looking it up with a key object. If the key object stays alive and the WeakMap stays alive, then the value must also stay alive. On other hand, if the key object or the WeakMap have been collected, obviously you can't do the lookup. So you can't observe whether GC has happened using a WeakMap. (This is by design.) Sorry about that! yes, you are correct that you can't use it that way. Someone told me that WeakMaps work the same way as Flash's dictionary with weak keys, but that is not the case. FWIW, flash game developers use dictionaries with 'weak keys' as I described earlier. If the memory runs low, the garbage collector will delete objects from that cache.
Re: [whatwg] Canvas 2D memory management
On Sat, Jan 12, 2013 at 11:52 AM, Ashley Gullen ash...@scirra.com wrote: I'd be curious to learn more about how IE handles it. Does it use least-recently-used or some other heuristics? I think letting the browser guess could result in sub-optimal load times. Consider a game with two levels: Level 1: uses texture set A and B Level 2: uses texture set A and C When switching from level 1 to level 2, we want to release texture set B then load texture set C, leaving set A in memory. If the browser guesses about releasing textures, it may release textures from set A while loading set C, when it would be more appropriate to release textures from set B. Since level 2 still needs set A, it will load them again. This means either the game still janks during level 2, or when loading textures from set A are needlessly released and loaded again, increasing the load time. I hadn't heard of ImageBitmap before, and it looks like it could solve the problem, assuming the existence of an ImageBitmap implies the texture is already loaded in memory (which it looks like the intent is). If set A's ImageBitmap objects are still referenced, and set B are unreferenced and collected when loading set C if the device runs out of memory, that appears to be optimal behavior. Well, it would be the optimal behavior if you knew for sure that A and (B or c) would fit in the browser's memory without having to resort to paging. Since you don't, you might still get worse janking (or outright failure) than if you could give hints to the JS engine that certain things are OK to be destroyed first. (This is probably a discussion for a JavaScript mailing list and not whatwg.) What's the implementation status of ImageBitmap? Do any browsers support it yet? I believe noone has implemented it yet. I scanned the WebKit and Mozilla code bases and didn't find any references. On 11 January 2013 22:56, Rik Cabanier caban...@gmail.com wrote: On Fri, Jan 11, 2013 at 2:36 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Sat, Jan 12, 2013 at 6:41 AM, Rik Cabanier caban...@gmail.comwrote: Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. WeakMaps don't work like that. You can only test whether something is in a WeakMap by looking it up with a key object. If the key object stays alive and the WeakMap stays alive, then the value must also stay alive. On other hand, if the key object or the WeakMap have been collected, obviously you can't do the lookup. So you can't observe whether GC has happened using a WeakMap. (This is by design.) Sorry about that! yes, you are correct that you can't use it that way. Someone told me that WeakMaps work the same way as Flash's dictionary with weak keys, but that is not the case. FWIW, flash game developers use dictionaries with 'weak keys' as I described earlier. If the memory runs low, the garbage collector will delete objects from that cache.
Re: [whatwg] Canvas 2D memory management
Hi Ashley, I brought this very topic up in the latest W3C performance workgroup meeting but pretty much all of my ideas were cut off due to a number of parties believing that these are ultimately UA-specific (browser specific) concerns, and should be dealt with at the browser level. Jason Weber of the IE team (cc'ed), in particular, was confident that IE handling it sufficiently already. To his credits, I simply didn't try to come up with a test case that proves otherwise, so I think that would be the next step needed to convince vendors that this is a generic issue. Maybe you can help? In any case, here [1] are my slides from the meeting and here [2] is the report from the workshop. [1] https://docs.google.com/presentation/d/1a1NfQmRtuQYtBgfPVVHQBwWzDgmPaHASsyM FWLwWMbI/edit#slide=id.p [2] http://www.w3.org/2012/11/performance-workshop/report.html#i10 Thanks! Paul On 09.01.13 17:00, Ashley Gullen ash...@scirra.com wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. We have a WebGL renderer which solves this by explicitly creating and deleting textures as necessary when switching levels, which guarantees that memory is managed efficiently. It also has the additional benefit that all necessary textures are pre-loaded, so there's no janking during the game as the first drawImage() of a particular asset in the level uploads a texture. I would like to suggest memory management features for the canvas 2D rendering context. By explicitly pre-loading images and releasing them at the end of the level we can guarantee that devices will not run out of memory, as well as making gameplay smoother. Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory 2) we can drawImage() every image on startup to force lazy-loading browsers to load everything that will be used, but there's still no way to indicate which images should be released at the end of a level. This could be left for the browser to determine (perhaps releasing by least-recently-used), but perhaps this should be required in the specification? 3) leave current behavior as it is and suggest WebGL for this type of application My preference is option 1, but I don't know if this works for all use cases and will work nicely with implementations. Any thoughts? Ashley Gullen Scirra.com
Re: [whatwg] Canvas 2D memory management
On Wed, Jan 9, 2013 at 8:00 AM, Ashley Gullen ash...@scirra.com wrote: Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. We have a WebGL renderer which solves this by explicitly creating and deleting textures as necessary when switching levels, which guarantees that memory is managed efficiently. It also has the additional benefit that all necessary textures are pre-loaded, so there's no janking during the game as the first drawImage() of a particular asset in the level uploads a texture. I would like to suggest memory management features for the canvas 2D rendering context. By explicitly pre-loading images and releasing them at the end of the level we can guarantee that devices will not run out of memory, as well as making gameplay smoother. Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used Is this what you're looking for: http://www.whatwg.org/specs/web-apps/current-work/#imagebitmap ctx.unload(image): release the image from memory Releasing all reference to ImageBitmap should release it from memory. Are you looking for a scheme that does not involve garbage collection? Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. 2) we can drawImage() every image on startup to force lazy-loading browsers to load everything that will be used, but there's still no way to indicate which images should be released at the end of a level. This could be left for the browser to determine (perhaps releasing by least-recently-used), but perhaps this should be required in the specification? 3) leave current behavior as it is and suggest WebGL for this type of application My preference is option 1, but I don't know if this works for all use cases and will work nicely with implementations. Any thoughts? Ashley Gullen Scirra.com
Re: [whatwg] Canvas 2D memory management
On Sat, Jan 12, 2013 at 6:41 AM, Rik Cabanier caban...@gmail.com wrote: Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. WeakMaps don't work like that. You can only test whether something is in a WeakMap by looking it up with a key object. If the key object stays alive and the WeakMap stays alive, then the value must also stay alive. On other hand, if the key object or the WeakMap have been collected, obviously you can't do the lookup. So you can't observe whether GC has happened using a WeakMap. (This is by design.) Rob -- Jesus called them together and said, “You know that the rulers of the Gentiles lord it over them, and their high officials exercise authority over them. Not so with you. Instead, whoever wants to become great among you must be your servant, and whoever wants to be first must be your slave — just as the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many.” [Matthew 20:25-28]
Re: [whatwg] Canvas 2D memory management
On Fri, Jan 11, 2013 at 2:36 PM, Robert O'Callahan rob...@ocallahan.orgwrote: On Sat, Jan 12, 2013 at 6:41 AM, Rik Cabanier caban...@gmail.com wrote: Some of you concerns with memory management could be addressed with WeakMaps. Basically, you can put all your images in a WeakMap and during the draw cycle, you pull them out and use them. If they're no longer there, it meant that the garbage collector has kicked in to free up memory and you need to reload (which will unfortunately cause a jank). I'm unsure how smart the garbage collector is because you probably want the items in the WeakMaps to be deleted last. WeakMaps don't work like that. You can only test whether something is in a WeakMap by looking it up with a key object. If the key object stays alive and the WeakMap stays alive, then the value must also stay alive. On other hand, if the key object or the WeakMap have been collected, obviously you can't do the lookup. So you can't observe whether GC has happened using a WeakMap. (This is by design.) Sorry about that! yes, you are correct that you can't use it that way. Someone told me that WeakMaps work the same way as Flash's dictionary with weak keys, but that is not the case. FWIW, flash game developers use dictionaries with 'weak keys' as I described earlier. If the memory runs low, the garbage collector will delete objects from that cache.
[whatwg] Canvas 2D memory management
Some developers are starting to design large scale games using our HTML5 game engine, and we're finding we're running in to memory management issues. Consider a device with 50mb of texture memory available. A game might contain 100mb of texture assets, but only use a maximum of 30mb of them at a time (e.g. if there are three levels each using 30mb of different assets, and a menu that uses 10mb of assets). This game ought to fit in memory at all times, but if a user agent is not smart about how image loading is handled, it could run out of memory. We have a WebGL renderer which solves this by explicitly creating and deleting textures as necessary when switching levels, which guarantees that memory is managed efficiently. It also has the additional benefit that all necessary textures are pre-loaded, so there's no janking during the game as the first drawImage() of a particular asset in the level uploads a texture. I would like to suggest memory management features for the canvas 2D rendering context. By explicitly pre-loading images and releasing them at the end of the level we can guarantee that devices will not run out of memory, as well as making gameplay smoother. Some ideas: 1) add new functions to the canvas 2D context, such as: ctx.load(image): cache an image in memory so it can be immediately drawn when drawImage() is first used ctx.unload(image): release the image from memory 2) we can drawImage() every image on startup to force lazy-loading browsers to load everything that will be used, but there's still no way to indicate which images should be released at the end of a level. This could be left for the browser to determine (perhaps releasing by least-recently-used), but perhaps this should be required in the specification? 3) leave current behavior as it is and suggest WebGL for this type of application My preference is option 1, but I don't know if this works for all use cases and will work nicely with implementations. Any thoughts? Ashley Gullen Scirra.com
