Re: exposing CANVAS or something like it to Web Workers
On Fri, Feb 8, 2013 at 9:14 AM, Travis Leithead travis.leith...@microsoft.com wrote: What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. Well, it's not a fleshed out proposal by any stretch, but you could imagine that an event could be used to signal that new frames were ready from the producer—then the main thread would know to pop. It sounds like this approach would require a bunch of new concepts -- a remote worker context, events signalled through it, etc. It would still be necessary to postMessage from the main thread back to the worker for flow control. Flow control is definitely necessary -- the producer can't just produce frames without any feedback about when they're actually consumed. We've had problems in Chrome's graphics pipeline in the past where lack of flow control led to slow and inconsistent frame rates. I'm excited about Gregg's proposal because it solves a lot of use cases that aren't currently addressed by CanvasProxy, using a couple of simple primitives that build upon others already in the platform (cross-document messaging and Transferables). How can we move forward so that user agents can experimentally implement these APIs? Ideally we'd prototype and experiment with them in some medium-sized applications before finalizing any specs in this area. Thanks, -Ken From: Gregg Tavares [mailto:g...@google.com] Sent: Friday, February 8, 2013 3:14 AM To: Travis Leithead Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead travis.leith...@microsoft.com wrote: Having thought about this before, I wonder why we don’t use a producer/consumer model rather than a transfer of canvas ownership model? A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), is to have an internal frame buffer accessible via a WorkerCanvas API which supports some set of canvas 2d/3d APIs as appropriate, and can “push” a completed frame onto a stack in the internal frame buffer. Thus the worker can produce frames as fast as desired. On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, which just offers the “pop” API to pop a frame from the internal frame buffer into the canvas. Then you just add some basic signaling events on both ends of the frame buffer and you’re good (as far as synchronizing the worker with the document). The producer (in the worker) is free to produce multiple frames in advance (if desired), while the consumer is able to pop frames when available. You could even have the framebuffer depth configurable. What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. From: Gregg Tavares [mailto:g...@google.com] Sent: Thursday, February 7, 2013 2:25 PM To: Ian Hickson Cc: Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers I put up a new proposal for canvas in workers http://wiki.whatwg.org/wiki/CanvasInWorkers Please take a look. This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki Looking forward to feedback. On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can
Re: exposing CANVAS or something like it to Web Workers
On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead travis.leith...@microsoft.com wrote: Having thought about this before, I wonder why we don’t use a producer/consumer model rather than a transfer of canvas ownership model? ** ** A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), is to have an internal frame buffer accessible via a WorkerCanvas API which supports some set of canvas 2d/3d APIs as appropriate, and can “push” a completed frame onto a stack in the internal frame buffer. Thus the worker can produce frames as fast as desired. ** ** On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, which just offers the “pop” API to pop a frame from the internal frame buffer into the canvas. ** ** Then you just add some basic signaling events on both ends of the frame buffer and you’re good (as far as synchronizing the worker with the document). The producer (in the worker) is free to produce multiple frames in advance (if desired), while the consumer is able to pop frames when available. You could even have the framebuffer depth configurable. What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. ** ** *From:* Gregg Tavares [mailto:g...@google.com] *Sent:* Thursday, February 7, 2013 2:25 PM *To:* Ian Hickson *Cc:* Charles Pritchard; Web Applications Working Group WG *Subject:* Re: exposing CANVAS or something like it to Web Workers ** ** I put up a new proposal for canvas in workers ** ** http://wiki.whatwg.org/wiki/CanvasInWorkers ** ** Please take a look. ** ** This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki ** ** Looking forward to feedback. ** ** ** ** ** ** On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as efficient as commit(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'** ** ** **
RE: exposing CANVAS or something like it to Web Workers
What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. Well, it's not a fleshed out proposal by any stretch, but you could imagine that an event could be used to signal that new frames were ready from the producer—then the main thread would know to pop. From: Gregg Tavares [mailto:g...@google.com] Sent: Friday, February 8, 2013 3:14 AM To: Travis Leithead Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead travis.leith...@microsoft.commailto:travis.leith...@microsoft.com wrote: Having thought about this before, I wonder why we don’t use a producer/consumer model rather than a transfer of canvas ownership model? A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), is to have an internal frame buffer accessible via a WorkerCanvas API which supports some set of canvas 2d/3d APIs as appropriate, and can “push” a completed frame onto a stack in the internal frame buffer. Thus the worker can produce frames as fast as desired. On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, which just offers the “pop” API to pop a frame from the internal frame buffer into the canvas. Then you just add some basic signaling events on both ends of the frame buffer and you’re good (as far as synchronizing the worker with the document). The producer (in the worker) is free to produce multiple frames in advance (if desired), while the consumer is able to pop frames when available. You could even have the framebuffer depth configurable. What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. From: Gregg Tavares [mailto:g...@google.commailto:g...@google.com] Sent: Thursday, February 7, 2013 2:25 PM To: Ian Hickson Cc: Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers I put up a new proposal for canvas in workers http://wiki.whatwg.org/wiki/CanvasInWorkers Please take a look. This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki Looking forward to feedback. On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.chmailto:i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as efficient as commit(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
I put up a new proposal for canvas in workers http://wiki.whatwg.org/wiki/CanvasInWorkers Please take a look. This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki Looking forward to feedback. On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as efficient as commit(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
RE: exposing CANVAS or something like it to Web Workers
Having thought about this before, I wonder why we don’t use a producer/consumer model rather than a transfer of canvas ownership model? A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), is to have an internal frame buffer accessible via a WorkerCanvas API which supports some set of canvas 2d/3d APIs as appropriate, and can “push” a completed frame onto a stack in the internal frame buffer. Thus the worker can produce frames as fast as desired. On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, which just offers the “pop” API to pop a frame from the internal frame buffer into the canvas. Then you just add some basic signaling events on both ends of the frame buffer and you’re good (as far as synchronizing the worker with the document). The producer (in the worker) is free to produce multiple frames in advance (if desired), while the consumer is able to pop frames when available. You could even have the framebuffer depth configurable. From: Gregg Tavares [mailto:g...@google.com] Sent: Thursday, February 7, 2013 2:25 PM To: Ian Hickson Cc: Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers I put up a new proposal for canvas in workers http://wiki.whatwg.org/wiki/CanvasInWorkers Please take a look. This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki Looking forward to feedback. On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.chmailto:i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as efficient as commit(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
On Wed, 2 Jan 2013, Gregg Tavares (社�~T�) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as efficient as commit(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
On Wed, Jan 2, 2013 at 2:52 PM, Gregg Tavares (社用) g...@google.com wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For a real world example see Google's MapsGL (http://support.google.com/maps/bin/answer.py?hl=enanswer=1630790) Apparently MapsGL uses 2 canvases and/or some DOM objects overlayed on top of each other. Dragging the mouse moves objects in all of those layers and they need to move simultaneously to have a good UX. You can imagine issues if a canvas is being rendering to from a worker. How would the user guarantee that changes from the worker are synchronized with changes to the DOM in the main thread? Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? Let me retract this. There is a way under the current API to solve sync which is to use drawImage to another canvas in the main thread and have the canvases being drawing to by the worker basically as offscreen canvases. It might not be the ideal solution since a drawImage call is an extra draw which is not cheap, especially if it's a large canvas. But, it does mean there's a solution for now without adding any extra api. Random thoughts *) Leave things the way they are and add another mechanism for syncing? In other words, by default things are not sync. Through some other API or settings the user can opt into getting synchronization *) Look at OpenGL swap groups as inspiration for an API? http://www.opengl.org/registry/specs/NV/wgl_swap_group.txt http://www.opengl.org/registry/specs/NV/glx_swap_group.txt *) Consider an 'oncommit' or 'onswap' event on 'Window'? The idea being you want to give the main thread a chance to update stuff (DOM elements) in response to a worker having called 'commit' on a canvas. Of course not sure how that would work if you have 2 workers each rendering to a different canvas. Note: I haven't thought through these issues at all and I've personally not had to deal with them but it seems clear from the MapsGL example that a solution will be needed for some subset of apps to have a good UX. I know for example there's game engine that has API to keep DOM elements located relative to game objects being rendered in a canvas to make it easy to give HTML based stats on the game objects as opposed to having to render the stats manually with canvas. On Fri, Nov 16, 2012 at 8:35 PM, Ian Hickson i...@hixie.ch wrote: On Fri, 16 Nov 2012, Charles Pritchard wrote: http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html Seems like we might use requestAnimationFrame in the main thread to postMessage to the worker as an alternative to using setInterval in workers for repaints. The idea in due course is to just expose rAF in workers. Please do read the e-mail above, which actually mentions that. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For a real world example see Google's MapsGL (http://support.google.com/maps/bin/answer.py?hl=enanswer=1630790) Apparently MapsGL uses 2 canvases and/or some DOM objects overlayed on top of each other. Dragging the mouse moves objects in all of those layers and they need to move simultaneously to have a good UX. You can imagine issues if a canvas is being rendering to from a worker. How would the user guarantee that changes from the worker are synchronized with changes to the DOM in the main thread? Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? Random thoughts *) Leave things the way they are and add another mechanism for syncing? In other words, by default things are not sync. Through some other API or settings the user can opt into getting synchronization *) Look at OpenGL swap groups as inspiration for an API? http://www.opengl.org/registry/specs/NV/wgl_swap_group.txt http://www.opengl.org/registry/specs/NV/glx_swap_group.txt *) Consider an 'oncommit' or 'onswap' event on 'Window'? The idea being you want to give the main thread a chance to update stuff (DOM elements) in response to a worker having called 'commit' on a canvas. Of course not sure how that would work if you have 2 workers each rendering to a different canvas. Note: I haven't thought through these issues at all and I've personally not had to deal with them but it seems clear from the MapsGL example that a solution will be needed for some subset of apps to have a good UX. I know for example there's game engine that has API to keep DOM elements located relative to game objects being rendered in a canvas to make it easy to give HTML based stats on the game objects as opposed to having to render the stats manually with canvas. On Fri, Nov 16, 2012 at 8:35 PM, Ian Hickson i...@hixie.ch wrote: On Fri, 16 Nov 2012, Charles Pritchard wrote: http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html Seems like we might use requestAnimationFrame in the main thread to postMessage to the worker as an alternative to using setInterval in workers for repaints. The idea in due course is to just expose rAF in workers. Please do read the e-mail above, which actually mentions that. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts I've now specced something like this; for details, see: http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
On Nov 16, 2012, at 1:50 PM, Ian Hickson i...@hixie.ch wrote: On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts I've now specced something like this; for details, see: http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html Seems like we might use requestAnimationFrame in the main thread to postMessage to the worker as an alternative to using setInterval in workers for repaints. -Charles
Re: exposing CANVAS or something like it to Web Workers
On Fri, 16 Nov 2012, Charles Pritchard wrote: http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html Seems like we might use requestAnimationFrame in the main thread to postMessage to the worker as an alternative to using setInterval in workers for repaints. The idea in due course is to just expose rAF in workers. Please do read the e-mail above, which actually mentions that. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: exposing CANVAS or something like it to Web Workers
That sounds like a reasonable approach to me. One thing missing is that Workers would need something like requestAnimationFrame. Any proposal that requires passing messages to the main thread to get something on the screen fails to satisfy the huge need for games to be able to get steady frame rates in the face of main-thread latency. Rob -- “You have heard that it was said, ‘Love your neighbor and hate your enemy.’ But I tell you, love your enemies and pray for those who persecute you, that you may be children of your Father in heaven. ... If you love those who love you, what reward will you get? Are not even the tax collectors doing that? And if you greet only your own people, what are you doing more than others? [Matthew 5:43-47]
Re: exposing CANVAS or something like it to Web Workers
So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas **) Create context in main page, pass to worker* Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? **) Create a context directly in a Worker (or anywhere for that matter)* As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic **) Create an offscreen canvas like object minus HTMLElement parts* * * Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell k...@google.com wrote: On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. That problem I have with this path is the cons mentioned above. It's a dead end. With real contexts you can do this webglcontext1.texImage2D(..., webglcontext2.canvas); and this context2d.drawImage(webgl.canvas, ...) But with a context created with new WebGLRenderingContext you can't do anything like that because size of the backbuffer is not defined. Eventually we'll want to support operation like that and we'll likely end up with something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that point this ability to go new WebGLRenderingContext will just be left over cruft. Also we'll have to define how to addEventListner for WebGLRenderContext for listening for lost context or for async context creation (yea, less important on workers). That stuff is currently not on WebGLRenderingContext. It seems like it should stay off if it. -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 1:11 PM, Gregg Tavares (勤) g...@google.com wrote: On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell k...@google.com wrote: On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. That problem I have with this path is the cons mentioned above. It's a dead end. With real contexts you can do this webglcontext1.texImage2D(..., webglcontext2.canvas); and this context2d.drawImage(webgl.canvas, ...) But with a context created with new WebGLRenderingContext you can't do anything like that because size of the backbuffer is not defined. Eventually we'll want to support operation like that and we'll likely end up with something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that point this ability to go new WebGLRenderingContext will just be left over cruft. Also we'll have to define how to addEventListner for WebGLRenderContext for listening for lost context or for async context creation (yea, less important on workers). That stuff is currently not on WebGLRenderingContext. It seems like it should stay off if it. These are all good points, and I agree that the ideal fix would be to use the Canvas element (or DrawingSurface, etc.) from a worker, rather than doing a WebGL-specific hack. Given how long discussions have been ongoing on this topic, though, I think this may just be too large a single step to take. Rather than add a constructor to WebGLRenderingContext that would need to remain there permanently, perhaps a factory method with a vendor prefix could be added (e.g. WebGLRenderingContext.webkitCreateOffscreenContext, WebGLRenderingContext.mozCreateOffscreenContext). The long-term plan could be to remove the factory method once a better solution is reached, as opposed to removing the vendor prefixes. This would at least allow some applications to explore the use of workers with WebGL, see whether this addresses any existing issues, and expose new issues. -Ken -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 2:30 PM, Gregg Tavares (勤) g...@google.com wrote:
Some Ideas
**) Create context in main page, pass to worker*
Transferring an active context from one thread to another (at the
implementation level) can be hard. It's much simpler to only create new
the contexts in the thread they'll be used, never allowing contexts to be
moved from thread to thread.
Here's a first-pass approach:
1: Add HTMLCanvasElement.getBackbuffer, which returns a lightweight
Backbuffer interface associated with the HTMLCanvasElement. This has only
one method, getContext, which is equivalent to canvas.getContext. (This is
exactly like the suggestion for making images available in threads.)
2: Backbuffer can be transferred from one thread to another, using the
transfer mechanism. On transfer, any WebGL contexts (and other context
types) in the thread associated with the same backbuffer are destroyed
(putting exactly what destroyed means aside for later). Additionally,
when a Backbuffer is transferred, the underlying backbuffer of the
HTMLCanvasElement is considered owned by the new thread. All instances of
Backbuffer for that HTMLCanvasElement, including newly-created ones, are
neutered unless the backbuffer is transferred back to the UI thread.
3: As long as the backbuffer of an HTMLCanvasElement is owned by a thread
other than the UI thread, calls to the following methods on the associated
HTMLCanvasElement raise an exception: getBackbuffer, getContext,
toBlob, toDataURL.
This takes effect synchronously, as soon as transfer happens (right when
you postMessage the Backbuffer somewhere else).
4: When a Worker receives a Backbuffer, it can call Backbuffer.getContext
to create a WebGLContext (with exactly the same rules as
HTMLCanvasElement.getContext). The results of rendering to that context
are visible in the associated HTMLCanvasElement in the main thread.
This ensures that only one thread can ever have an open context for any
backbuffer, and that pixel readback functions (toBlob and toDataURL) can't
expose the asynchronous nature of what's going on. The transfer mechanics
are rough and need refining.
For offscreen rendering in a thread, just allow constructing Backbuffer;
for example,
var backbuffer = new Backbuffer(1024, 768);
var ctx = backbuffer.getContext(webgl);
backbuffer.resize(1920, 1200);
Backbuffer.resize is equivalent to resizing a Canvas, and only available
for Backbuffers you create yourself, not for ones created via
HTMLCanvasElement.getBackBuffer.
One problem that I havn't attempted to solve here: when the
HTMLCanvasElement is resized in the UI thread, it results in changes to
drawingBufferWidth/drawingBufferHeight. This would expose asynchronous
behavior. Instead, it would probably need to apply the change (from the
thread's perspective) in a queued task, so you have to return to the event
loop for it to be applied.
If a thread is killed by the browser (eg. due to a CPU quota), the
backbuffers it owns are orphaned; you can no longer create contexts for
it. You need to create a new Canvas. This isn't great, but workers don't
really seem to try to make it possible to recover from this anyway.
interface CanvasBackbuffer {
object? getContext(DOMString contextId, any... args);
}
[Constructor(unsigned long width, unsigned long height)]
interface Backbuffer : CanvasBackbuffer {
void resize(unsigned long width, unsigned long height);
}
interface HTMLCanvasElement : HTMLElement {
CanvasBackbuffer getBackbuffer();
}
The Backbuffer ctor is available in workers.
Importantly, you can start by just implementing the Backbuffer
constructor. That would only allow the simpler case of offscreen rendering.
--
Glenn Maynard
exposing CANVAS or something like it to Web Workers
I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts Any thoughts?
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts Any thoughts? Have we gotten any further with use cases? See http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144 for an old use case thread that went nowhere. Or http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274 where you requested general DOM access which cannot be implemented everywhere. -- Anne — Opera Software http://annevankesteren.nl/ http://www.opera.com/
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 1:01 PM, Gregg Tavares (勤) wrote:
I'd like to work on exposing something like CANVAS to web workers.
Ideally how over it works I'd like to be able to
*) get a 2d context in a web worker
*) get a WebGL context in a web worker
*) download images in a web worker and the images with both 2d
contexts and WebGL contexts
Any thoughts?
As far as implementation, I'd love to be able to pass webkit's
document.getCSSCanvasContext('2d') around.
It seems like a safe place to experiment.
I can get a lot done with CanvasPattern as a transferable, without
needing to add Image (or video) into the worker context.
Notes:
1. getCSSCanvasContext is non-standard. It works with CSS image
-webkit-canvas.
2. I heard that a more generic -moz-element() paint server is supposed
to replace -webkit-canvas in time.
3. Passing the CSS canvas context would let me render off-frame and
update a canvas visible on the document automatically.
Canvas -should- have toBlob and a typed array buffer for ImageData.
They are both useful for passing image data back to the main frame.
From my experience with WebGL, I think it should be considered with
added care and lower priority.
There are stability, speed and memory issues. WebGL in workers seems to
augment 2d, are there other big benefits?
-Charles
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 1:16 PM, Anne van Kesteren ann...@annevk.nl wrote: On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts Any thoughts? Have we gotten any further with use cases? See http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144 for an old use case thread that went nowhere. Or http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274 where you requested general DOM access which cannot be implemented everywhere. One of the strongest use-cases we have found within mozilla is to enable compiling applications written in other languages and for other platforms to the web platform. For example emscripten [1] supports compiling C/C++/OpenGL programs to javascript + WebGL. Such programs generally use things like synchronous IO and have their own event loops, thus not returning to the platform event loop. So a good strategy for tools like emscripten is to create code which is intended to run in a worker. That way you don't have to return to the event loop, and you have access to things like full featured synchronous XHR (including support for .responseType) without degrading the user experience. However a big problem is that you can't do graphics inside a worker. It would be great to expose canvas since that is the graphics model these programs have today. [1] https://github.com/kripken/emscripten / Jonas
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 1:16 PM, Anne van Kesteren wrote: On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤)g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker *) get a WebGL context in a web worker *) download images in a web worker and the images with both 2d contexts and WebGL contexts Any thoughts? Have we gotten any further with use cases? See http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144 for an old use case thread that went nowhere. Or 1. Speeding up onmousemove-based drawing: In my drawing projects (based on mouse/pen input), we lose mouse events / pen pressure information when the main thread is busy rendering what the user is drawing. Processing the drawing commands off-thread would lighten the load. 2. Avoiding blocking during redrawing of complex scenes or pre-rendering of animations. With complex scenes, where we're repainting, we don't particularly want to block the main thread while a scene is loading. But, we'd also like to use as much horsepower as the user's machine can lend. A complex scene may block for a few seconds -- we can of course use green threading approaches, but that adds quite a bit of extra guess-work and does not fully exploit the user's machine for speed.
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. This would also require some equivalent to requestAnimationFrame in the worker thread. *) download images in a web worker and the images with both 2d contexts and WebGL contexts Don't forget to point people to the recent discussion on this, so the conversation doesn't reboot: https://www.khronos.org/webgl/public-mailing-list/archives/1205/msg00059.html In summary, the idea is to provide a much more basic interface for images, which would represent an image in an opaque, non-DOM way, to make it implementable in workers; you'd retrieve it with a method on HTMLImageElement after the image is completely available. It'd be structured clonable, to hand it to workers. There should also be an XHR2 responseType mode to retrieve an object like this directly, to allow loading images from workers (again without using the DOM), since workers shouldn't need to rely on a helper in the UI thread just to load textures. This seems like it would solve this sub-problem pretty well, at least. On Mon, May 14, 2012 at 3:16 PM, Anne van Kesteren ann...@annevk.nl wrote: Have we gotten any further with use cases? See http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144 for an old use case thread that went nowhere. Or http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274 where you requested general DOM access which cannot be implemented everywhere. The use case is being able to draw without blocking the UI thread. WebGL, like OpenGL, tries to be nonblocking by queuing draw operations, but it's impossible to do this in all cases: as soon as you call any method that returns data (eg. getError), WebGL has to synchronously flush the draw queue. This isn't as severe as synchronous XHR, but it may be done every frame--a 10ms delay at 60Hz is going to have visible effects on the UI. WebGL is also useful for computational tasks, which may perform much longer drawing operations. If you're doing this, it's even more important that it not block the UI thread. Compiling shaders can also take a long time for complex shaders. Since this is almost always followed by a synchronous call to check the result, it's currently hard to load shaders without affecting the browser UI. -- Glenn Maynard
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 6:42 PM, Jonas Sicking jo...@sicking.cc wrote: If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. Sort of. You want to disable the methods on Canvas--toDataURL, toBlob and getContext--but you don't want to disable inherited methods (you should still be able to access eg. parentNode). I think handling this on a method-by-method basis is fine, since there aren't many of them--just those three, I think. I still don't have much intuition about how to do this, but another thought: assuming you create a context in the worker which is attached to the canvas, you need to be able to create new contexts in the future if the worker dies, since workers can be killed by the browser at any time. A mechanism that says you can create a worker-context for a canvas, but then the canvas element is locked to that worker isn't good enough (but it's fine to say you can only have one worker associated with a canvas at a time). If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. Right. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! A tricky bit: you need to know which element to sync to, so the browser knows which monitor's vsync to use. According to [1] only WebKit's requestAnimationFrame actually takes an element. (That's surprising; this seems obvious. Are multi-monitor systems always vsync-locked to each other, which would make this not matter?) I mention this because this method would need to accept a context in lieu of an element (assuming whatever mechanism we land on it results in having a WebGLContext in the worker but no HTMLCanvasElement, whcih seems likely). This in mind, it would be good if all Canvas context types had a common base class, even if it's an empty interface. The return value of HTMLCanvasElement.getContext could be changed to it, instead of object. [1] https://developer.mozilla.org/en/DOM/window.requestAnimationFrame -- Glenn Maynard
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote:
On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote:
On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com
wrote:
I'd like to work on exposing something like CANVAS to web workers.
Ideally how over it works I'd like to be able to
*) get a 2d context in a web worker
I'd recommend not trying to tackle 2d and 3d contexts at once, and only
worrying about WebGL to start.
Another issue: rendering in a worker thread onto a canvas which is
displayed
in the main thread. This needs to be solved in a way that doesn't cause
the
asynchronous nature of what's happening to be visible to scripts.
toDataURL
and toBlob would probably need to be prohibited on the canvas element.
I'm
not sure what the actual API would look like.
If/when we do this, I think it should be done in such a way that the
main window can't access the canvas object at all. Similar to what
happens when an ArrayBuffer is transferred to a Worker using
structured cloning. Once a canvas is transferred to a Worker, any
access to it should throw or return null/0/. If you want to transfer
pixel data to the main thread, it seems less racy to do that by
getting the pixel data in the Worker which owns the canvas and then
transfer that to the main thread using postMessage.
How about separating the canvasy parts of canvas from canvas and the imagy
parts of image from image.
In other words, Imagine canvas is implemented like this
class Canvas : public HTMLElement {
private:
CanvasSurface* m_surface; // everything about canvas that is not
HTMLElement
};
And that Image is similarly implemented as
class Image : public HTMLElement {
private:
Picture* m_picture; // everything about Image that is not HTMLElement
}
now imagine you can instantiate inner implementation of these things. The
parts that are not HTMLElement
var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext(2d);
var pic = new Picture;
pic.src = http://someplace.com/someimage.jpg;;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}
Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.
Would that work? What problems would that have?
This would also require some
equivalent to requestAnimationFrame in the worker thread.
Agreed!
/ Jonas
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 7:56 PM, Glenn Maynard wrote: A tricky bit: you need to know which element to sync to, so the browser knows which monitor's vsync to use. According to [1] only WebKit's requestAnimationFrame actually takes an element. (That's surprising; this seems obvious. Does WebKit actually use the element to determine vsync? How do they handle cases when the element spans monitors? As far as I know WebKit's implementation uses the element to optimize out callbacks when the element is not visible, but that's it. Note that Gecko, for example, does not tie requestAnimationFrame callbacks to vsync. I can't speak for other UAs. I mention this because this method would need to accept a context in lieu of an element What would the context be used for? -Boris
Re: exposing CANVAS or something like it to Web Workers
On May 14, 2012, at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh? I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context. We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too. I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered. -Charles
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 8:03 PM, Gregg Tavares (勤) wrote:
var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext(2d);
var pic = new Picture;
pic.src = http://someplace.com/someimage.jpg;;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}
Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.
Would that work? What problems would that have?
Two things that come to mind immediately:
1) Various canvas 2d context methods depend on the styles of the canvas
to define how they actually behave. Clearly this would need some sort
of changes for Workers anyway; the question is what those changes would
need to be.
2) How (or maybe whether) this should work with pic.src =
http://someplace.com/someimage.svg; is an interesting question.
3) This leaves open the question of how to get data from the worker
back to the main thread to paint there.
#1 is an issue with any proposal that puts a 2d context in a worker.
#2 is only an issue if we do the image loads in workers somehow; if we
wanted to just rasterize the SVG and pass the resulting data to the
worker there would be no problem, obviously.
For #3, being able to transfer a canvas context from the page to a
worker but leave it painting to the canvas may still be desirable...
-Boris
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 5:03 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 5/14/12 7:56 PM, Glenn Maynard wrote: A tricky bit: you need to know which element to sync to, so the browser knows which monitor's vsync to use. According to [1] only WebKit's requestAnimationFrame actually takes an element. (That's surprising; this seems obvious. Does WebKit actually use the element to determine vsync? How do they handle cases when the element spans monitors? As far as I know WebKit's implementation uses the element to optimize out callbacks when the element is not visible, but that's it. The element isn't used for anything currently in WebKit. Which vsync is determined by the monitor the tab/window/whatever lands on. When this spans monitors, something random happens (there aren't many good options). Note that Gecko, for example, does not tie requestAnimationFrame callbacks to vsync. I can't speak for other UAs. I think you'll want to. - James I mention this because this method would need to accept a context in lieu of an element What would the context be used for? -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 8:21 PM, Charles Pritchard wrote: SVG and animated gif would render the same as it would in an Image that has not been added to the dom or is otherwise display: none. I'm not sure that would be workable in a worker for SVG, for the same reasons that responseXML is not available in workers: it involves DOM elements. Unless the SVG rasterization happens on the main thread under the hood and the raster data is then sent over to the worker. This might have ... surprising performance characteristics. -Boris
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 5:12 PM, Boris Zbarsky bzbar...@mit.edu wrote:
On 5/14/12 8:03 PM, Gregg Tavares (勤) wrote:
var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext(2d)**;
var pic = new Picture;
pic.src =
http://someplace.com/**someimage.jpghttp://someplace.com/someimage.jpg
;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}
Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.
Would that work? What problems would that have?
Two things that come to mind immediately:
1) Various canvas 2d context methods depend on the styles of the canvas
to define how they actually behave. Clearly this would need some sort of
changes for Workers anyway; the question is what those changes would need
to be.
Which methods are these?
2) How (or maybe whether) this should work with pic.src =
http://someplace.com/**someimage.svg http://someplace.com/someimage.svg
is an interesting question.
3) This leaves open the question of how to get data from the worker back
to the main thread to paint there.
I feel that's a somewhat orthogonal issue to getting a Canvas or
CanvasSurface in a worker. At least for the WebGL case, WebGL already
defines ways to share data safely between contexts so that use case is
already covered.
For a 2d context I can certainly imagine lots of ways. Some fast, some slow
slow) pass the result of toDataURL back
slow) pass an ImageData.data back (transfer of ownership)
fast) some how use a CanvasSurface as the source of a video tag
maybe there are some other ideas? But, even if we didn't resolve this, it's
still massively useful, at least for WebGL, to be able to instantiate a
WebGL context in a worker.
#1 is an issue with any proposal that puts a 2d context in a worker.
#2 is only an issue if we do the image loads in workers somehow; if we
wanted to just rasterize the SVG and pass the resulting data to the worker
there would be no problem, obviously.
For #3, being able to transfer a canvas context from the page to a worker
but leave it painting to the canvas may still be desirable...
-Boris
Re: exposing CANVAS or something like it to Web Workers
On May 14, 2012, at 5:50 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 5/14/12 8:21 PM, Charles Pritchard wrote: SVG and animated gif would render the same as it would in an Image that has not been added to the dom or is otherwise display: none. I'm not sure that would be workable in a worker for SVG, for the same reasons that responseXML is not available in workers: it involves DOM elements. Unless the SVG rasterization happens on the main thread under the hood and the raster data is then sent over to the worker. This might have ... surprising performance characteristics. I agree... Can we get this off the main thread? Svg via image is not quite the same as svg via HTMLDocument (I guess I mean, embedded). Afaik, svg via image does not have any script controls but it does have xsl things. I've never tried to abuse the distinction via embedded blob Uris and such. Put in other words: img src=image.svg may use an entirely different implementation than svg in an HTML document.
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote: 1) Various canvas 2d context methods depend on the styles of the canvas to define how they actually behave. Clearly this would need some sort of changes for Workers anyway; the question is what those changes would need to be. Which methods are these? Anything involving setting color (e.g. the strokeStyle setter, the fillStyle setter), due to currentColor. Anything involving text because font styles come from the element or document. Those are the ones that come to mind offhand, but I haven't looked at the various recent additions to the 2d context closely. -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 8:58 PM, Charles Pritchard wrote: I agree... Can we get this off the main thread? Maybe. It would be pretty nontrivial in Gecko; last I looked it would be pretty painful in WebKit too. Can't speak for other UAs. Svg via image is not quite the same as svg via HTMLDocument (I guess I mean, embedded). Sure, but that doesn't matter. Afaik, svg via image does not have any script controls but it does have xsl things. And CSS. Put in other words:img src=image.svg may use an entirely different implementation thansvg in an HTML document. It _may_, but it would likely be pretty buggy. And at least in Gecko, I don't think we're willing to duplicate enough of the rendering engine to do this. -Boris
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote: 1) Various canvas 2d context methods depend on the styles of the canvas to define how they actually behave. Clearly this would need some sort of changes for Workers anyway; the question is what those changes would need to be. Which methods are these? Anything involving setting color (e.g. the strokeStyle setter, the fillStyle setter), due to currentColor. Anything involving text because font styles come from the element or document. Good to know. That doesn't sound like a showstopper though. If a canvas/CanvasSurface is available in workers the simplest solution would just be that currentColor defaults to something black? or nothing . Pick one. Those are the ones that come to mind offhand, but I haven't looked at the various recent additions to the 2d context closely. -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 6:08 PM, Boris Zbarsky wrote: On 5/14/12 8:58 PM, Charles Pritchard wrote: I agree... Can we get this off the main thread? Maybe. It would be pretty nontrivial in Gecko; last I looked it would be pretty painful in WebKit too. Can't speak for other UAs. Can it be pumped through what's essentially an iframe on a null origin? I don't know enough about browser internals to help on this one. Yes, loading an SVG image is a heavy call. For 90% of the SVG content out there, it'd probably be faster to parse and draw the SVG via Canvas and JS. Still it's a hell of a lot nicer to load it via img tag. I'd just assumed that img src=pic.svg / was loaded off-thread / async much like img calls. There's nothing that gets carried from the document to the img other than the width/height, which I believe is carried through. Which is a good thing, of course. -Charles
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 9:15 PM, Charles Pritchard wrote: On 5/14/2012 6:08 PM, Boris Zbarsky wrote: On 5/14/12 8:58 PM, Charles Pritchard wrote: I agree... Can we get this off the main thread? Maybe. It would be pretty nontrivial in Gecko; last I looked it would be pretty painful in WebKit too. Can't speak for other UAs. Can it be pumped through what's essentially an iframe on a null origin? The issue at least in Gecko is that the DOM uses various effectively-global variables (shared caches of various sorts, non-threadsafe services, etc). The origin is irrelevant. I'd just assumed that img src=pic.svg / was loaded off-thread / async much like img calls. The _loading_ is async. In Gecko the parsing of the XML, DOM construction, style computations, layout are on the main thread, like they are for HTML. It's the DOM construction, style computations, layout parts that are not so easy to push off to a background thread... -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 6:14 PM, Gregg Tavares (勤) wrote: On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky bzbar...@mit.edu mailto:bzbar...@mit.edu wrote: On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote: 1) Various canvas 2d context methods depend on the styles of the canvas to define how they actually behave. Clearly this would need some sort of changes for Workers anyway; the question is what those changes would need to be. Which methods are these? Anything involving setting color (e.g. the strokeStyle setter, the fillStyle setter), due to currentColor. Anything involving text because font styles come from the element or document. Good to know. That doesn't sound like a showstopper though. If a canvas/CanvasSurface is available in workers the simplest solution would just be that currentColor defaults to something black? or nothing . Pick one. Font is still a little tricky from loading fonts via CSS. Font is tricky anyway, though, so it wouldn't be that much of a step backward. Would we assume that if a font is available from the parent context it's going to be available to the worker? currentColor would just default to black, as we're not talking about a color inherited from the DOM. Those are the ones that come to mind offhand, but I haven't looked at the various recent additions to the 2d context closely. The recent additions are more proposals than additions. They're proposals from Tab and Ian and not yet implemented. They revolve around a Path object, which we've not yet discussed. Otherwise, they include lightweight nodes and DOM fallback content, which isn't relevant to workers. My thinking is the same as yours: fillStyle/strokeStyle and font are the ones that come to mind. Pattern and Gradient are items that could conceivably be cloned and/or shared. They can both be as efficient as sending Blob via postMessage. I think Gregg was just settling on not-sending Canvas over postMessage but rather creating the instance inside of each Worker.
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard ch...@jumis.com wrote: On May 14, 2012, at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh? I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context. We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too. I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered. I was saying that I think we should transfer the context from the main thread to the worker thread. The worker thread should then be able to use the context to draw directly to the screen without interacting with the main thread. You should generally not need to transfer pixel data. But I think it should be possible to grab pixel data on whichever thread is currently owning a context. This will implicitly make it possible to transfer pixel data if anyone wants to do it. / Jonas
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 6:23 PM, Jonas Sicking wrote: This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh? I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context. We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too. I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered. I was saying that I think we should transfer the context from the main thread to the worker thread. The worker thread should then be able to use the context to draw directly to the screen without interacting with the main thread. You should generally not need to transfer pixel data. But I think it should be possible to grab pixel data on whichever thread is currently owning a context. This will implicitly make it possible to transfer pixel data if anyone wants to do it. OK, that's the same concept as I was hoping for with document.getCSSCanvasContext. Mozilla is a bit ahead of webkit, going with element(any-element) -- webkit has -webkit-canvas(css-canvas-id). It's a very different approach. I'd still go ahead and just have requestAnimationFrame pump events from the main frame. There are so many things that could block the main frame to where an rAF repaint just isn't necessary from the worker-end. It may also keep some animations continuing. I'm not positive, but something like window.prompt may block execution on the main thread, but if rAF were running on the worker, its repaints would still happen. Generally, while the main thread is blocked, repainting on it is not going to be that useful; we're shuffling the repaints off so they aren't the cause of blocking. Those are the two sides of it that I see... but it's still a very different proposal than the idea of just having non-transferable canvas contexts. ... Boris brought up some good points about possible blocking in implementations with loading SVG images in the worker thread via Picture (or a chopped down Image). At present, those sound more like implementation-issues, not particularly issues in the validity of a spec. http://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0744.html They do sound a little unfortunate, but I'd sooner trade some possible blocking between a Worker and the main thread, than having to carry a full SVG parser written in JS in my worker threads. As an author, I've always had to take precautions for SVG and decide whether I want to render or give myself up to the browser implementation. -Charles
Re: exposing CANVAS or something like it to Web Workers
On Mon, May 14, 2012 at 6:23 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard ch...@jumis.com wrote: On May 14, 2012, at 4:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) g...@google.com wrote: I'd like to work on exposing something like CANVAS to web workers. Ideally how over it works I'd like to be able to *) get a 2d context in a web worker I'd recommend not trying to tackle 2d and 3d contexts at once, and only worrying about WebGL to start. Another issue: rendering in a worker thread onto a canvas which is displayed in the main thread. This needs to be solved in a way that doesn't cause the asynchronous nature of what's happening to be visible to scripts. toDataURL and toBlob would probably need to be prohibited on the canvas element. I'm not sure what the actual API would look like. If/when we do this, I think it should be done in such a way that the main window can't access the canvas object at all. Similar to what happens when an ArrayBuffer is transferred to a Worker using structured cloning. Once a canvas is transferred to a Worker, any access to it should throw or return null/0/. If you want to transfer pixel data to the main thread, it seems less racy to do that by getting the pixel data in the Worker which owns the canvas and then transfer that to the main thread using postMessage. This would also require some equivalent to requestAnimationFrame in the worker thread. Agreed! / Jonas I'm a bit lost-- wouldn't we just postMessage from the document over to the web worker when we want a refresh? I agree that we ought to be transferring pixel data not Canvas contexts; with the possible exception of CSS context. We could just create new canvas instances inside the worker thread. I'd still prefer to clone CanvasPattern as a means of transferring paint over to the worker, though sending pixel data would work too. I heard Picture come up-- it seems like that object might have additional semantics for high resolution alternatives that may need to be considered. I was saying that I think we should transfer the context from the main thread to the worker thread. The worker thread should then be able to use the context to draw directly to the screen without interacting with the main thread. That's a great goal but is transferring the right way to do it? context.canvas is currently a reference to the canvas from which the context as created. Will that just go NULL on transfer? Will it change to some other object that is not a Canvas but acts similarly? What happens to the context on the main page? All of it's methods become no-ops and it's properties become non-live? How will things synchronize if I call canvas.toDataURL after passing the context to a work? Exception? nothing? It seems hard to define how that would synchronize If there was a way to allow a worker to generate a media stream and one of those ways involves an in worker canvas/canvassurface, then it seems like none of the above questions have to be answered. Instead the worker creates the surface, and then through some other API attached it to a video tag. You can call video.toDataURL, that's well defined (I assume) Synchronization is also somewhat defined in that direction. Doesn't that seem like the path of least resistance? You should generally not need to transfer pixel data. But I think it should be possible to grab pixel data on whichever thread is currently owning a context. This will implicitly make it possible to transfer pixel data if anyone wants to do it. / Jonas
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 6:07 PM, Boris Zbarsky wrote:
On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:
1) Various canvas 2d context methods depend on the styles of the
canvas to define how they actually behave. Clearly this would need
some sort of changes for Workers anyway; the question is what those
changes would need to be.
Which methods are these?
Anything involving setting color (e.g. the strokeStyle setter, the
fillStyle setter), due to currentColor. Anything involving text
because font styles come from the element or document.
Those are the ones that come to mind offhand, but I haven't looked at
the various recent additions to the 2d context closely.
What would web fonts do in this situation, in Mozilla? If I've confirmed
that a font is loaded in the main thread, would it be available to a
worker for use in rendering?
Some implementations of Canvas font methods have been buggy, but
dropping fillText/strokeText altogether would be a loss and may look
strange in the specs.
I've seen font not working correctly with other canvas state variables
in WebKit + Chrome; so it's not as though font is fully supported in the
main thread, currently.
It may be easier to postpone Image / Picture semantics in workers. I
think patterns could still be salvaged from createPattern.
It sounds like there are practical issues in current implementations.
When authoring,
With a picture, I can still xhr request it in the worker, then I send it
to the main thread, and load it in the main thread, then send it back to
the worker.
As an author, if I'm loading images I expect them to be a normal part of
the page's load time and responsiveness I can still off-load my
js-heavy work onto workers.
In Canvas 2d anyway, I can easily get by without picture in workers: (a
= new Picture()).src = ; a.onload { ctx.drawImage(a,0,0); }; if I
can use fillRect with pattern.
Otherwise, I'd have to do the extra steps of pushing pixel arrays back
and forth.
All of the extra work is still worth it to get work done in workers, in
cases where this level of work is needed. It's just a few extra lines of
JS code (every time).
-Charles
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 10:00 PM, Charles Pritchard wrote: What would web fonts do in this situation, in Mozilla? Probably cry. ;) If I've confirmed that a font is loaded in the main thread, would it be available to a worker for use in rendering? Not without some pretty serious reworking. Which might need to happen. Of course basic text layout would also not be available without some serious reworking (e.g. making the textrun cache threadsafe or creating per-thread textrun caches or something), so the question of web fonts is somewhat academic at the moment. -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 7:09 PM, Boris Zbarsky wrote: On 5/14/12 10:00 PM, Charles Pritchard wrote: What would web fonts do in this situation, in Mozilla? Probably cry. ;) If I've confirmed that a font is loaded in the main thread, would it be available to a worker for use in rendering? Not without some pretty serious reworking. Which might need to happen. Of course basic text layout would also not be available without some serious reworking (e.g. making the textrun cache threadsafe or creating per-thread textrun caches or something), so the question of web fonts is somewhat academic at the moment. I meant solely for Canvas 2d. I can live with staying away from fillText/strokeText on a worker thread if I'm loading fonts. It's been broken on the main thread anyway, requiring intermediate Canvas surfaces for some operations. ... SVG image and drawImage is mixed anyway; we can't transfer the data between threads as drawImage SVG will usually flag the Canvas as dirty in implementations. We could just use Canvas 2d to handle pattern uploads for WebGL. Seems like that'd work without requiring fancy footwork to gain Picture/Image support in the worker. SVG images would get fixed some other day. -Charles
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 10:18 PM, Charles Pritchard wrote: On 5/14/2012 7:09 PM, Boris Zbarsky wrote: On 5/14/12 10:00 PM, Charles Pritchard wrote: What would web fonts do in this situation, in Mozilla? Probably cry. ;) If I've confirmed that a font is loaded in the main thread, would it be available to a worker for use in rendering? Not without some pretty serious reworking. Which might need to happen. Of course basic text layout would also not be available without some serious reworking (e.g. making the textrun cache threadsafe or creating per-thread textrun caches or something), so the question of web fonts is somewhat academic at the moment. I meant solely for Canvas 2d. Yes, I understand that. Canvas 2d text still needs to be able to do things like font fallback, shaping, bidi, etc, etc, etc. last I checked. -Boris
Re: exposing CANVAS or something like it to Web Workers
On 5/14/2012 7:24 PM, Boris Zbarsky wrote: On 5/14/12 10:18 PM, Charles Pritchard wrote: On 5/14/2012 7:09 PM, Boris Zbarsky wrote: On 5/14/12 10:00 PM, Charles Pritchard wrote: What would web fonts do in this situation, in Mozilla? Probably cry. ;) If I've confirmed that a font is loaded in the main thread, would it be available to a worker for use in rendering? Not without some pretty serious reworking. Which might need to happen. Of course basic text layout would also not be available without some serious reworking (e.g. making the textrun cache threadsafe or creating per-thread textrun caches or something), so the question of web fonts is somewhat academic at the moment. I meant solely for Canvas 2d. Yes, I understand that. Canvas 2d text still needs to be able to do things like font fallback, shaping, bidi, etc, etc, etc. last I checked. Oh, the rendering isn't thread safe either? Yes, Canvas 2d text does [is supposed to] use all of those items. Well, I'll give up strokeText/fillText entirely in workers if it'll get me the goods faster. For a11y, I'm going to need to track my text in the main thread anyway. I can pre-render there if need be. -Charles
Re: exposing CANVAS or something like it to Web Workers
On 5/14/12 10:32 PM, Charles Pritchard wrote: Yes, I understand that. Canvas 2d text still needs to be able to do things like font fallback, shaping, bidi, etc, etc, etc. last I checked. Oh, the rendering isn't thread safe either? Right. The rendering involves access to style data structures that are not threadsafe. It involves caching shaped textruns and such; that cache is not threadsafe. The actual shaper may or may not be threadsafe. The bidi code is not threadsafe. These are all solvable issues... on some timescale. ;) -Boris
