Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

Chrome extensions have a background.html capability with a full dom; I've used 
those to prototype the concept. I've also used webkit's CSS fill 
(-webkit-canvas) in the mix.

A good deal of experimentation can be done there, prior to hacking up C++.



On Feb 20, 2013, at 11:43 AM, Kenneth Russell  wrote:

> On Fri, Feb 8, 2013 at 9:14 AM, Travis Leithead
>  wrote:
>>>> What would be the advantage? If you wanted to keep dom elements in sync
>>>> with the canvas you'd still have to post something from the worker back to
>>>> the main thread so the main thread would know to pop.
>> 
>> 
>> 
>> Well, it's not a fleshed out proposal by any stretch, but you could imagine
>> that an event could be used to signal that new frames were ready from the
>> producer—then the main thread would know to pop.
> 
> It sounds like this approach would require a bunch of new concepts --
> a remote worker context, events signalled through it, etc. It would
> still be necessary to postMessage from the main thread back to the
> worker for flow control. Flow control is definitely necessary -- the
> producer can't just produce frames without any feedback about when
> they're actually consumed. We've had problems in Chrome's graphics
> pipeline in the past where lack of flow control led to slow and
> inconsistent frame rates.
> 
> I'm excited about Gregg's proposal because it solves a lot of use
> cases that aren't currently addressed by CanvasProxy, using a couple
> of simple primitives that build upon others already in the platform
> (cross-document messaging and Transferables).
> 
> How can we move forward so that user agents can experimentally
> implement these APIs? Ideally we'd prototype and experiment with them
> in some medium-sized applications before finalizing any specs in this
> area.
> 
> Thanks,
> 
> -Ken
> 
> 
>> From: Gregg Tavares [mailto:g...@google.com]
>> Sent: Friday, February 8, 2013 3:14 AM
>> To: Travis Leithead
>> Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG
>> 
>> 
>> Subject: Re: exposing CANVAS or something like it to Web Workers
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead
>>  wrote:
>> 
>> Having thought about this before, I wonder why we don’t use a
>> producer/consumer model rather than a transfer of canvas ownership model?
>> 
>> 
>> 
>> A completely orthogonal idea (just my rough 2c after reading Gregg’s
>> proposal), is to have an internal frame buffer accessible via a WorkerCanvas
>> API which supports some set of canvas 2d/3d APIs as appropriate, and can
>> “push” a completed frame onto a stack in the internal frame buffer. Thus the
>> worker can produce frames as fast as desired.
>> 
>> 
>> 
>> On the document side, canvas gets a 3rd kind of context—a
>> WorkerRemoteContext, which just offers the “pop” API to pop a frame from the
>> internal frame buffer into the canvas.
>> 
>> 
>> 
>> Then you just add some basic signaling events on both ends of the frame
>> buffer and you’re good (as far as synchronizing the worker with the
>> document). The producer (in the worker) is free to produce multiple frames
>> in advance (if desired), while the consumer is able to pop frames when
>> available. You could even have the framebuffer depth configurable.
>> 
>> 
>> 
>> What would be the advantage? If you wanted to keep dom elements in sync with
>> the canvas you'd still have to post something from the worker back to the
>> main thread so the main thread would know to pop.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> From: Gregg Tavares [mailto:g...@google.com]
>> Sent: Thursday, February 7, 2013 2:25 PM
>> To: Ian Hickson
>> Cc: Charles Pritchard; Web Applications Working Group WG
>> Subject: Re: exposing CANVAS or something like it to Web Workers
>> 
>> 
>> 
>> I put up a new proposal for canvas in workers
>> 
>> 
>> 
>> http://wiki.whatwg.org/wiki/CanvasInWorkers
>> 
>> 
>> 
>> Please take a look.
>> 
>> 
>> 
>> This proposal comes from offline discussions with representatives from the
>> various browsers as well as input from the Google Maps team. I can post a
>> summary here if you'd like but it might be easier to read the wiki
>> 
>> 
>> 
>> Looking forward to feedback.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
&g

Re: exposing CANVAS or something like it to Web Workers

[Kenneth Russell]

On Fri, Feb 8, 2013 at 9:14 AM, Travis Leithead
 wrote:
>>> What would be the advantage? If you wanted to keep dom elements in sync
>>> with the canvas you'd still have to post something from the worker back to
>>> the main thread so the main thread would know to pop.
>
>
>
> Well, it's not a fleshed out proposal by any stretch, but you could imagine
> that an event could be used to signal that new frames were ready from the
> producer—then the main thread would know to pop.

It sounds like this approach would require a bunch of new concepts --
a remote worker context, events signalled through it, etc. It would
still be necessary to postMessage from the main thread back to the
worker for flow control. Flow control is definitely necessary -- the
producer can't just produce frames without any feedback about when
they're actually consumed. We've had problems in Chrome's graphics
pipeline in the past where lack of flow control led to slow and
inconsistent frame rates.

I'm excited about Gregg's proposal because it solves a lot of use
cases that aren't currently addressed by CanvasProxy, using a couple
of simple primitives that build upon others already in the platform
(cross-document messaging and Transferables).

How can we move forward so that user agents can experimentally
implement these APIs? Ideally we'd prototype and experiment with them
in some medium-sized applications before finalizing any specs in this
area.

Thanks,

-Ken


> From: Gregg Tavares [mailto:g...@google.com]
> Sent: Friday, February 8, 2013 3:14 AM
> To: Travis Leithead
> Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG
>
>
> Subject: Re: exposing CANVAS or something like it to Web Workers
>
>
>
>
>
>
>
> On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead
>  wrote:
>
> Having thought about this before, I wonder why we don’t use a
> producer/consumer model rather than a transfer of canvas ownership model?
>
>
>
> A completely orthogonal idea (just my rough 2c after reading Gregg’s
> proposal), is to have an internal frame buffer accessible via a WorkerCanvas
> API which supports some set of canvas 2d/3d APIs as appropriate, and can
> “push” a completed frame onto a stack in the internal frame buffer. Thus the
> worker can produce frames as fast as desired.
>
>
>
> On the document side, canvas gets a 3rd kind of context—a
> WorkerRemoteContext, which just offers the “pop” API to pop a frame from the
> internal frame buffer into the canvas.
>
>
>
> Then you just add some basic signaling events on both ends of the frame
> buffer and you’re good (as far as synchronizing the worker with the
> document). The producer (in the worker) is free to produce multiple frames
> in advance (if desired), while the consumer is able to pop frames when
> available. You could even have the framebuffer depth configurable.
>
>
>
> What would be the advantage? If you wanted to keep dom elements in sync with
> the canvas you'd still have to post something from the worker back to the
> main thread so the main thread would know to pop.
>
>
>
>
>
>
>
> From: Gregg Tavares [mailto:g...@google.com]
> Sent: Thursday, February 7, 2013 2:25 PM
> To: Ian Hickson
> Cc: Charles Pritchard; Web Applications Working Group WG
> Subject: Re: exposing CANVAS or something like it to Web Workers
>
>
>
> I put up a new proposal for canvas in workers
>
>
>
> http://wiki.whatwg.org/wiki/CanvasInWorkers
>
>
>
> Please take a look.
>
>
>
> This proposal comes from offline discussions with representatives from the
> various browsers as well as input from the Google Maps team. I can post a
> summary here if you'd like but it might be easier to read the wiki
>
>
>
> Looking forward to feedback.
>
>
>
>
>
>
>
> On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson  wrote:
>
> On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote:
>>
>> Another issue as come up and that is one of being able
>> to synchronize updates of a canvas in
>> worker with changes in the main page.
>
> For 2D, the intended solution is to just ship the ImageBitamp from the
> worker canvas to the main thread via a MessagePort and then render it on
> the canvas at the appropriate time.
>
> I don't know how you would do it for WebGL.
>
>
>
>> Similarly, let's say you have 2 canvases and are rendering to both in a
>> worker.  Does
>>
>>context1.commit();
>>context2.commit();
>>
>> guarantee you'll see both commits together?
>
> No, unfortunately not. There's no synchronisation between workers and the
> main thread (b

RE: exposing CANVAS or something like it to Web Workers

[Travis Leithead]

>> What would be the advantage? If you wanted to keep dom elements in sync with 
>> the canvas you'd still have to post something from the worker back to the 
>> main thread so the main thread would know to pop.

Well, it's not a fleshed out proposal by any stretch, but you could imagine 
that an event could be used to signal that new frames were ready from the 
producer—then the main thread would know to pop.

From: Gregg Tavares [mailto:g...@google.com]
Sent: Friday, February 8, 2013 3:14 AM
To: Travis Leithead
Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG
Subject: Re: exposing CANVAS or something like it to Web Workers



On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead 
mailto:travis.leith...@microsoft.com>> wrote:
Having thought about this before, I wonder why we don’t use a producer/consumer 
model rather than a transfer of canvas ownership model?

A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), 
is to have an internal frame buffer accessible via a WorkerCanvas API which 
supports some set of canvas 2d/3d APIs as appropriate, and can “push” a 
completed frame onto a stack in the internal frame buffer. Thus the worker can 
produce frames as fast as desired.

On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, 
which just offers the “pop” API to pop a frame from the internal frame buffer 
into the canvas.

Then you just add some basic signaling events on both ends of the frame buffer 
and you’re good (as far as synchronizing the worker with the document). The 
producer (in the worker) is free to produce multiple frames in advance (if 
desired), while the consumer is able to pop frames when available. You could 
even have the framebuffer depth configurable.

What would be the advantage? If you wanted to keep dom elements in sync with 
the canvas you'd still have to post something from the worker back to the main 
thread so the main thread would know to pop.



From: Gregg Tavares [mailto:g...@google.com<mailto:g...@google.com>]
Sent: Thursday, February 7, 2013 2:25 PM
To: Ian Hickson
Cc: Charles Pritchard; Web Applications Working Group WG
Subject: Re: exposing CANVAS or something like it to Web Workers

I put up a new proposal for canvas in workers

http://wiki.whatwg.org/wiki/CanvasInWorkers

Please take a look.

This proposal comes from offline discussions with representatives from the 
various browsers as well as input from the Google Maps team. I can post a 
summary here if you'd like but it might be easier to read the wiki

Looking forward to feedback.



On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson 
mailto:i...@hixie.ch>> wrote:
On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote:
>
> Another issue as come up and that is one of being able
> to synchronize updates of a canvas in
> worker with changes in the main page.
For 2D, the intended solution is to just ship the ImageBitamp from the
worker canvas to the main thread via a MessagePort and then render it on
the canvas at the appropriate time.

I don't know how you would do it for WebGL.


> Similarly, let's say you have 2 canvases and are rendering to both in a
> worker.  Does
>
>context1.commit();
>context2.commit();
>
> guarantee you'll see both commits together?
No, unfortunately not. There's no synchronisation between workers and the
main thread (by design, to prevent any possibility of deadlocks), and
there's not currently a batching API.

However, if this becomes a common problem (which we can determine by
seeing if we get bugs complaining about different parts of apps/games
seeming to slide around or generally be slightly out of sync, or if we see
a lot of authors shunting multiple ImageBitmap objects across MessagePort
channels) we can always add an explicit batching API to make this kind of
thing easy.

Note that in theory, for 2D at least, shunting ImageBitmaps across threads
can be as efficient as commit().

--
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: exposing CANVAS or something like it to Web Workers

[Gregg Tavares]

On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead <
travis.leith...@microsoft.com> wrote:

>  Having thought about this before, I wonder why we don’t use a
> producer/consumer model rather than a transfer of canvas ownership model?
> 
>
> ** **
>
> A completely orthogonal idea (just my rough 2c after reading Gregg’s
> proposal), is to have an internal frame buffer accessible via a
> WorkerCanvas API which supports some set of canvas 2d/3d APIs as
> appropriate, and can “push” a completed frame onto a stack in the internal
> frame buffer. Thus the worker can produce frames as fast as desired.
>
> ** **
>
> On the document side, canvas gets a 3rd kind of context—a
> WorkerRemoteContext, which just offers the “pop” API to pop a frame from
> the internal frame buffer into the canvas.
>
> ** **
>
> Then you just add some basic signaling events on both ends of the frame
> buffer and you’re good (as far as synchronizing the worker with the
> document). The producer (in the worker) is free to produce multiple frames
> in advance (if desired), while the consumer is able to pop frames when
> available. You could even have the framebuffer depth configurable.
>

What would be the advantage? If you wanted to keep dom elements in sync
with the canvas you'd still have to post something from the worker back to
the main thread so the main thread would know to pop.



> 
>
> ** **
>
> *From:* Gregg Tavares [mailto:g...@google.com]
> *Sent:* Thursday, February 7, 2013 2:25 PM
> *To:* Ian Hickson
> *Cc:* Charles Pritchard; Web Applications Working Group WG
> *Subject:* Re: exposing CANVAS or something like it to Web Workers
>
> ** **
>
> I put up a new proposal for canvas in workers
>
> ** **
>
> http://wiki.whatwg.org/wiki/CanvasInWorkers
>
> ** **
>
> Please take a look. 
>
> ** **
>
> This proposal comes from offline discussions with representatives from the
> various browsers as well as input from the Google Maps team. I can post a
> summary here if you'd like but it might be easier to read the wiki
>
> ** **
>
> Looking forward to feedback.
>
> ** **
>
> ** **
>
> ** **
>
> On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson  wrote:
>
>  On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote:
> >
> > Another issue as come up and that is one of being able
> > to synchronize updates of a canvas in
> > worker with changes in the main page.
>
> For 2D, the intended solution is to just ship the ImageBitamp from the
> worker canvas to the main thread via a MessagePort and then render it on
> the canvas at the appropriate time.
>
> I don't know how you would do it for WebGL.
>
>
>
> > Similarly, let's say you have 2 canvases and are rendering to both in a
> > worker.  Does
> >
> >context1.commit();
> >context2.commit();
> >
> > guarantee you'll see both commits together?
>
> No, unfortunately not. There's no synchronisation between workers and the
> main thread (by design, to prevent any possibility of deadlocks), and
> there's not currently a batching API.
>
> However, if this becomes a common problem (which we can determine by
> seeing if we get bugs complaining about different parts of apps/games
> seeming to slide around or generally be slightly out of sync, or if we see
> a lot of authors shunting multiple ImageBitmap objects across MessagePort
> channels) we can always add an explicit batching API to make this kind of
> thing easy.
>
> Note that in theory, for 2D at least, shunting ImageBitmaps across threads
> can be as efficient as commit().
>
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'**
> **
>
>  ** **
>

RE: exposing CANVAS or something like it to Web Workers

[Travis Leithead]

Having thought about this before, I wonder why we don’t use a producer/consumer 
model rather than a transfer of canvas ownership model?

A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), 
is to have an internal frame buffer accessible via a WorkerCanvas API which 
supports some set of canvas 2d/3d APIs as appropriate, and can “push” a 
completed frame onto a stack in the internal frame buffer. Thus the worker can 
produce frames as fast as desired.

On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, 
which just offers the “pop” API to pop a frame from the internal frame buffer 
into the canvas.

Then you just add some basic signaling events on both ends of the frame buffer 
and you’re good (as far as synchronizing the worker with the document). The 
producer (in the worker) is free to produce multiple frames in advance (if 
desired), while the consumer is able to pop frames when available. You could 
even have the framebuffer depth configurable.

From: Gregg Tavares [mailto:g...@google.com]
Sent: Thursday, February 7, 2013 2:25 PM
To: Ian Hickson
Cc: Charles Pritchard; Web Applications Working Group WG
Subject: Re: exposing CANVAS or something like it to Web Workers

I put up a new proposal for canvas in workers

http://wiki.whatwg.org/wiki/CanvasInWorkers

Please take a look.

This proposal comes from offline discussions with representatives from the 
various browsers as well as input from the Google Maps team. I can post a 
summary here if you'd like but it might be easier to read the wiki

Looking forward to feedback.



On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson 
mailto:i...@hixie.ch>> wrote:
On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote:
>
> Another issue as come up and that is one of being able
> to synchronize updates of a canvas in
> worker with changes in the main page.
For 2D, the intended solution is to just ship the ImageBitamp from the
worker canvas to the main thread via a MessagePort and then render it on
the canvas at the appropriate time.

I don't know how you would do it for WebGL.


> Similarly, let's say you have 2 canvases and are rendering to both in a
> worker.  Does
>
>context1.commit();
>context2.commit();
>
> guarantee you'll see both commits together?
No, unfortunately not. There's no synchronisation between workers and the
main thread (by design, to prevent any possibility of deadlocks), and
there's not currently a batching API.

However, if this becomes a common problem (which we can determine by
seeing if we get bugs complaining about different parts of apps/games
seeming to slide around or generally be slightly out of sync, or if we see
a lot of authors shunting multiple ImageBitmap objects across MessagePort
channels) we can always add an explicit batching API to make this kind of
thing easy.

Note that in theory, for 2D at least, shunting ImageBitmaps across threads
can be as efficient as commit().

--
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: exposing CANVAS or something like it to Web Workers

[Gregg Tavares]

I put up a new proposal for canvas in workers

http://wiki.whatwg.org/wiki/CanvasInWorkers

Please take a look.

This proposal comes from offline discussions with representatives from the
various browsers as well as input from the Google Maps team. I can post a
summary here if you'd like but it might be easier to read the wiki

Looking forward to feedback.




On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson  wrote:

> On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote:
> >
> > Another issue as come up and that is one of being able
> > to synchronize updates of a canvas in
> > worker with changes in the main page.
>
> For 2D, the intended solution is to just ship the ImageBitamp from the
> worker canvas to the main thread via a MessagePort and then render it on
> the canvas at the appropriate time.
>
> I don't know how you would do it for WebGL.
>
>
> > Similarly, let's say you have 2 canvases and are rendering to both in a
> > worker.  Does
> >
> >context1.commit();
> >context2.commit();
> >
> > guarantee you'll see both commits together?
>
> No, unfortunately not. There's no synchronisation between workers and the
> main thread (by design, to prevent any possibility of deadlocks), and
> there's not currently a batching API.
>
> However, if this becomes a common problem (which we can determine by
> seeing if we get bugs complaining about different parts of apps/games
> seeming to slide around or generally be slightly out of sync, or if we see
> a lot of authors shunting multiple ImageBitmap objects across MessagePort
> channels) we can always add an explicit batching API to make this kind of
> thing easy.
>
> Note that in theory, for 2D at least, shunting ImageBitmaps across threads
> can be as efficient as commit().
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: exposing CANVAS or something like it to Web Workers

[Ian Hickson]

On Wed, 2 Jan 2013, Gregg Tavares (社�~T�) wrote:
>
> Another issue as come up and that is one of being able
> to synchronize updates of a canvas in
> worker with changes in the main page.

For 2D, the intended solution is to just ship the ImageBitamp from the 
worker canvas to the main thread via a MessagePort and then render it on 
the canvas at the appropriate time.

I don't know how you would do it for WebGL.


> Similarly, let's say you have 2 canvases and are rendering to both in a
> worker.  Does
> 
>context1.commit();
>context2.commit();
> 
> guarantee you'll see both commits together?

No, unfortunately not. There's no synchronisation between workers and the 
main thread (by design, to prevent any possibility of deadlocks), and 
there's not currently a batching API.

However, if this becomes a common problem (which we can determine by 
seeing if we get bugs complaining about different parts of apps/games 
seeming to slide around or generally be slightly out of sync, or if we see 
a lot of authors shunting multiple ImageBitmap objects across MessagePort 
channels) we can always add an explicit batching API to make this kind of 
thing easy.

Note that in theory, for 2D at least, shunting ImageBitmaps across threads 
can be as efficient as commit().

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: exposing CANVAS or something like it to Web Workers

[Gregg Tavares]

On Wed, Jan 2, 2013 at 2:52 PM, Gregg Tavares (社用)  wrote:

> Another issue as come up and that is one of being able
> to synchronize updates of a canvas in
> worker with changes in the main page.
>
> For a real world example see Google's MapsGL
> (http://support.google.com/maps/bin/answer.py?hl=en&answer=1630790)
>
> Apparently MapsGL uses 2 canvases and/or some DOM objects overlayed on top
> of each other.
> Dragging the mouse moves objects in all of those layers and they need to
> move simultaneously
> to have a good UX.
>
> You can imagine issues if a canvas is being rendering to from a worker.
> How would the user
> guarantee that changes from the worker are synchronized with changes to
> the DOM in the
> main thread?
>
> Similarly, let's say you have 2 canvases and are rendering to both in a
> worker.  Does
>
>context1.commit();
>context2.commit();
>
> guarantee you'll see both commits together?
>

Let me retract this. There is a way under the current API to solve sync
which is to
use drawImage to another canvas in the main thread and have the canvases
being
drawing to by the worker basically as offscreen canvases.

It might not be the ideal solution since a drawImage call is an extra draw
which is
not cheap, especially if it's a large canvas. But, it does mean there's a
solution
for now without adding any extra api.



>
> Random thoughts
>
> *) Leave things the way they are and add another mechanism for syncing?
>
> In other words, by default things are not sync. Through some other API or
> settings the user can opt
> into getting synchronization
>
> *) Look at OpenGL "swap groups" as inspiration for an API?
>
> http://www.opengl.org/registry/specs/NV/wgl_swap_group.txt
> http://www.opengl.org/registry/specs/NV/glx_swap_group.txt
>
> *) Consider an 'oncommit' or 'onswap' event on 'Window'?
>
> The idea being you want to give the main thread a chance to update stuff
> (DOM elements) in response
> to a worker having called 'commit' on a canvas.
>
> Of course not sure how that would work if you have 2 workers each
> rendering to a different canvas.
>
> Note: I haven't thought through these issues at all and I've personally
> not had to deal with them but it
> seems clear from the MapsGL example that a solution will be needed for
> some subset of apps to have
> a good UX. I know for example there's game engine that has API to keep DOM
> elements located
> relative to game objects being rendered in a canvas to make it easy to
> give HTML based stats on
> the game objects as opposed to having to render the stats manually with
> canvas.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Nov 16, 2012 at 8:35 PM, Ian Hickson  wrote:
>
>> On Fri, 16 Nov 2012, Charles Pritchard wrote:
>> > >
>> > >
>> > >
>> http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html
>> >
>> > Seems like we might use requestAnimationFrame in the main thread to
>> > postMessage to the worker as an alternative to using setInterval in
>> > workers for repaints.
>>
>> The idea in due course is to just expose rAF in workers. Please do read
>> the e-mail above, which actually mentions that.
>>
>> --
>> Ian Hickson   U+1047E)\._.,--,'``.fL
>> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
>> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>>
>>
>

Re: exposing CANVAS or something like it to Web Workers

[社用]

Another issue as come up and that is one of being able
to synchronize updates of a canvas in
worker with changes in the main page.

For a real world example see Google's MapsGL
(http://support.google.com/maps/bin/answer.py?hl=en&answer=1630790)

Apparently MapsGL uses 2 canvases and/or some DOM objects overlayed on top
of each other.
Dragging the mouse moves objects in all of those layers and they need to
move simultaneously
to have a good UX.

You can imagine issues if a canvas is being rendering to from a worker. How
would the user
guarantee that changes from the worker are synchronized with changes to the
DOM in the
main thread?

Similarly, let's say you have 2 canvases and are rendering to both in a
worker.  Does

   context1.commit();
   context2.commit();

guarantee you'll see both commits together?

Random thoughts

*) Leave things the way they are and add another mechanism for syncing?

In other words, by default things are not sync. Through some other API or
settings the user can opt
into getting synchronization

*) Look at OpenGL "swap groups" as inspiration for an API?

http://www.opengl.org/registry/specs/NV/wgl_swap_group.txt
http://www.opengl.org/registry/specs/NV/glx_swap_group.txt

*) Consider an 'oncommit' or 'onswap' event on 'Window'?

The idea being you want to give the main thread a chance to update stuff
(DOM elements) in response
to a worker having called 'commit' on a canvas.

Of course not sure how that would work if you have 2 workers each rendering
to a different canvas.

Note: I haven't thought through these issues at all and I've personally not
had to deal with them but it
seems clear from the MapsGL example that a solution will be needed for some
subset of apps to have
a good UX. I know for example there's game engine that has API to keep DOM
elements located
relative to game objects being rendered in a canvas to make it easy to give
HTML based stats on
the game objects as opposed to having to render the stats manually with
canvas.















On Fri, Nov 16, 2012 at 8:35 PM, Ian Hickson  wrote:

> On Fri, 16 Nov 2012, Charles Pritchard wrote:
> > >
> > >
> > >
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html
> >
> > Seems like we might use requestAnimationFrame in the main thread to
> > postMessage to the worker as an alternative to using setInterval in
> > workers for repaints.
>
> The idea in due course is to just expose rAF in workers. Please do read
> the e-mail above, which actually mentions that.
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>
>

Re: exposing CANVAS or something like it to Web Workers

[Ian Hickson]

On Fri, 16 Nov 2012, Charles Pritchard wrote:
> > 
> >   
> > http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html
> 
> Seems like we might use requestAnimationFrame in the main thread to 
> postMessage to the worker as an alternative to using setInterval in 
> workers for repaints.

The idea in due course is to just expose rAF in workers. Please do read 
the e-mail above, which actually mentions that.

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On Nov 16, 2012, at 1:50 PM, Ian Hickson  wrote:

> On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote:
>> 
>> I'd like to work on exposing something like CANVAS to web workers.
>> 
>> Ideally how over it works I'd like to be able to
>> 
>> *) get a 2d context in a web worker
>> *) get a WebGL context in a web worker
>> *) download images in a web worker and the images with both 2d contexts and
>> WebGL contexts
> 
> I've now specced something like this; for details, see:
> 
>   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html

Seems like we might use requestAnimationFrame in the main thread to postMessage 
to the worker as an alternative to using setInterval in workers for repaints.

-Charles

Re: exposing CANVAS or something like it to Web Workers

[Ian Hickson]

On Mon, 14 May 2012, Gregg Tavares (�~K�) wrote:
>
> I'd like to work on exposing something like CANVAS to web workers.
> 
> Ideally how over it works I'd like to be able to
> 
> *) get a 2d context in a web worker
> *) get a WebGL context in a web worker
> *) download images in a web worker and the images with both 2d contexts and
> WebGL contexts

I've now specced something like this; for details, see:

   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Nov/0199.html

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: exposing CANVAS or something like it to Web Workers

[Robert O'Callahan]

That sounds like a reasonable approach to me. One thing missing is that
Workers would need something like requestAnimationFrame.

Any proposal that requires passing messages to the main thread to get
something on the screen fails to satisfy the huge need for games to be able
to get steady frame rates in the face of main-thread latency.

Rob
-- 
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]

Re: exposing CANVAS or something like it to Web Workers

[Glenn Maynard]

On Wed, May 16, 2012 at 2:30 PM, Gregg Tavares (勤)  wrote:

> Some Ideas
>
> **) Create context in main page, pass to worker*
>

Transferring an active context from one thread to another (at the
implementation level) can be hard.  It's much simpler to only create new
the contexts in the thread they'll be used, never allowing contexts to be
moved from thread to thread.

Here's a first-pass approach:

1: Add HTMLCanvasElement.getBackbuffer, which returns a lightweight
Backbuffer interface associated with the HTMLCanvasElement.  This has only
one method, getContext, which is equivalent to canvas.getContext.  (This is
exactly like the suggestion for making images available in threads.)
2: Backbuffer can be transferred from one thread to another, using the
transfer mechanism.  On transfer, any WebGL contexts (and other context
types) in the thread associated with the same backbuffer are destroyed
(putting exactly what "destroyed" means aside for later).  Additionally,
when a Backbuffer is transferred, the underlying backbuffer of the
HTMLCanvasElement is considered owned by the new thread.  All instances of
Backbuffer for that HTMLCanvasElement, including newly-created ones, are
neutered unless the backbuffer is transferred back to the UI thread.
3: As long as the backbuffer of an HTMLCanvasElement is owned by a thread
other than the UI thread, calls to the following methods on the associated
HTMLCanvasElement raise an exception: getBackbuffer, getContext,
toBlob, toDataURL.
This takes effect synchronously, as soon as transfer happens (right when
you postMessage the Backbuffer somewhere else).
4: When a Worker receives a Backbuffer, it can call Backbuffer.getContext
to create a WebGLContext (with exactly the same rules as
HTMLCanvasElement.getContext).  The results of rendering to that context
are visible in the associated HTMLCanvasElement in the main thread.

This ensures that only one thread can ever have an open context for any
backbuffer, and that pixel readback functions (toBlob and toDataURL) can't
expose the asynchronous nature of what's going on.  The transfer mechanics
are rough and need refining.

For offscreen rendering in a thread, just allow constructing Backbuffer;
for example,

var backbuffer = new Backbuffer(1024, 768);
var ctx = backbuffer.getContext("webgl");
backbuffer.resize(1920, 1200);

Backbuffer.resize is equivalent to resizing a Canvas, and only available
for Backbuffers you create yourself, not for ones created via
HTMLCanvasElement.getBackBuffer.

One problem that I havn't attempted to solve here: when the
HTMLCanvasElement is resized in the UI thread, it results in changes to
drawingBufferWidth/drawingBufferHeight.  This would expose asynchronous
behavior.  Instead, it would probably need to apply the change (from the
thread's perspective) in a queued task, so you have to return to the event
loop for it to be applied.

If a thread is killed by the browser (eg. due to a CPU quota), the
backbuffers it owns are orphaned; you can no longer create contexts for
it.  You need to create a new Canvas.  This isn't great, but workers don't
really seem to try to make it possible to recover from this anyway.

interface CanvasBackbuffer {
object? getContext(DOMString contextId, any... args);
}

[Constructor(unsigned long width, unsigned long height)]
interface Backbuffer : CanvasBackbuffer {
void resize(unsigned long width, unsigned long height);
}

interface HTMLCanvasElement : HTMLElement {
CanvasBackbuffer getBackbuffer();
}

The Backbuffer ctor is available in workers.

Importantly, you can start by just implementing the Backbuffer
constructor.  That would only allow the simpler case of offscreen rendering.

-- 
Glenn Maynard

Re: exposing CANVAS or something like it to Web Workers

[Kenneth Russell]

On Wed, May 16, 2012 at 1:11 PM, Gregg Tavares (勤)  wrote:
>
>
> On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell  wrote:
>>
>> On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) 
>> wrote:
>> > So how to take this forward?
>> >
>> > My #1 priority is to get WebGL in workers. Lots of developers have
>> > expressed
>> > a need for this from decoding compressed textures in a worker to
>> > offloading
>> > thousands of draw calls per frame to a worker. WebGL already defines
>> > sharing
>> > mechanisms so at least for the WebGL case I don't need to solve how to
>> > display anything back in the main worker. The worker can draw to a
>> > texture.
>> > The main page can use that texture. WebGL/OpenGL already define that
>> > relationship.
>> >
>> > But, to get WebGL in a worker I need a way to get a
>> > WebGLRenderingContext
>> > into a worker.
>> >
>> > Some Ideas
>> >
>> > *) Create context in main page, pass to worker
>> >
>> > Pros:
>> >
>> > . No new APIs or objects.
>> >
>> > Cons:
>> >
>> > . Transfer is messy.
>> >
>> > What happens to all the entry points and properties to the context
>> > object
>> > left in the main page?
>> > What happens to the "cavnas" parameter on the transferred context?
>> > How do you synchronize the main page setting canvas.width or
>> > canvas.height
>> > with the worker trying to render to it?
>> >
>> >
>> > *) Create a context directly in a Worker (or anywhere for that matter)
>> >
>> > As in "var gl = new WebGLRenderingContext"
>> >
>> > Pros:
>> >
>> > . ???
>> >
>> > Cons:
>> >
>> > . requires defining how the backbuffer size is set.
>> >    Maybe there is no backbuffer for a directly created
>> > WebGLRenderingContext?
>> > . if there is no backbuffer then using one of these contexts to draw
>> > into a
>> > texture
>> >   or 2d context is problematic
>>
>> This alternative seems like the one that can be moved forward most
>> easily, since essentially all of the changes would be within the WebGL
>> spec.
>>
>>  - Extend the spec to support context sharing. (Each
>> WebGLRenderingContext points to an opaque WebGLContextGroup object.)
>>  - Define structured cloning semantics for WebGLContextGroup.
>>  - Add a constructor or factory method to WebGLRenderingContext
>> allowing creation with a WebGLContextGroup. Contexts created in this
>> manner would have no back buffer by default. (FBOs could still be used
>> to do rendering with the context.)
>>  - Allow some or all of the WebGLObject types (textures, etc.) to be
>> either copied during structured cloning or transferred.
>>
>> Then the worker can at least upload textures to the GPU completely
>> asynchronously from the main thread, and inform the main thread via
>> postMessage when they have finished uploading. That seems to be a
>> tractable first step and one that would already have immediate
>> benefits for developers.
>
>
> That problem I have with this path is the cons mentioned above. It's a dead
> end.
>
> With real contexts you can do this
>
>     webglcontext1.texImage2D(..., webglcontext2.canvas);
>
> and this
>
>     context2d.drawImage(webgl.canvas, ...)
>
> But with a context created with new WebGLRenderingContext you can't do
> anything like that because size of the backbuffer is not defined. Eventually
> we'll want to support operation like that and we'll likely end up with
> something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that
> point this ability to go "new WebGLRenderingContext" will just be left over
> cruft.
>
> Also we'll have to define how to addEventListner for WebGLRenderContext for
> listening for lost context or for async context creation (yea, less
> important on workers). That stuff is currently not on WebGLRenderingContext.
> It seems like it should stay off if it.

These are all good points, and I agree that the ideal fix would be to
use the Canvas element (or DrawingSurface, etc.) from a worker, rather
than doing a WebGL-specific hack. Given how long discussions have been
ongoing on this topic, though, I think this may just be too large a
single step to take.

Rather than add a constructor to WebGLRenderingContext that would need
to remain there permanently, perhaps a factory method with a vendor
prefix could be added (e.g.
"WebGLRenderingContext.webkitCreateOffscreenContext",
"WebGLRenderingContext.mozCreateOffscreenContext"). The long-term plan
could be to remove the factory method once a better solution is
reached, as opposed to removing the vendor prefixes.

This would at least allow some applications to explore the use of
workers with WebGL, see whether this addresses any existing issues,
and expose new issues.

-Ken



>>
>>
>> -Ken
>>
>>
>> > *) Create an offscreen canvas like object minus HTMLElement parts
>> >
>> > Pros:
>> >
>> > . same or nearly the API as already exists for canvas
>> > . flexible. getContext can return different kinds of context. Maybe only
>> > "webgl" for now?
>> >
>> > Cons:
>> >
>> > . ???
>> >
>> >
>> >
>> >
>
>

Re: exposing CANVAS or something like it to Web Workers

[勤]

On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell  wrote:

> On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) 
> wrote:
> > So how to take this forward?
> >
> > My #1 priority is to get WebGL in workers. Lots of developers have
> expressed
> > a need for this from decoding compressed textures in a worker to
> offloading
> > thousands of draw calls per frame to a worker. WebGL already defines
> sharing
> > mechanisms so at least for the WebGL case I don't need to solve how to
> > display anything back in the main worker. The worker can draw to a
> texture.
> > The main page can use that texture. WebGL/OpenGL already define that
> > relationship.
> >
> > But, to get WebGL in a worker I need a way to get a WebGLRenderingContext
> > into a worker.
> >
> > Some Ideas
> >
> > *) Create context in main page, pass to worker
> >
> > Pros:
> >
> > . No new APIs or objects.
> >
> > Cons:
> >
> > . Transfer is messy.
> >
> > What happens to all the entry points and properties to the context object
> > left in the main page?
> > What happens to the "cavnas" parameter on the transferred context?
> > How do you synchronize the main page setting canvas.width or
> canvas.height
> > with the worker trying to render to it?
> >
> >
> > *) Create a context directly in a Worker (or anywhere for that matter)
> >
> > As in "var gl = new WebGLRenderingContext"
> >
> > Pros:
> >
> > . ???
> >
> > Cons:
> >
> > . requires defining how the backbuffer size is set.
> >Maybe there is no backbuffer for a directly created
> > WebGLRenderingContext?
> > . if there is no backbuffer then using one of these contexts to draw
> into a
> > texture
> >   or 2d context is problematic
>
> This alternative seems like the one that can be moved forward most
> easily, since essentially all of the changes would be within the WebGL
> spec.
>
>  - Extend the spec to support context sharing. (Each
> WebGLRenderingContext points to an opaque WebGLContextGroup object.)
>  - Define structured cloning semantics for WebGLContextGroup.
>  - Add a constructor or factory method to WebGLRenderingContext
> allowing creation with a WebGLContextGroup. Contexts created in this
> manner would have no back buffer by default. (FBOs could still be used
> to do rendering with the context.)
>  - Allow some or all of the WebGLObject types (textures, etc.) to be
> either copied during structured cloning or transferred.
>
> Then the worker can at least upload textures to the GPU completely
> asynchronously from the main thread, and inform the main thread via
> postMessage when they have finished uploading. That seems to be a
> tractable first step and one that would already have immediate
> benefits for developers.
>

That problem I have with this path is the cons mentioned above. It's a dead
end.

With real contexts you can do this

webglcontext1.texImage2D(..., webglcontext2.canvas);

and this

context2d.drawImage(webgl.canvas, ...)

But with a context created with new WebGLRenderingContext you can't do
anything like that because size of the backbuffer is not defined.
Eventually we'll want to support operation like that and we'll likely end
up with something like DrawingSurface or CanvasSurface or OffscreenCanvas.
At that point this ability to go "new WebGLRenderingContext" will just be
left over cruft.

Also we'll have to define how to addEventListner for WebGLRenderContext for
listening for lost context or for async context creation (yea, less
important on workers). That stuff is currently not on
WebGLRenderingContext. It seems like it should stay off if it.








>
> -Ken
>
>
> > *) Create an offscreen canvas like object minus HTMLElement parts
> >
> > Pros:
> >
> > . same or nearly the API as already exists for canvas
> > . flexible. getContext can return different kinds of context. Maybe only
> > "webgl" for now?
> >
> > Cons:
> >
> > . ???
> >
> >
> >
> >
>

Re: exposing CANVAS or something like it to Web Workers

[Kenneth Russell]

On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤)  wrote:
> So how to take this forward?
>
> My #1 priority is to get WebGL in workers. Lots of developers have expressed
> a need for this from decoding compressed textures in a worker to offloading
> thousands of draw calls per frame to a worker. WebGL already defines sharing
> mechanisms so at least for the WebGL case I don't need to solve how to
> display anything back in the main worker. The worker can draw to a texture.
> The main page can use that texture. WebGL/OpenGL already define that
> relationship.
>
> But, to get WebGL in a worker I need a way to get a WebGLRenderingContext
> into a worker.
>
> Some Ideas
>
> *) Create context in main page, pass to worker
>
> Pros:
>
> . No new APIs or objects.
>
> Cons:
>
> . Transfer is messy.
>
> What happens to all the entry points and properties to the context object
> left in the main page?
> What happens to the "cavnas" parameter on the transferred context?
> How do you synchronize the main page setting canvas.width or canvas.height
> with the worker trying to render to it?
>
>
> *) Create a context directly in a Worker (or anywhere for that matter)
>
> As in "var gl = new WebGLRenderingContext"
>
> Pros:
>
> . ???
>
> Cons:
>
> . requires defining how the backbuffer size is set.
>    Maybe there is no backbuffer for a directly created
> WebGLRenderingContext?
> . if there is no backbuffer then using one of these contexts to draw into a
> texture
>   or 2d context is problematic

This alternative seems like the one that can be moved forward most
easily, since essentially all of the changes would be within the WebGL
spec.

 - Extend the spec to support context sharing. (Each
WebGLRenderingContext points to an opaque WebGLContextGroup object.)
 - Define structured cloning semantics for WebGLContextGroup.
 - Add a constructor or factory method to WebGLRenderingContext
allowing creation with a WebGLContextGroup. Contexts created in this
manner would have no back buffer by default. (FBOs could still be used
to do rendering with the context.)
 - Allow some or all of the WebGLObject types (textures, etc.) to be
either copied during structured cloning or transferred.

Then the worker can at least upload textures to the GPU completely
asynchronously from the main thread, and inform the main thread via
postMessage when they have finished uploading. That seems to be a
tractable first step and one that would already have immediate
benefits for developers.

-Ken


> *) Create an offscreen canvas like object minus HTMLElement parts
>
> Pros:
>
> . same or nearly the API as already exists for canvas
> . flexible. getContext can return different kinds of context. Maybe only
> "webgl" for now?
>
> Cons:
>
> . ???
>
>
>
>

Re: exposing CANVAS or something like it to Web Workers

[勤]

So how to take this forward?

My #1 priority is to get WebGL in workers. Lots of developers have
expressed a need for this from decoding compressed textures in a worker to
offloading thousands of draw calls per frame to a worker. WebGL already
defines sharing mechanisms so at least for the WebGL case I don't need to
solve how to display anything back in the main worker. The worker can draw
to a texture. The main page can use that texture. WebGL/OpenGL already
define that relationship.

But, to get WebGL in a worker I need a way to get a WebGLRenderingContext
into a worker.

Some Ideas

**) Create context in main page, pass to worker*

Pros:

. No new APIs or objects.

Cons:

. Transfer is messy.

What happens to all the entry points and properties to the context object
left in the main page?
What happens to the "cavnas" parameter on the transferred context?
How do you synchronize the main page setting canvas.width or canvas.height
with the worker trying to render to it?


**) Create a context directly in a Worker (or anywhere for that matter)*

As in "var gl = new WebGLRenderingContext"

Pros:

. ???

Cons:

. requires defining how the backbuffer size is set.
   Maybe there is no backbuffer for a directly created
WebGLRenderingContext?
. if there is no backbuffer then using one of these contexts to draw into a
texture
  or 2d context is problematic


**) Create an offscreen canvas like object minus HTMLElement parts*
*
*
Pros:

. same or nearly the API as already exists for canvas
. flexible. getContext can return different kinds of context. Maybe only
"webgl" for now?

Cons:

. ???

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 10:32 PM, Charles Pritchard wrote:

Yes, I understand that. Canvas 2d text still needs to be able to do
things like font fallback, shaping, bidi, etc, etc, etc. last I checked.


Oh, the rendering isn't thread safe either?


Right.  The rendering involves access to style data structures that are 
not threadsafe.  It involves caching shaped textruns and such; that 
cache is not threadsafe.  The actual shaper may or may not be 
threadsafe.  The bidi code is not threadsafe.


These are all solvable issues... on some timescale.  ;)

-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 7:24 PM, Boris Zbarsky wrote:

On 5/14/12 10:18 PM, Charles Pritchard wrote:

On 5/14/2012 7:09 PM, Boris Zbarsky wrote:

On 5/14/12 10:00 PM, Charles Pritchard wrote:

What would web fonts do in this situation, in Mozilla?


Probably cry. ;)


If I've confirmed that a font is loaded in the main thread, would it
be available to a
worker for use in rendering?


Not without some pretty serious reworking. Which might need to happen.

Of course basic text layout would also not be available without some
serious reworking (e.g. making the textrun cache threadsafe or
creating per-thread textrun caches or something), so the question of
web fonts is somewhat academic at the moment.



I meant solely for Canvas 2d.


Yes, I understand that.  Canvas 2d text still needs to be able to do 
things like font fallback, shaping, bidi, etc, etc, etc. last I checked.


Oh, the rendering isn't thread safe either? Yes, Canvas 2d text does [is 
supposed to] use all of those items.


Well, I'll give up strokeText/fillText entirely in workers if it'll get 
me the goods faster.
For a11y, I'm going to need to track my text in the main thread anyway. 
I can pre-render there if need be.


-Charles

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 10:18 PM, Charles Pritchard wrote:

On 5/14/2012 7:09 PM, Boris Zbarsky wrote:

On 5/14/12 10:00 PM, Charles Pritchard wrote:

What would web fonts do in this situation, in Mozilla?


Probably cry. ;)


If I've confirmed that a font is loaded in the main thread, would it
be available to a
worker for use in rendering?


Not without some pretty serious reworking. Which might need to happen.

Of course basic text layout would also not be available without some
serious reworking (e.g. making the textrun cache threadsafe or
creating per-thread textrun caches or something), so the question of
web fonts is somewhat academic at the moment.



I meant solely for Canvas 2d.


Yes, I understand that.  Canvas 2d text still needs to be able to do 
things like font fallback, shaping, bidi, etc, etc, etc. last I checked.


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 7:09 PM, Boris Zbarsky wrote:

On 5/14/12 10:00 PM, Charles Pritchard wrote:

What would web fonts do in this situation, in Mozilla?


Probably cry.  ;)

If I've confirmed that a font is loaded in the main thread, would it 
be available to a

worker for use in rendering?


Not without some pretty serious reworking.  Which might need to happen.

Of course basic text layout would also not be available without some 
serious reworking (e.g. making the textrun cache threadsafe or 
creating per-thread textrun caches or something), so the question of 
web fonts is somewhat academic at the moment.




I meant solely for Canvas 2d.

I can live with staying away from fillText/strokeText on a worker thread 
if I'm loading fonts.
It's been broken on the main thread anyway, requiring intermediate 
Canvas surfaces for some operations.


...

SVG image and drawImage is mixed anyway; we can't transfer the data 
between threads as drawImage SVG will usually flag the Canvas as dirty 
in implementations.


We could just use Canvas 2d to handle pattern uploads for WebGL. Seems 
like that'd work without requiring fancy footwork to gain Picture/Image 
support in the worker.

SVG images would get fixed some other day.


-Charles

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 10:00 PM, Charles Pritchard wrote:

What would web fonts do in this situation, in Mozilla?


Probably cry.  ;)


If I've confirmed that a font is loaded in the main thread, would it be 
available to a
worker for use in rendering?


Not without some pretty serious reworking.  Which might need to happen.

Of course basic text layout would also not be available without some 
serious reworking (e.g. making the textrun cache threadsafe or creating 
per-thread textrun caches or something), so the question of web fonts is 
somewhat academic at the moment.


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 6:07 PM, Boris Zbarsky wrote:

On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:

1)  Various canvas 2d context methods depend on the styles of the
canvas to define how they actually behave.  Clearly this would need
some sort of changes for Workers anyway; the question is what those
changes would need to be.

Which methods are these?


Anything involving setting color (e.g. the strokeStyle setter, the 
fillStyle setter), due to "currentColor".  Anything involving text 
because font styles come from the element or document.


Those are the ones that come to mind offhand, but I haven't looked at 
the various recent additions to the 2d context closely.


What would web fonts do in this situation, in Mozilla? If I've confirmed 
that a font is loaded in the main thread, would it be available to a 
worker for use in rendering?


Some implementations of Canvas font methods have been buggy, but 
dropping fillText/strokeText altogether would be a loss and may look 
strange in the specs.
I've seen font not working correctly with other canvas state variables 
in WebKit + Chrome; so it's not as though font is fully supported in the 
main thread, currently.


It may be easier to postpone Image / Picture semantics in workers. I 
think patterns could still be salvaged from createPattern.

It sounds like there are practical issues in current implementations.

When authoring,
With a picture, I can still xhr request it in the worker, then I send it 
to the main thread, and load it in the main thread, then send it back to 
the worker.
As an author, if I'm loading images I expect them to be a normal part of 
the page's load time and responsiveness I can still off-load my 
js-heavy work onto workers.


In Canvas 2d anyway, I can easily get by without picture in workers: (a 
= new Picture()).src = ; a.onload { ctx.drawImage(a,0,0); };  if I 
can use fillRect with pattern.
Otherwise, I'd have to do the extra steps of pushing pixel arrays back 
and forth.


All of the extra work is still worth it to get work done in workers, in 
cases where this level of work is needed. It's just a few extra lines of 
JS code (every time).


-Charles

Re: exposing CANVAS or something like it to Web Workers

[勤]

On Mon, May 14, 2012 at 6:23 PM, Jonas Sicking  wrote:

> On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard 
> wrote:
> > On May 14, 2012, at 4:42 PM, Jonas Sicking  wrote:
> >
> >> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
> >>> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤) 
> wrote:
> >>>
>  I'd like to work on exposing something like CANVAS to web workers.
> 
>  Ideally how over it works I'd like to be able to
> 
>  *) get a 2d context in a web worker
> >>>
> >>>
> >>> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
> >>> worrying about WebGL to start.
> >>>
> >>> Another issue: rendering in a worker thread onto a canvas which is
> displayed
> >>> in the main thread.  This needs to be solved in a way that doesn't
> cause the
> >>> asynchronous nature of what's happening to be visible to scripts.
>  toDataURL
> >>> and toBlob would probably need to be prohibited on the canvas element.
>  I'm
> >>> not sure what the actual API would look like.
> >>
> >> If/when we do this, I think it should be done in such a way that the
> >> main window can't access the canvas object at all. Similar to what
> >> happens when an ArrayBuffer is transferred to a Worker using
> >> structured cloning. Once a canvas is transferred to a Worker, any
> >> access to it should throw or return null/0/"". If you want to transfer
> >> pixel data to the main thread, it seems less racy to do that by
> >> getting the pixel data in the Worker which owns the canvas and then
> >> transfer that to the main thread using postMessage.
> >>
> >>> This would also require some
> >>> equivalent to requestAnimationFrame in the worker thread.
> >>
> >> Agreed!
> >>
> >> / Jonas
> >
> >
> >
> > I'm a bit lost-- wouldn't we just postMessage from the document over to
> the web worker when we want a refresh?
> >
> > I agree that we ought to be transferring pixel data not Canvas contexts;
> with the possible exception of CSS context.
> >
> > We could just create new canvas instances inside the worker thread. I'd
> still prefer to clone CanvasPattern as a means of transferring paint over
> to the worker, though sending pixel data would work too.
> >
> > I heard Picture come up-- it seems like that object might have
> additional semantics for high resolution alternatives that may need to be
> considered.
>
> I was saying that I think we should transfer the context from the main
> thread to the worker thread. The worker thread should then be able to
> use the context to draw directly to the screen without interacting
> with the main thread.
>

That's a great goal but is transferring the right way to do it?

context.canvas is currently a reference to the canvas from which the
context as created. Will that just go NULL on transfer? Will it change to
some other object that is not a Canvas but acts similarly?

What happens to the context on the main page? All of it's methods become
no-ops and it's properties become non-live?

How will things synchronize if I call canvas.toDataURL after passing the
context to a work? Exception? nothing? It seems hard to define how that
would synchronize

If there was a way to allow a worker to generate a media stream and one of
those ways involves an in worker canvas/canvassurface, then it seems like
none of the above questions have to be answered. Instead the worker creates
the surface, and then through some other API attached it to a video tag.
You can call video.toDataURL, that's well defined (I assume)

Synchronization is also somewhat defined in that direction. Doesn't that
seem like the path of least resistance?





>
> You should generally not need to transfer pixel data. But I think it
> should be possible to grab pixel data on whichever thread is currently
> owning a context. This will implicitly make it possible to transfer
> pixel data if anyone wants to do it.
>
> / Jonas
>

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 6:23 PM, Jonas Sicking wrote:

>>>  This would also require some
>>>  equivalent to requestAnimationFrame in the worker thread.

>>
>>  Agreed!
>>
>>  / Jonas

>
>
>
>  I'm a bit lost-- wouldn't we just postMessage from the document over to the 
web worker when we want a refresh?
>
>  I agree that we ought to be transferring pixel data not Canvas contexts; 
with the possible exception of CSS context.
>
>  We could just create new canvas instances inside the worker thread. I'd 
still prefer to clone CanvasPattern as a means of transferring paint over to the 
worker, though sending pixel data would work too.
>
>  I heard Picture come up-- it seems like that object might have additional 
semantics for high resolution alternatives that may need to be considered.

I was saying that I think we should transfer the context from the main
thread to the worker thread. The worker thread should then be able to
use the context to draw directly to the screen without interacting
with the main thread.

You should generally not need to transfer pixel data. But I think it
should be possible to grab pixel data on whichever thread is currently
owning a context. This will implicitly make it possible to transfer
pixel data if anyone wants to do it.


OK, that's the same concept as I was hoping for with 
document.getCSSCanvasContext.


Mozilla is a bit ahead of webkit, going with  "element(any-element)" -- 
webkit has "-webkit-canvas(css-canvas-id)".


It's a very different approach. I'd still go ahead and just have 
requestAnimationFrame pump events from the main frame.
There are so many things that could block the main frame to where an rAF 
repaint just isn't necessary from the worker-end.


It may also keep some animations continuing. I'm not positive, but 
something like window.prompt may block execution on the main thread, but 
if rAF were running on the worker, its repaints would still happen.
Generally, while the main thread is blocked, repainting on it is not 
going to be that useful; we're shuffling the repaints off so they aren't 
the cause of blocking.


Those are the two sides of it that I see... but it's still a very 
different proposal than the idea of just having non-transferable canvas 
contexts.


...

Boris brought up some good points about possible blocking in 
implementations with loading SVG images in the worker thread via Picture 
(or a chopped down Image). At present, those sound more like 
implementation-issues, not particularly issues in the validity of a spec.

http://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0744.html

They do sound a little unfortunate, but I'd sooner trade some possible 
blocking between a Worker and the main thread, than having to carry a 
full SVG parser written in JS in my worker threads.
As an author, I've always had to take precautions for SVG and decide 
whether I want to render or give myself up to the browser implementation.


-Charles

Re: exposing CANVAS or something like it to Web Workers

[Jonas Sicking]

On Mon, May 14, 2012 at 5:10 PM, Charles Pritchard  wrote:
> On May 14, 2012, at 4:42 PM, Jonas Sicking  wrote:
>
>> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
>>> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  wrote:
>>>
 I'd like to work on exposing something like CANVAS to web workers.

 Ideally how over it works I'd like to be able to

 *) get a 2d context in a web worker
>>>
>>>
>>> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
>>> worrying about WebGL to start.
>>>
>>> Another issue: rendering in a worker thread onto a canvas which is displayed
>>> in the main thread.  This needs to be solved in a way that doesn't cause the
>>> asynchronous nature of what's happening to be visible to scripts.  toDataURL
>>> and toBlob would probably need to be prohibited on the canvas element.  I'm
>>> not sure what the actual API would look like.
>>
>> If/when we do this, I think it should be done in such a way that the
>> main window can't access the canvas object at all. Similar to what
>> happens when an ArrayBuffer is transferred to a Worker using
>> structured cloning. Once a canvas is transferred to a Worker, any
>> access to it should throw or return null/0/"". If you want to transfer
>> pixel data to the main thread, it seems less racy to do that by
>> getting the pixel data in the Worker which owns the canvas and then
>> transfer that to the main thread using postMessage.
>>
>>> This would also require some
>>> equivalent to requestAnimationFrame in the worker thread.
>>
>> Agreed!
>>
>> / Jonas
>
>
>
> I'm a bit lost-- wouldn't we just postMessage from the document over to the 
> web worker when we want a refresh?
>
> I agree that we ought to be transferring pixel data not Canvas contexts; with 
> the possible exception of CSS context.
>
> We could just create new canvas instances inside the worker thread. I'd still 
> prefer to clone CanvasPattern as a means of transferring paint over to the 
> worker, though sending pixel data would work too.
>
> I heard Picture come up-- it seems like that object might have additional 
> semantics for high resolution alternatives that may need to be considered.

I was saying that I think we should transfer the context from the main
thread to the worker thread. The worker thread should then be able to
use the context to draw directly to the screen without interacting
with the main thread.

You should generally not need to transfer pixel data. But I think it
should be possible to grab pixel data on whichever thread is currently
owning a context. This will implicitly make it possible to transfer
pixel data if anyone wants to do it.

/ Jonas

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 9:15 PM, Charles Pritchard wrote:

On 5/14/2012 6:08 PM, Boris Zbarsky wrote:

On 5/14/12 8:58 PM, Charles Pritchard wrote:

I agree... Can we get this off the main thread?


"Maybe".

It would be pretty nontrivial in Gecko; last I looked it would be
pretty painful in WebKit too. Can't speak for other UAs.


Can it be pumped through what's essentially an iframe on a null origin?


The issue at least in Gecko is that the DOM uses various 
effectively-global variables (shared caches of various sorts, 
non-threadsafe services, etc).  The origin is irrelevant.



I'd just assumed that  was loaded off-thread /
async much like  calls.


The _loading_ is async.  In Gecko the parsing of the XML, DOM 
construction, style computations, layout are on the main thread, like 
they are for HTML.


It's the "DOM construction, style computations, layout" parts that are 
not so easy to push off to a background thread...


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 6:14 PM, Gregg Tavares (勤) wrote:



On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky > wrote:


On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:

   1)  Various canvas 2d context methods depend on the styles
of the
   canvas to define how they actually behave.  Clearly this
would need
   some sort of changes for Workers anyway; the question is
what those
   changes would need to be.

Which methods are these?


Anything involving setting color (e.g. the strokeStyle setter, the
fillStyle setter), due to "currentColor".  Anything involving text
because font styles come from the element or document.


Good to know.

That doesn't sound like a showstopper though. If a 
canvas/CanvasSurface is available in workers the simplest solution 
would just be that "currentColor" defaults to something "black?" or 
nothing "". Pick one.


Font is still a little tricky from loading fonts via CSS. Font is tricky 
anyway, though, so it wouldn't be that much of a step backward.


Would we assume that if a font is available from the parent context it's 
going to be available to the worker?


currentColor would just default to black, as we're not talking about a 
color inherited from the DOM.




Those are the ones that come to mind offhand, but I haven't looked
at the various recent additions to the 2d context closely.



The recent additions are more proposals than additions. They're 
proposals from Tab and Ian and not yet implemented.


They revolve around a Path object, which we've not yet discussed.
Otherwise, they include lightweight nodes and DOM fallback content, 
which isn't relevant to workers.


My thinking is the same as yours: fillStyle/strokeStyle and font are the 
ones that come to mind.
Pattern and Gradient are items that could conceivably be cloned and/or 
shared.

They can both be as efficient as sending Blob via postMessage.

I think Gregg was just settling on not-sending Canvas over postMessage 
but rather creating the instance inside of each Worker.

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 6:08 PM, Boris Zbarsky wrote:

On 5/14/12 8:58 PM, Charles Pritchard wrote:

I agree... Can we get this off the main thread?


"Maybe".

It would be pretty nontrivial in Gecko; last I looked it would be 
pretty painful in WebKit too.  Can't speak for other UAs.


Can it be pumped through what's essentially an iframe on a null origin?

I don't know enough about browser internals to help on this one.
Yes, loading an SVG image is a heavy call.

For 90% of the SVG content out there, it'd probably be faster to parse 
and draw the SVG via Canvas and JS.

Still it's a hell of a lot nicer to load it via  tag.

I'd just assumed that  was loaded off-thread / 
async much like  calls.


There's nothing that gets carried from the document to the  other 
than the width/height, which I believe is carried through.

Which is a good thing, of course.


-Charles

Re: exposing CANVAS or something like it to Web Workers

[勤]

On Mon, May 14, 2012 at 6:07 PM, Boris Zbarsky  wrote:

> On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:
>
>>1)  Various canvas 2d context methods depend on the styles of the
>>canvas to define how they actually behave.  Clearly this would need
>>some sort of changes for Workers anyway; the question is what those
>>changes would need to be.
>>
>> Which methods are these?
>>
>
> Anything involving setting color (e.g. the strokeStyle setter, the
> fillStyle setter), due to "currentColor".  Anything involving text because
> font styles come from the element or document.
>

Good to know.

That doesn't sound like a showstopper though. If a canvas/CanvasSurface is
available in workers the simplest solution would just be that
"currentColor" defaults to something "black?" or nothing "". Pick one.


>
> Those are the ones that come to mind offhand, but I haven't looked at the
> various recent additions to the 2d context closely.
>
> -Boris
>

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 8:58 PM, Charles Pritchard wrote:

I agree... Can we get this off the main thread?


"Maybe".

It would be pretty nontrivial in Gecko; last I looked it would be pretty 
painful in WebKit too.  Can't speak for other UAs.



Svg via image is not quite the same as svg via HTMLDocument (I guess I mean, 
embedded).


Sure, but that doesn't matter.


Afaik, svg via image does not have any script controls but it does have xsl 
things.


And CSS.


Put in other words:  may use an entirely different implementation 
than  in an HTML document.


It _may_, but it would likely be pretty buggy.  And at least in Gecko, I 
don't think we're willing to duplicate enough of the rendering engine to 
do this.


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 8:55 PM, Gregg Tavares (勤) wrote:

1)  Various canvas 2d context methods depend on the styles of the
canvas to define how they actually behave.  Clearly this would need
some sort of changes for Workers anyway; the question is what those
changes would need to be.

Which methods are these?


Anything involving setting color (e.g. the strokeStyle setter, the 
fillStyle setter), due to "currentColor".  Anything involving text 
because font styles come from the element or document.


Those are the ones that come to mind offhand, but I haven't looked at 
the various recent additions to the 2d context closely.


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On May 14, 2012, at 5:50 PM, Boris Zbarsky  wrote:

> On 5/14/12 8:21 PM, Charles Pritchard wrote:
>> SVG and animated gif would render the same as it would in an
>> Image that has not been added to the dom or is otherwise display: none.
> 
> I'm not sure that would be workable in a worker for SVG, for the same reasons 
> that responseXML is not available in workers: it involves DOM elements.  
> Unless the SVG rasterization happens on the main thread under the hood and 
> the raster data is then sent over to the worker.  This might have ... 
> surprising performance characteristics.

I agree... Can we get this off the main thread? Svg via image is not quite the 
same as svg via HTMLDocument (I guess I mean, embedded).

Afaik, svg via image does not have any script controls but it does have xsl 
things. 

I've never tried to abuse the distinction via embedded blob Uris and such.

Put in other words:  may use an entirely different 
implementation than  in an HTML document.

Re: exposing CANVAS or something like it to Web Workers

[勤]

On Mon, May 14, 2012 at 5:12 PM, Boris Zbarsky  wrote:

> On 5/14/12 8:03 PM, Gregg Tavares (勤) wrote:
>
>> var canvasSurface = new CanvasSurface();
>> var ctx = canvasSurface.getContext("2d")**;
>> var pic = new Picture;
>> pic.src = 
>> "http://someplace.com/**someimage.jpg
>> ";
>> pic.onload = function() {
>>ctx.drawImage(pic, 0, );
>> }
>>
>> Let's assume you can instantiate these things in either the page or a
>> worker. Nether can be transfered.
>>
>> Would that work? What problems would that have?
>>
>
> Two things that come to mind immediately:
>
> 1)  Various canvas 2d context methods depend on the styles of the canvas
> to define how they actually behave.  Clearly this would need some sort of
> changes for Workers anyway; the question is what those changes would need
> to be.
>

Which methods are these?



>
> 2)  How (or maybe whether) this should work with pic.src = "
> http://someplace.com/**someimage.svg "
> is an interesting question.
>
> 3)  This leaves open the question of how to get data from the worker back
> to the main thread to paint there.
>

I feel that's a somewhat orthogonal issue to getting a Canvas or
CanvasSurface in a worker. At least for the WebGL case, WebGL already
defines ways to share data safely between contexts so that use case is
already covered.

For a 2d context I can certainly imagine lots of ways. Some fast, some slow

slow) pass the result of toDataURL back

slow) pass an ImageData.data back (transfer of ownership)

fast) some how use a CanvasSurface as the source of a video tag

maybe there are some other ideas? But, even if we didn't resolve this, it's
still massively useful, at least for WebGL, to be able to instantiate a
WebGL context in a worker.





>
> #1 is an issue with any proposal that puts a 2d context in a worker.
>
> #2 is only an issue if we do the image loads in workers somehow; if we
> wanted to just rasterize the SVG and pass the resulting data to the worker
> there would be no problem, obviously.
>
> For #3, being able to transfer a canvas context from the page to a worker
> but leave it painting to the canvas may still be desirable...
>
> -Boris
>
>

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 8:21 PM, Charles Pritchard wrote:

SVG and animated gif would render the same as it would in an
Image that has not been added to the dom or is otherwise display: none.


I'm not sure that would be workable in a worker for SVG, for the same 
reasons that responseXML is not available in workers: it involves DOM 
elements.  Unless the SVG rasterization happens on the main thread under 
the hood and the raster data is then sent over to the worker.  This 
might have ... surprising performance characteristics.


-Boris

Re: exposing CANVAS or something like it to Web Workers

[James Robinson]

On Mon, May 14, 2012 at 5:03 PM, Boris Zbarsky  wrote:

> On 5/14/12 7:56 PM, Glenn Maynard wrote:
>
>> A tricky bit: you need to know which element to sync to, so the browser
>> knows which monitor's vsync to use.  According to [1] only WebKit's
>> requestAnimationFrame actually takes an element.  (That's surprising;
>> this seems obvious.
>>
>
> Does WebKit actually use the element to determine vsync?  How do they
> handle cases when the element spans monitors?


> As far as I know WebKit's implementation uses the element to optimize out
> callbacks when the element is not visible, but that's it.
>

The element isn't used for anything currently in WebKit.  "Which vsync" is
determined by the monitor the tab/window/whatever lands on.  When this
spans monitors, something random happens (there aren't many good options).

>
> Note that Gecko, for example, does not tie requestAnimationFrame callbacks
> to vsync.  I can't speak for other UAs.


I think you'll want to.

- James


>
>
>  I mention this because
>> this method would need to accept a context in lieu of an element
>>
>
> What would the context be used for?
>
> -Boris
>
>

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On May 14, 2012, at 5:03 PM, Gregg Tavares (勤)  wrote:

> 
> 
> On Mon, May 14, 2012 at 4:42 PM, Jonas Sicking  wrote:
> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
> > On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  wrote:
> >
> >> I'd like to work on exposing something like CANVAS to web workers.
> >>
> >> Ideally how over it works I'd like to be able to
> >>
> >> *) get a 2d context in a web worker
> >
> >
> > I'd recommend not trying to tackle 2d and 3d contexts at once, and only
> > worrying about WebGL to start.
> >
> > Another issue: rendering in a worker thread onto a canvas which is displayed
> > in the main thread.  This needs to be solved in a way that doesn't cause the
> > asynchronous nature of what's happening to be visible to scripts.  toDataURL
> > and toBlob would probably need to be prohibited on the canvas element.  I'm
> > not sure what the actual API would look like.
> 
> If/when we do this, I think it should be done in such a way that the
> main window can't access the canvas object at all. Similar to what
> happens when an ArrayBuffer is transferred to a Worker using
> structured cloning. Once a canvas is transferred to a Worker, any
> access to it should throw or return null/0/"". If you want to transfer
> pixel data to the main thread, it seems less racy to do that by
> getting the pixel data in the Worker which owns the canvas and then
> transfer that to the main thread using postMessage.
> 
> How about separating the canvasy parts of canvas from canvas and the imagy 
> parts of image from image.
> 
> In other words, Imagine canvas is implemented like this
> 
> class Canvas : public HTMLElement {
>   private:
> CanvasSurface* m_surface;  // everything about canvas that is not 
> HTMLElement
> };
> 
> And that Image is similarly implemented as
> 
> class Image : public HTMLElement {
>   private:
> Picture* m_picture;  // everything about Image that is not HTMLElement
> }
> 
> now imagine you can instantiate inner implementation of these things. The 
> parts that are not HTMLElement
> 
> var canvasSurface = new CanvasSurface();
> var ctx = canvasSurface.getContext("2d");
> var pic = new Picture;
> pic.src = "http://someplace.com/someimage.jpg";;
> pic.onload = function() {
>ctx.drawImage(pic, 0, );
> }
> 
> Let's assume you can instantiate these things in either the page or a worker. 
> Nether can be transfered.
> 
> Would that work? What problems would that have?
> 

Seems fine. If it works with XHR I'd imagine it'll work with Picture.

Blob uri behavior may come up as an issue. I believe they should "just work". 
SVG and animated gif would render the same as it would in an Image that has not 
been added to the dom or is otherwise display: none.

I suspect we may have blob Uris corresponding to live video streams. If those 
come about and work with onload I'd expect them to work here (simply grabbing 
the last/first available frame).






>  
> 
> > This would also require some
> > equivalent to requestAnimationFrame in the worker thread.
> 
> Agreed!
> 
> / Jonas
> 

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 8:03 PM, Gregg Tavares (勤) wrote:

var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext("2d");
var pic = new Picture;
pic.src = "http://someplace.com/someimage.jpg";;
pic.onload = function() {
ctx.drawImage(pic, 0, );
}

Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.

Would that work? What problems would that have?


Two things that come to mind immediately:

1)  Various canvas 2d context methods depend on the styles of the canvas 
to define how they actually behave.  Clearly this would need some sort 
of changes for Workers anyway; the question is what those changes would 
need to be.


2)  How (or maybe whether) this should work with pic.src = 
"http://someplace.com/someimage.svg"; is an interesting question.


3)  This leaves open the question of how to get data from the worker 
back to the main thread to paint there.


#1 is an issue with any proposal that puts a 2d context in a worker.

#2 is only an issue if we do the image loads in workers somehow; if we 
wanted to just rasterize the SVG and pass the resulting data to the 
worker there would be no problem, obviously.


For #3, being able to transfer a canvas context from the page to a 
worker but leave it painting to the canvas may still be desirable...


-Boris

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On May 14, 2012, at 4:42 PM, Jonas Sicking  wrote:

> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
>> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  wrote:
>> 
>>> I'd like to work on exposing something like CANVAS to web workers.
>>> 
>>> Ideally how over it works I'd like to be able to
>>> 
>>> *) get a 2d context in a web worker
>> 
>> 
>> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
>> worrying about WebGL to start.
>> 
>> Another issue: rendering in a worker thread onto a canvas which is displayed
>> in the main thread.  This needs to be solved in a way that doesn't cause the
>> asynchronous nature of what's happening to be visible to scripts.  toDataURL
>> and toBlob would probably need to be prohibited on the canvas element.  I'm
>> not sure what the actual API would look like.
> 
> If/when we do this, I think it should be done in such a way that the
> main window can't access the canvas object at all. Similar to what
> happens when an ArrayBuffer is transferred to a Worker using
> structured cloning. Once a canvas is transferred to a Worker, any
> access to it should throw or return null/0/"". If you want to transfer
> pixel data to the main thread, it seems less racy to do that by
> getting the pixel data in the Worker which owns the canvas and then
> transfer that to the main thread using postMessage.
> 
>> This would also require some
>> equivalent to requestAnimationFrame in the worker thread.
> 
> Agreed!
> 
> / Jonas



I'm a bit lost-- wouldn't we just postMessage from the document over to the web 
worker when we want a refresh?

I agree that we ought to be transferring pixel data not Canvas contexts; with 
the possible exception of CSS context.

We could just create new canvas instances inside the worker thread. I'd still 
prefer to clone CanvasPattern as a means of transferring paint over to the 
worker, though sending pixel data would work too.

I heard Picture come up-- it seems like that object might have additional 
semantics for high resolution alternatives that may need to be considered.

-Charles

Re: exposing CANVAS or something like it to Web Workers

[勤]

On Mon, May 14, 2012 at 4:42 PM, Jonas Sicking  wrote:

> On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
> > On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  > wrote:
> >
> >> I'd like to work on exposing something like CANVAS to web workers.
> >>
> >> Ideally how over it works I'd like to be able to
> >>
> >> *) get a 2d context in a web worker
> >
> >
> > I'd recommend not trying to tackle 2d and 3d contexts at once, and only
> > worrying about WebGL to start.
> >
> > Another issue: rendering in a worker thread onto a canvas which is
> displayed
> > in the main thread.  This needs to be solved in a way that doesn't cause
> the
> > asynchronous nature of what's happening to be visible to scripts.
>  toDataURL
> > and toBlob would probably need to be prohibited on the canvas element.
>  I'm
> > not sure what the actual API would look like.
>
> If/when we do this, I think it should be done in such a way that the
> main window can't access the canvas object at all. Similar to what
> happens when an ArrayBuffer is transferred to a Worker using
> structured cloning. Once a canvas is transferred to a Worker, any
> access to it should throw or return null/0/"". If you want to transfer
> pixel data to the main thread, it seems less racy to do that by
> getting the pixel data in the Worker which owns the canvas and then
> transfer that to the main thread using postMessage.
>

How about separating the canvasy parts of canvas from canvas and the imagy
parts of image from image.

In other words, Imagine canvas is implemented like this

class Canvas : public HTMLElement {
  private:
CanvasSurface* m_surface;  // everything about canvas that is not
HTMLElement
};

And that Image is similarly implemented as

class Image : public HTMLElement {
  private:
Picture* m_picture;  // everything about Image that is not HTMLElement
}

now imagine you can instantiate inner implementation of these things. The
parts that are not HTMLElement

var canvasSurface = new CanvasSurface();
var ctx = canvasSurface.getContext("2d");
var pic = new Picture;
pic.src = "http://someplace.com/someimage.jpg";;
pic.onload = function() {
   ctx.drawImage(pic, 0, );
}

Let's assume you can instantiate these things in either the page or a
worker. Nether can be transfered.

Would that work? What problems would that have?





>
> > This would also require some
> > equivalent to requestAnimationFrame in the worker thread.
>
> Agreed!
>
> / Jonas
>

Re: exposing CANVAS or something like it to Web Workers

[Boris Zbarsky]

On 5/14/12 7:56 PM, Glenn Maynard wrote:

A tricky bit: you need to know which element to sync to, so the browser
knows which monitor's vsync to use.  According to [1] only WebKit's
requestAnimationFrame actually takes an element.  (That's surprising;
this seems obvious.


Does WebKit actually use the element to determine vsync?  How do they 
handle cases when the element spans monitors?


As far as I know WebKit's implementation uses the element to optimize 
out callbacks when the element is not visible, but that's it.


Note that Gecko, for example, does not tie requestAnimationFrame 
callbacks to vsync.  I can't speak for other UAs.



I mention this because
this method would need to accept a context in lieu of an element


What would the context be used for?

-Boris

Re: exposing CANVAS or something like it to Web Workers

[Glenn Maynard]

On Mon, May 14, 2012 at 6:42 PM, Jonas Sicking  wrote:

> If/when we do this, I think it should be done in such a way that the
> main window can't access the canvas object at all. Similar to what
> happens when an ArrayBuffer is transferred to a Worker using
> structured cloning. Once a canvas is transferred to a Worker, any
> access to it should throw or return null/0/"".


Sort of.  You want to disable the methods on Canvas--toDataURL, toBlob and
getContext--but you don't want to disable inherited methods (you should
still be able to access eg. parentNode).  I think handling this on a
method-by-method basis is fine, since there aren't many of them--just those
three, I think.

I still don't have much intuition about how to do this, but another
thought: assuming you create a context in the worker which is attached to
the canvas, you need to be able to create new contexts in the future if the
worker dies, since workers can be killed by the browser at any time.  A
mechanism that says "you can create a worker-context for a canvas, but then
the canvas element is locked to that worker" isn't good enough (but it's
fine to say "you can only have one worker associated with a canvas at a
time").

If you want to transfer
> pixel data to the main thread, it seems less racy to do that by
> getting the pixel data in the Worker which owns the canvas and then
> transfer that to the main thread using postMessage.
>

Right.


>  > This would also require some
> > equivalent to requestAnimationFrame in the worker thread.
>
> Agreed!
>

A tricky bit: you need to know which element to sync to, so the browser
knows which monitor's vsync to use.  According to [1] only WebKit's
requestAnimationFrame actually takes an element.  (That's surprising; this
seems obvious.  Are multi-monitor systems always vsync-locked to each
other, which would make this not matter?)  I mention this because this
method would need to accept a context in lieu of an element (assuming
whatever mechanism we land on it results in having a WebGLContext in the
worker but no HTMLCanvasElement, whcih seems likely).

This in mind, it would be good if all Canvas context types had a common
base class, even if it's an empty interface.  The return value of
HTMLCanvasElement.getContext could be changed to it, instead of object.

[1] https://developer.mozilla.org/en/DOM/window.requestAnimationFrame

-- 
Glenn Maynard

Re: exposing CANVAS or something like it to Web Workers

[Jonas Sicking]

On Mon, May 14, 2012 at 3:28 PM, Glenn Maynard  wrote:
> On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  wrote:
>
>> I'd like to work on exposing something like CANVAS to web workers.
>>
>> Ideally how over it works I'd like to be able to
>>
>> *) get a 2d context in a web worker
>
>
> I'd recommend not trying to tackle 2d and 3d contexts at once, and only
> worrying about WebGL to start.
>
> Another issue: rendering in a worker thread onto a canvas which is displayed
> in the main thread.  This needs to be solved in a way that doesn't cause the
> asynchronous nature of what's happening to be visible to scripts.  toDataURL
> and toBlob would probably need to be prohibited on the canvas element.  I'm
> not sure what the actual API would look like.

If/when we do this, I think it should be done in such a way that the
main window can't access the canvas object at all. Similar to what
happens when an ArrayBuffer is transferred to a Worker using
structured cloning. Once a canvas is transferred to a Worker, any
access to it should throw or return null/0/"". If you want to transfer
pixel data to the main thread, it seems less racy to do that by
getting the pixel data in the Worker which owns the canvas and then
transfer that to the main thread using postMessage.

> This would also require some
> equivalent to requestAnimationFrame in the worker thread.

Agreed!

/ Jonas

Re: exposing CANVAS or something like it to Web Workers

[Glenn Maynard]

On Mon, May 14, 2012 at 3:01 PM, Gregg Tavares (勤)  wrote:

> I'd like to work on exposing something like CANVAS to web workers.
>
> Ideally how over it works I'd like to be able to
>
> *) get a 2d context in a web worker
>

I'd recommend not trying to tackle 2d and 3d contexts at once, and only
worrying about WebGL to start.

Another issue: rendering in a worker thread onto a canvas which is
displayed in the main thread.  This needs to be solved in a way that
doesn't cause the asynchronous nature of what's happening to be visible to
scripts.  toDataURL and toBlob would probably need to be prohibited on the
canvas element.  I'm not sure what the actual API would look like.  This
would also require some equivalent to requestAnimationFrame in the worker
thread.

*) download images in a web worker and the images with both 2d contexts and
> WebGL contexts


Don't forget to point people to the recent discussion on this, so the
conversation doesn't reboot:

https://www.khronos.org/webgl/public-mailing-list/archives/1205/msg00059.html

In summary, the idea is to provide a much more basic interface for images,
which would represent an image in an opaque, non-DOM way, to make it
implementable in workers; you'd retrieve it with a method on
HTMLImageElement after the image is completely available.  It'd be
structured clonable, to hand it to workers.  There should also be an XHR2
responseType mode to retrieve an object like this directly, to allow
loading images from workers (again without using the DOM), since workers
shouldn't need to rely on a helper in the UI thread just to load textures.

This seems like it would solve this sub-problem pretty well, at least.


On Mon, May 14, 2012 at 3:16 PM, Anne van Kesteren  wrote:

> Have we gotten any further with use cases? See
>
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144
> for an old use case thread that went nowhere. Or
>
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274
> where you requested general DOM access which cannot be implemented
> everywhere.
>

The use case is being able to draw without blocking the UI thread.  WebGL,
like OpenGL, tries to be nonblocking by queuing draw operations, but it's
impossible to do this in all cases: as soon as you call any method that
returns data (eg. getError), WebGL has to synchronously flush the draw
queue.  This isn't as severe as synchronous XHR, but it may be done every
frame--a 10ms delay at 60Hz is going to have visible effects on the UI.

WebGL is also useful for computational tasks, which may perform much longer
drawing operations.  If you're doing this, it's even more important that it
not block the UI thread.

Compiling shaders can also take a long time for complex shaders.  Since
this is almost always followed by a synchronous call to check the result,
it's currently hard to load shaders without affecting the browser UI.

-- 
Glenn Maynard

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 1:16 PM, Anne van Kesteren wrote:

On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤)  wrote:

I'd like to work on exposing something like CANVAS to web workers.

Ideally how over it works I'd like to be able to

*) get a 2d context in a web worker
*) get a WebGL context in a web worker
*) download images in a web worker and the images with both 2d contexts and
WebGL contexts

Any thoughts?

Have we gotten any further with use cases? See
http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144
for an old use case thread that went nowhere. Or


1. Speeding up "onmousemove"-based drawing:
In my drawing projects (based on mouse/pen input), we lose mouse events 
/ pen pressure information when the main thread is busy rendering what 
the user is drawing.

Processing the drawing commands off-thread would lighten the load.

2. Avoiding blocking during redrawing of complex scenes or pre-rendering 
of animations.


With complex scenes, where we're repainting, we don't particularly want 
to block the main thread while a scene is loading.
But, we'd also like to use as much horsepower as the user's machine can 
lend.


A complex scene may block for a few seconds -- we can of course use 
green threading approaches, but that adds quite a bit of extra 
guess-work and

does not fully exploit the user's machine for speed.

Re: exposing CANVAS or something like it to Web Workers

[Jonas Sicking]

On Mon, May 14, 2012 at 1:16 PM, Anne van Kesteren  wrote:
> On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤)  wrote:
>> I'd like to work on exposing something like CANVAS to web workers.
>>
>> Ideally how over it works I'd like to be able to
>>
>> *) get a 2d context in a web worker
>> *) get a WebGL context in a web worker
>> *) download images in a web worker and the images with both 2d contexts and
>> WebGL contexts
>>
>> Any thoughts?
>
> Have we gotten any further with use cases? See
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144
> for an old use case thread that went nowhere. Or
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274
> where you requested general DOM access which cannot be implemented
> everywhere.

One of the strongest use-cases we have found within mozilla is to
enable compiling applications written in other languages and for other
platforms to the web platform. For example emscripten [1] supports
compiling C/C++/OpenGL programs to javascript + WebGL.

Such programs generally use things like synchronous IO and have their
own event loops, thus not returning to the platform event loop.

So a good strategy for tools like emscripten is to create code which
is intended to run in a worker. That way you don't have to return to
the event loop, and you have access to things like full featured
synchronous XHR (including support for .responseType) without
degrading the user experience.

However a big problem is that you can't do graphics inside a worker.
It would be great to expose canvas since that is the graphics model
these programs have today.

[1] https://github.com/kripken/emscripten

/ Jonas

Re: exposing CANVAS or something like it to Web Workers

[Charles Pritchard]

On 5/14/2012 1:01 PM, Gregg Tavares (勤) wrote:

I'd like to work on exposing something like CANVAS to web workers.

Ideally how over it works I'd like to be able to

*) get a 2d context in a web worker
*) get a WebGL context in a web worker
*) download images in a web worker and the images with both 2d 
contexts and WebGL contexts


Any thoughts?


As far as implementation, I'd love to be able to pass webkit's 
document.getCSSCanvasContext('2d') around.

It seems like a safe place to experiment.

I can get a lot done with CanvasPattern as a transferable, without 
needing to add Image (or video) into the worker context.


Notes:
1. getCSSCanvasContext is non-standard. It works with CSS image 
-webkit-canvas.
2. I heard that a more generic "-moz-element()" paint server is supposed 
to replace -webkit-canvas in time.
3. Passing the CSS canvas context would let me render off-frame and 
update a canvas visible on the document automatically.


Canvas -should- have toBlob and a typed array buffer for ImageData.
They are both useful for passing image data back to the main frame.


From my experience with WebGL, I think it should be considered with 
added care and lower priority.
There are stability, speed and memory issues. WebGL in workers seems to 
augment 2d, are there other big benefits?



-Charles

Re: exposing CANVAS or something like it to Web Workers

[Anne van Kesteren]

On Mon, May 14, 2012 at 10:01 PM, Gregg Tavares (勤)  wrote:
> I'd like to work on exposing something like CANVAS to web workers.
>
> Ideally how over it works I'd like to be able to
>
> *) get a 2d context in a web worker
> *) get a WebGL context in a web worker
> *) download images in a web worker and the images with both 2d contexts and
> WebGL contexts
>
> Any thoughts?

Have we gotten any further with use cases? See
http://lists.w3.org/Archives/Public/public-whatwg-archive/2010Mar/thread.html#msg144
for an old use case thread that went nowhere. Or
http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Feb/thread.html#msg274
where you requested general DOM access which cannot be implemented
everywhere.


-- 
Anne — Opera Software
http://annevankesteren.nl/
http://www.opera.com/