Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Kenneth Russell]

On Wed, Feb 10, 2016 at 12:06 PM, Justin Novosad  wrote:

> On Wed, Feb 10, 2016 at 2:29 PM, Boris Zbarsky  wrote:
>
> > On 2/10/16 1:25 PM, Domenic Denicola wrote:
> >
> >> In new JavaScript-only APIs we've made the decision to move away from
> the
> >> potentially-confusing HTML style crossOrigin enums in favor of the
> >> RequestCredentials enum used by Fetch:
> >> https://fetch.spec.whatwg.org/#requestcredentials. You can see this in
> >> e.g. https://github.com/whatwg/html/pull/608 where I chose the same
> >> initial crossOrigin design and Anne convinced me to move to
> credentials. I
> >> imagine we'll continue to use crossorigin="" and corresponding reflected
> >> crossOrigin IDL attributes for any HTML elements, but for JS-only APIs
> >> RequestCredentials is the way to go.
> >>
> >
> > That's not _quite_ the same thing.  The HTML setup basically lets you
> > specify one of:
> >
> > 1)  No CORS (attr not set).
> > 2)  CORS, RequestCredentials == "include" (crossrigin="use-credentials")
> > 3)  CORS, RequestCredentials == "same-origin" (any other attr value)
> >
> > Note that in the pull request you reference your default was not actually
> > any of those situations, as far as I can tell, so I agree that using
> > "crossOrigin" there was not a good fit.  But for the ImageBitmap case, we
> > do want to support case 1 above, at least assuming tainted ImageBitmap
> is a
> > thing.  If it's not, then I agree that just a RequestCredentials value is
> > probably sufficient and all the loads involved should use CORS.
> >
>
> Tainted ImageBitmaps is a thing: https://github.com/whatwg/html/pull/385
> That said, this is not one of the options we intend to move forward with in
> the near term (It is not in the highly desired list). It is not clear that
> the feature is even needed. I am no CORS expert, but I think you can get an
> untainted ImageBitmap from a cross-origin image by using: XHR (with
> credentials) -> blob -> createImageBitmap
>

Agreed -- I don't think the ImageBitmap constructor from URL, or the
crossOrigin ImageBitmap constructor argument, are needed any more. It would
be good to fully implement ImageBitmap (including asynchronous decoding),
plus some of the WebGL-related constructor arguments, and verify that the
texture uploads are much faster. At that point let's delete the crossOrigin
ImageBitmap constructor argument, and the proposal for creating
ImageBitmaps from URLs.

-Ken




>
>
> > That said, the actual phrasing around "crossOrigin" in
> > https://wiki.whatwg.org/wiki/ImageBitmap_Options doesn't make much sense
> > (e.g. it in fact is not a "CORS settings attribute" because it's not a
> > markup attribute at all).  But we can wordsmith it better once we agree
> on
> > what we actually want it to do.
> >
> > -Boris
> >
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Kenneth Russell]

On Wed, Feb 10, 2016 at 10:25 AM, Domenic Denicola  wrote:

> From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Xida
> Chen
>
> > We intend to push this feature forward in Blink, particularly we intend
> to
> > spec and implement the "Strongly desired options" listed on the Whatwg
> > proposal page. We would appreciate comments and suggestions on the
> > proposal.
> >
> > Any thoughts or objections before we start drafting a change to the spec?
>
> Exciting stuff!
>

Agreed -- thanks Xida for pushing this feature forward. It'll significantly
improve texture handling performance in WebGL applications.



>
> > 'none': Do not change orientation.
>
> Does this mean "disregard image orientation metadata" or does it mean "do
> not change orientation from what the metadata specifies"?
>
> > enum colorspaceConversion, default = 'default'
>
> Is there any precedent on the web platform (or in Open GL APIs, perhaps?)
> for whether "colorspace" is one word or two words? If two words, this
> should be colorSpaceConversion. Happy to defer to your expertise here.
>

The WebGL spec has been using a single word for this, but two words are
probably more correct.



>
> > 'default': Implementation-specific behavior, possibly optimized for the
> implementation's graphics framework.
>
> This is a bit unfortunate as an interop hazard. I suppose it's no worse
> than today though. Has this been discussed in the past? Apologies, I was
> only able to skim the previous email thread.
>

This provides the functionality of the existing
UNPACK_COLORSPACE_CONVERSION_WEBGL flag in the WebGL spec. By default
images uploaded to WebGL are decoded "however the browser does it" -- which
isn't tightly specified -- but most WebGL apps that draw images just do
this and get correct results.

However, in special cases and for sophisticated apps, it's absolutely
necessary to get the raw data that's in the image, which is why there has
to be a switch to turn it off.

-Ken




>
> ---
>
> This comment is on something outside the "strongly desired features"
> section so probably not relevant to your immediate work. Just wanted to
> mention it.
>
> > DOMString? crossOrigin
>
> In new JavaScript-only APIs we've made the decision to move away from the
> potentially-confusing HTML style crossOrigin enums in favor of the
> RequestCredentials enum used by Fetch:
> https://fetch.spec.whatwg.org/#requestcredentials. You can see this in
> e.g. https://github.com/whatwg/html/pull/608 where I chose the same
> initial crossOrigin design and Anne convinced me to move to credentials. I
> imagine we'll continue to use crossorigin="" and corresponding reflected
> crossOrigin IDL attributes for any HTML elements, but for JS-only APIs
> RequestCredentials is the way to go.
>

Re: [whatwg] Opinions on window.console = "foo", and other oddities of window.console

[Kenneth Russell]

The test harnesses used by the working group which I chair rely on being
able to replace window.console. I'd like to see it continue to be mutable.
Thanks.


On Mon, Feb 8, 2016 at 3:25 PM, Domenic Denicola  wrote:

> As you may know, we now have a standard for the console object:
> http://console.spec.whatwg.org/
>
> One of the first issues we encountered while investigating console
> behavior is https://github.com/whatwg/console/issues/1, which is that
> browsers currently allow setting `window.console` (and presumably
> `self.console` in workers) to any value. This is pretty weird, but every
> tested browser (Firefox, Chrome, Edge, and Safari) follows it.
>
> Probably the default course of action here is to just spec this. But we
> wanted to check to see if any implementers thought this was weird, and
> wanted to experiment with e.g. removing the setter.
>
> Even if we do go this route, we have another weirdness:
>
> - Firefox has an accessor descriptor (get/set), Chrome, Safari, and Edge
> have a data descriptor (value)
> - Firefox, Chrome, and Edge are enumerable; Safari is not enumerable
>
> I guess Chrome and Safari's data descriptors are not surprising, given
> that their bindings infrastructure for Window isn't fully Web
> IDL-compatible yet. But Edge is pretty surprising. Travis, do you have any
> info on that?
>
> So I think the plan of record is: `attribute any console`, with prose
> describing how the getter returns "the window's console object" which is
> initially set to a new instance of Console, but the setter can change it to
> any value. This means accessor descriptors and enumerable, since that's how
> IDL works. If anyone thinks this is bad, let us know. Otherwise the
> standard will be updated any day now in that direction.
>

Re: [whatwg] OffscreenCanvas from a worker and synchronization with DOM/CSS

[Kenneth Russell]

That's right. From discussions with Mozilla I don't think that code path's
enabled in their implementation yet, but it will be (as well as in Chrome's
future implementation of the same spec).


On Sat, Jan 23, 2016 at 12:40 PM, Gregg Tavares  wrote:

> Never mind me. For whatever reason my mind blanked out.
>
> You can transfer to the main thread and then apply to a canvas.
>

Re: [whatwg] High-density canvases

[Kenneth Russell]

On Wed, Apr 15, 2015 at 1:46 PM, Dean Jackson  wrote:
>
> On 16 Apr 2015, at 6:42 am, Elliott Sprehn  wrote:
>
> 3. Accessing rendered pixel size is layout-inducing. To avoid layout
>>
>> thrashing, we should consider making this an asynchronous getter (e.g.
>> asyncGetBoundignClientRect). This would also prevent renderedsizechanged
>> events from firing from within the evaluation of the
>> renderedPixelWidth/Height
>> attributes, which is weird.
>
>
> renderedsizechanged feels like it's related to the general per element
> resize event problem. It'd be unfortunate to add an event to the browser
> specifically for canvas instead of solving the general problem. In that case
> authors will start resorting to hacks where they insert canvases into the
> page to listen for resizes (we've seen this with other events like
> overflowchanged).
>
> Perhaps we should add per element resize events and spec that they fire
> after the layout, but before the paint, during normal frame creation. That
> does mean you can cause an infinite loop if your resize handler keeps
> mutating the page, but there's so many other ways to cause that already.
>
>
> +1

If that's a viable option let's do that. It would solve many
longstanding problems.

-Ken

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Sun, Apr 12, 2015 at 2:41 PM, Robert O'Callahan  wrote:
> On Sat, Apr 11, 2015 at 1:49 PM, Kenneth Russell  wrote:
>>
>> Suppose src="myimage.png" is set on an image element and then, while
>> it is downloading, an ImageBitmap is transferred into it.
>
>
> This should behave as if "src" was removed just before the ImageBitmap is
> transferred into it.
>
> In spec terms, I think we'd define a new state flag "has an ImageBitmap".
> Transferring an ImageBitmap into an  element would set that flag and
> trigger "update the image data". After step 2 of "update the image data",
> when the flag is set, we'd synchronously do what "update the image data"
> would do with no source: clear "last selected source" (step 4), and do steps
> 9.1 and 9.2 (but never fire the "error" event). Then, transferring the
> ImageBitmap would do the "load complete" work of step 14: set the
> "completely available" state, set the image data, and add the image to the
> list of available images.
>
> Note that this is also exactly what we'd need to do to support srcObject on
> , which would be nice to have so you can render a Blob with an
> HTMLImageElement without dealing with the annoying createObjectURL lifetime
> issues.
>
> Thus:
>
>> 1. What is displayed in the webpage as myimage.png is downloading?
>
>
> As normal before an ImageBitmap is transferred, the ImageBitmap afterward.
>
>> 2. Do the downloaded bits overwrite what you transferred or do we stop
>> the download?
>
>
> Stop the download.
>
>>
>> 3. Are onload and onerror events fired? This question applies both to
>> the in-progress download and to the transferred-in ImageBitmap.
>
>
> No.
>
>>
>> 4. What should the 'complete' property and 'currentSrc' attribute reflect?
>
>
> True and the empty string respectively.
>
>>
>> 5. Can the developer go back to having the img element contain what
>> was set on 'src' and not what was transferred in?
>
>
> As written, yes, by performing another relevant mutation to trigger "update
> the image data" again.
>
>> srcset and the picture element make the situation even more complex.
>
>
> I think the above covers it.

Thanks for these answers.

They sound reasonable to me from a complexity standpoint, though the
needed updates to the  tag's spec sound a little involved. I have
another concern which is mentioned below.


>> In comparison, displaying ImageBitmaps with a custom canvas context
>> offers simple semantics and avoids other problems raised on this
>> thread like requiring a layout, or firing events, upon recipt of a new
>> ImageBitmap.
>
>
> AFAIK there aren't any issues with events. The issue with layout is simply
> whether or not we want the element receiving an ImageBitmap to have the
> expected intrinsic size. AFAICT for unnecessary layouts to occur would
> require repeated ImageBitmap transfers into the same HTMLImageElement, where
> the ImageBitmaps have varying sizes and the element has auto 'width' or
> 'height', but the author actually intends that the images all be scaled or
> cropped to the same size, and doesn't notice this is not happening. Is there
> a more realistic scenario?

If repeatedly transferring ImageBitmaps of the same size into the same
 won't cause repeated re-layouts, that alleviates my concern
about efficiency of using images as the output path. I expect that a
single OffscreenCanvas will be used to produce output to multiple
areas on the page (whether into images or canvases) but that it will
be sized large enough so that all of the ImageBitmaps it produces will
cover the largest of those areas, avoiding repeatedly resizing the
OffscreenCanvas.

There is one other problem which has come up repeatedly in canvas
applications: needing to be able to accurately measure the number of
device pixels covered by a canvas, so pixel-accurate rendering can be
done. https://wiki.whatwg.org/wiki/CanvasRenderedPixelSize addresses
this and it's currently being implemented in Chromium. There is no
such mechanism for images. It would be necessary to understand exactly
how many device pixels the output image is covering in the document so
that the OffscreenCanvas can be sized appropriately. Using a canvas
element for the display of these ImageBitmaps avoids this problem. It
also makes the API more symmetric; a canvas-like object produces the
ImageBitmap, and a canvas element is used to view it. Do you have any
suggestion for how this issue would be solved with ?

-Ken

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Fri, Apr 10, 2015 at 2:33 PM, Robert O'Callahan  wrote:
> On Sat, Apr 11, 2015 at 2:18 AM, Justin Novosad  wrote:
>>
>> Riddle me this: What would be the value an HTMLImageElement's src
>> attribute
>> after an ImageBitmap is transferred in? A data URL? What if the
>> ImageBitmap
>> was sourced from an actual resource with a URL, would we pipe that
>> through?  In cases where there is no tractable URL (ex: ImageBitmap was
>> grabbed fram a tainted canvas), then what?
>
>
> We have the same issue for HTMLMediaElement.srcObject:
> http://dev.w3.org/2011/webrtc/editor/archives/20140619/getusermedia.html#widl-HTMLMediaElement-srcObject
> The current approach is to leave 'src' unchanged, and the spec says that if
> there is a current srcObject (or ImageBitmap in our case), render that and
> ignore 'src'.

Here are some more questions, some of which are from a colleague. I'm
not trying to claim credit for thinking of them, but want to make it
clear that there are several more ambiguities with attempting to use
an image element to display ImageBitmaps.

Suppose src="myimage.png" is set on an image element and then, while
it is downloading, an ImageBitmap is transferred into it.

1. What is displayed in the webpage as myimage.png is downloading?
2. Do the downloaded bits overwrite what you transferred or do we stop
the download?
3. Are onload and onerror events fired? This question applies both to
the in-progress download and to the transferred-in ImageBitmap.
4. What should the 'complete' property and 'currentSrc' attribute reflect?
5. Can the developer go back to having the img element contain what
was set on 'src' and not what was transferred in?

srcset and the picture element make the situation even more complex.

In comparison, displaying ImageBitmaps with a custom canvas context
offers simple semantics and avoids other problems raised on this
thread like requiring a layout, or firing events, upon recipt of a new
ImageBitmap.

-Ken

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Thu, Apr 9, 2015 at 6:25 PM, Tab Atkins Jr.  wrote:
> On Thu, Apr 9, 2015 at 5:23 PM, Kenneth Russell  wrote:
>> 1. An image element that didn't have a width or height explicitly
>> specified in the markup has an ImageBitmap transferred in. Will its
>> width and height attributes change? Will layout have to occur on the
>> page?
>
> Its width and height *attributes* won't change, but its *intrinsic*
> width and height will, to match the bitmap and density.  (Btw, the
> ImageBitmap feature needs to have a notion of density.)

I think the idea of introducing "density" to ImageBitmap needs a
completely separate thread. I don't understand all the ramifications,
but it looks to me like the fact that ImageBitmap's width and height
are specified in CSS pixels makes it impossible for users' code to
compute the exact number of pixels in the ImageBitmap. This is still a
problem for developers using  and
https://wiki.whatwg.org/wiki/CanvasRenderedPixelSize is intended to
solve it.

> Yes, layout occurs, assuming the size changed.
>
>> 2. The same image element has another ImageBitmap transferred in which
>> has a different width and height. Do the image's width and height
>> change? Will layout have to occur on the page?
>
> Yes and yes, assuming the size changed.

This is problematic. It's essential that displaying a newly produced
ImageBitmap be a cheap operation. Layout is expensive.


>> When an image's src is set to a URL and its width and height
>> attributes aren't set, the page is laid out again when the image is
>> loaded. For acceptable performance it's important that this not happen
>> when displaying a new ImageBitmap. Using a new canvas context type
>> solves this problem. The OffscreenCanvas proposal defines the
>> ImageBitmap as being rendered at the upper left corner of the canvas's
>> box with no scaling applied, and clipped to the box. It's not as
>> flexible as having the object-fit and object-position CSS properties
>> available, but will give developers explicit control over when they
>> want layout to happen, and still let them display their content as
>> they wish.
>
> Why not just use object-fit/position?  I think we support those.  No
> need to make new weird behavior when you can just opt into the desired
> behavior with existing standardized stuff.

Can the right settings of object-fit and object-position avoid the
layouts in the abovementioned scenarios?

-Ken

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Wed, Apr 8, 2015 at 8:46 AM, Ashley Gullen  wrote:
> (sorry for double email) Just one question about how requestAnimationFrame
> is expected to work. Could we get rAF added to workers? Would it be able to
> simply fire whenever it does on the main thread, as if it postMessage'd to
> the worker every time it was called but without the extra postMessage
> latency? To do that I guess workers need to be associated with a window. For
> normal workers this seems fine but if OffscreenCanvas is expected to work in
> shared or service workers this is a little tricker to define. Perhaps
> requestAnimationFrame could be a method on OffscreenCanvas and it fires
> based on the window the HTMLCanvasElement it came from is currently in? Then
> you have a convenient way to synchronise rendering with the window it will
> ultimately be displayed in, without having to know in advance which window
> that is.

A usable requestAnimationFrame will be needed in workers in order to
reliably do animation. I'm hoping that that can be a somewhat
orthogonal proposal to this one. There are two ways to use an
OffscreenCanvas and only one of them causes rendering results to be
pushed implicitly to an existing canvas on the main thread. (The other
way involves directly instantiating an OffscreenCanvas and using
postMessage to do some work on the main thread to display its
results.) requestAnimationFrame is typically triggered on the main
thread as early as possible after the previous frame's vsync interval,
and I'd expect that other workers' requestAnimationFrame callbacks
should be triggered in the same way. Dedicated workers are always
associated with a particular page which is displayed in a window, so
the connection to an on-screen window should always be there.

-Ken


> Ashley
>
>
>
> On 8 April 2015 at 16:36, Ashley Gullen  wrote:
>>
>> This looks like it will cover running a HTML5 game engine from a worker
>> very nicely and with little performance overhead. Good stuff!
>>
>> Ashley
>>
>>
>> On 8 April 2015 at 00:41, Kenneth Russell  wrote:
>>>
>>> On Wed, Mar 25, 2015 at 6:41 PM, Kenneth Russell  wrote:
>>> > On Wed, Mar 25, 2015 at 1:22 PM, Robert O'Callahan
>>> >  wrote:
>>> >> On Wed, Mar 25, 2015 at 11:41 PM, Anne van Kesteren 
>>> >> wrote:
>>> >>>
>>> >>> On Fri, Mar 20, 2015 at 11:15 PM, Robert O'Callahan
>>> >>>  wrote:
>>> >>> > My understanding is that the current consensus proposal for canvas
>>> >>> > in
>>> >>> > Workers is not what's in the spec, but this:
>>> >>> > https://wiki.whatwg.org/wiki/WorkerCanvas
>>> >>> > See "Canvas in Workers" threads from October 2013 for the
>>> >>> > discussion.
>>> >>> > svn
>>> >>> > is failing me but the CanvasProxy proposal in the spec definitely
>>> >>> > predates
>>> >>> > those threads.
>>> >>> >
>>> >>> > Ian, unless I'm wrong, it would be helpful to remove the
>>> >>> > CanvasProxy
>>> >>> > stuff
>>> >>> > from the spec to avoid confusion.
>>> >>> >
>>> >>> > That proposal contains WorkerCanvas.toBlob, which needs to be
>>> >>> > updated to
>>> >>> > use promises.
>>> >>>
>>> >>> There's also https://wiki.whatwg.org/wiki/WorkerCanvas2 it seems. It
>>> >>> would be interesting to know what the latest on this is.
>>> >>
>>> >>
>>> >> Right now that appears to be just a copy of WorkerCanvas with some
>>> >> boilerplate text added. I assume someone's working on it and will
>>> >> mention it
>>> >> on this list if/when they're ready to discuss it :-).
>>> >
>>> > Yes, apologies for letting it sit there in an incomplete state.
>>> > Recently feedback from more browser implementers was gathered about
>>> > the WorkerCanvas proposal. I will assemble it into WorkerCanvas2 and
>>> > follow up on this thread in a day or two.
>>>
>>> I apologize for taking so long to update this proposal, but it's now
>>> in a reasonable state:
>>> https://wiki.whatwg.org/wiki/OffscreenCanvas
>>>
>>> (Renamed per feedback from Anne -- thanks.)
>>>
>>> Please offer your feedback. Multiple browser implementers agreed on
>>> this form of the API at a recent face-to-face meeting of the WebGL
>>> working group, and the proposal should be able to address multiple use
>>> cases. I'm very interested in prototyping it to see if it can offer
>>> better parallelism.
>>>
>>> -Ken
>>>
>>>
>>>
>>> > -Ken
>>> >
>>> >
>>> >> Rob
>>> >> --
>>> >> oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso
>>> >> oaonogoroyo
>>> >> owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
>>> >> osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo
>>> >> owohooo
>>> >> osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o
>>> >> o‘oRoaocoao,o’o
>>> >> oioso
>>> >> oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo
>>> >> oaonoyooonoeo
>>> >> owohooo
>>> >> osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono
>>> >> odoaonogoeoro
>>> >> ooofo
>>> >> otohoeo ofoioroeo ooofo ohoeololo.
>>
>>
>

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Tue, Apr 7, 2015 at 6:42 PM, Robert O'Callahan  wrote:
> On Wed, Apr 8, 2015 at 11:41 AM, Kenneth Russell  wrote:
>>
>> I apologize for taking so long to update this proposal, but it's now
>> in a reasonable state:
>> https://wiki.whatwg.org/wiki/OffscreenCanvas
>>
>> (Renamed per feedback from Anne -- thanks.)
>>
>> Please offer your feedback. Multiple browser implementers agreed on
>> this form of the API at a recent face-to-face meeting of the WebGL
>> working group, and the proposal should be able to address multiple use
>> cases. I'm very interested in prototyping it to see if it can offer
>> better parallelism.
>
>
> It seems fine.

Great.

> As far as I can tell, the only significant difference between
> this and WorkerCanvas is using an HTMLCanvasElement "bitmaprenderer" as a
> zero-copy way to render an ImageBitmap, instead of HTMLImageElement.

That's the main difference. Another is that OffscreenCanvas can be
directly instantiated from both the main thread and workers.

>Can you explain what the problem was with using HTMLImageElement?

One browser implementer pointed out that HTMLImageElement has several
complex internal states related to loading, and they do not want to
conflate that with the display of ImageBitmaps. I've documented this
in the OffscreenCanvas proposal. (I originally thought Canvas had more
complex internal state, but now think that it can be easier to define
new behavior against a new canvas context than .)

-Ken


> Rob
> --
> oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo
> owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
> osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo
> owohooo
> osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o
> oioso
> oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo
> owohooo
> osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro
> ooofo
> otohoeo ofoioroeo ooofo ohoeololo.

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Wed, Mar 25, 2015 at 6:41 PM, Kenneth Russell  wrote:
> On Wed, Mar 25, 2015 at 1:22 PM, Robert O'Callahan  
> wrote:
>> On Wed, Mar 25, 2015 at 11:41 PM, Anne van Kesteren 
>> wrote:
>>>
>>> On Fri, Mar 20, 2015 at 11:15 PM, Robert O'Callahan
>>>  wrote:
>>> > My understanding is that the current consensus proposal for canvas in
>>> > Workers is not what's in the spec, but this:
>>> > https://wiki.whatwg.org/wiki/WorkerCanvas
>>> > See "Canvas in Workers" threads from October 2013 for the discussion.
>>> > svn
>>> > is failing me but the CanvasProxy proposal in the spec definitely
>>> > predates
>>> > those threads.
>>> >
>>> > Ian, unless I'm wrong, it would be helpful to remove the CanvasProxy
>>> > stuff
>>> > from the spec to avoid confusion.
>>> >
>>> > That proposal contains WorkerCanvas.toBlob, which needs to be updated to
>>> > use promises.
>>>
>>> There's also https://wiki.whatwg.org/wiki/WorkerCanvas2 it seems. It
>>> would be interesting to know what the latest on this is.
>>
>>
>> Right now that appears to be just a copy of WorkerCanvas with some
>> boilerplate text added. I assume someone's working on it and will mention it
>> on this list if/when they're ready to discuss it :-).
>
> Yes, apologies for letting it sit there in an incomplete state.
> Recently feedback from more browser implementers was gathered about
> the WorkerCanvas proposal. I will assemble it into WorkerCanvas2 and
> follow up on this thread in a day or two.

I apologize for taking so long to update this proposal, but it's now
in a reasonable state:
https://wiki.whatwg.org/wiki/OffscreenCanvas

(Renamed per feedback from Anne -- thanks.)

Please offer your feedback. Multiple browser implementers agreed on
this form of the API at a recent face-to-face meeting of the WebGL
working group, and the proposal should be able to address multiple use
cases. I'm very interested in prototyping it to see if it can offer
better parallelism.

-Ken



> -Ken
>
>
>> Rob
>> --
>> oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo
>> owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
>> osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo
>> owohooo
>> osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o
>> oioso
>> oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo
>> owohooo
>> osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro
>> ooofo
>> otohoeo ofoioroeo ooofo ohoeololo.

Re: [whatwg] Canvas image to blob/dataurl within Worker

[Kenneth Russell]

On Wed, Mar 25, 2015 at 1:22 PM, Robert O'Callahan  wrote:
> On Wed, Mar 25, 2015 at 11:41 PM, Anne van Kesteren 
> wrote:
>>
>> On Fri, Mar 20, 2015 at 11:15 PM, Robert O'Callahan
>>  wrote:
>> > My understanding is that the current consensus proposal for canvas in
>> > Workers is not what's in the spec, but this:
>> > https://wiki.whatwg.org/wiki/WorkerCanvas
>> > See "Canvas in Workers" threads from October 2013 for the discussion.
>> > svn
>> > is failing me but the CanvasProxy proposal in the spec definitely
>> > predates
>> > those threads.
>> >
>> > Ian, unless I'm wrong, it would be helpful to remove the CanvasProxy
>> > stuff
>> > from the spec to avoid confusion.
>> >
>> > That proposal contains WorkerCanvas.toBlob, which needs to be updated to
>> > use promises.
>>
>> There's also https://wiki.whatwg.org/wiki/WorkerCanvas2 it seems. It
>> would be interesting to know what the latest on this is.
>
>
> Right now that appears to be just a copy of WorkerCanvas with some
> boilerplate text added. I assume someone's working on it and will mention it
> on this list if/when they're ready to discuss it :-).

Yes, apologies for letting it sit there in an incomplete state.
Recently feedback from more browser implementers was gathered about
the WorkerCanvas proposal. I will assemble it into WorkerCanvas2 and
follow up on this thread in a day or two.

-Ken


> Rob
> --
> oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo
> owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
> osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo
> owohooo
> osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o
> oioso
> oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo
> owohooo
> osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro
> ooofo
> otohoeo ofoioroeo ooofo ohoeololo.

Re: [whatwg] High-density canvases

[Kenneth Russell]

On Mon, Jun 23, 2014 at 6:06 PM, Robert O'Callahan  wrote:
> On Tue, Jun 24, 2014 at 12:27 PM, Robert O'Callahan 
> wrote:
>>
>> I'll do that now.
>
>
> Done.
> http://wiki.whatwg.org/wiki/CanvasRenderedPixelSize

Fantastic. Thanks Rob. That looks great. Filed crbug.com/388532 about
implementing them in Chrome.

-Ken

Re: [whatwg] High-density canvases

[Kenneth Russell]

On Mon, Jun 23, 2014 at 4:25 PM, Robert O'Callahan  wrote:
> On Tue, Jun 24, 2014 at 8:54 AM, Kenneth Russell  wrote:
>>
>> It's hard to predict. A more general API would be better than a
>> canvas-specific one, assuming it solves the problem. getBoxQuads can
>> return different types of CSS boxes (content, padding, border, margin)
>> -- will it be obvious to web developers which to use, and is it more
>> likely all implementations will handle them all correctly rather than
>> a couple of new properties on the canvas?
>>
>> Could you suggest a name for the new API? getBoxQuadsInDevicePixels?
>> getDevicePixelBoxQuads?
>
>
> AFAIK all we need from this API is device pixel sizes. Extending getBoxQuads
> with an option to return geometry in device pixels doesn't really make sense
> to me since it doesn't make sense to ask "what is element A's quad relative
> to element B in device pixels". So a general, dedicated API would probably
> look something like:
>   interface DOMSize {
> readonly attribute unrestricted double width;
> readonly attribute unrestricted double height;
>   };
>   dictionary BoxSizeOptions {
> CSSBoxType box = "border";
>   };
>   sequence getBoxDevicePixelSizes(optional BoxSizeOptions options);
>
> So "renderedPixelWidth/Height" on HTMLCanvasElement is definitely simpler. I
> have no strong opinion about which way to go, but I lean towards the
> attributes unless someone has a good use-case for the general API. We could
> add the general API later if we need to, the duplication would not be a big
> deal.

If there's no concern about potential duplicated functionality then
let's add the attributes to the canvas element. They'll be easier for
developers to understand and easier to verify as obviously correct.

How should we proceed? Would you add an entry to
http://wiki.whatwg.org/wiki/Category:Proposals ?

Re: [whatwg] High-density canvases

[Kenneth Russell]

On Thu, Jun 19, 2014 at 4:20 PM, Robert O'Callahan  wrote:
> On Fri, Jun 20, 2014 at 6:06 AM, Kenneth Russell  wrote:
>>
>> It is a little unfortunate that a canvas-specific solution is needed.
>> Is it known whether document.getBoxQuads solves this problem in
>> Firefox?
>
>
> It doesn't.
>
> Gecko (and I assume other engines) typically snaps CSS box edges to device
> pixel edges at render time, so that CSS backgrounds and borders look crisp.
> So if a canvas has a CSS background exactly filling its content box, our
> snapping of the content box is what determines the ideal device pixel size
> of the canvas buffer.
>
> Authors can estimate the canvas device pixel size using getBoxQuads to get
> the canvas content box in fractional CSS pixels and multiplying by
> devicePixelRatio (assuming no CSS transforms or other effects present). But
> if that estimate results in a non-integral device pixel size, there's no way
> for authors to know how we will snap that to an integral size.
>
> We could move these attributes up the element hierarchy, or better still to
> a new generic API that works on all elements (e.g. some variant of
> getBoxQuads). (CSS fragmentation might split an element into multiple boxes
> with different sizes.) I don't know of any good use-cases for that, but
> maybe there are some?

It's hard to predict. A more general API would be better than a
canvas-specific one, assuming it solves the problem. getBoxQuads can
return different types of CSS boxes (content, padding, border, margin)
-- will it be obvious to web developers which to use, and is it more
likely all implementations will handle them all correctly rather than
a couple of new properties on the canvas?

Could you suggest a name for the new API? getBoxQuadsInDevicePixels?
getDevicePixelBoxQuads?

-Ken

Re: [whatwg] High-density canvases

[Kenneth Russell]

On Thu, Jun 19, 2014 at 7:54 AM, Stephen White  wrote:
> On Thu, Jun 12, 2014 at 11:42 PM, Robert O'Callahan 
> wrote:
>
>> I think I'd rather not take control of canvas resizing away from
>> applications, even opt-in. That leads to complexity such as extra API for
>> slaving other canvases. I also think we'd be better off sticking to the
>> invariant that the units of canvas coordinate space are single pixels in
>> the canvas bitmap; I think that simplifies things for implementers and
>> authors.
>>
>
> I agree.
>
>
>> Here's an alternative proposal which I think is a bit simpler and more
>> flexible:
>> Expose two new DOM attributes on HTMLCanvasElement:
>> readonly attribute long preferredWidth;
>> readonly attribute long preferredHeight;
>> These attributes are the UA's suggested canvas size for optimizing output
>> quality. It's basically what Ian's proposal would have set as the automatic
>> size. We would also add a "preferredsizechange" event when those attributes
>> change.
>>
>> Applications that want DPI-aware canvases would read those attributes and
>> use them to set the size, just once if they only want to paint once, during
>> each requestAnimationFrame callback for games and other animations, and in
>> the preferredsizechange event handler if they are willing to paint multiple
>> times but aren't actually animating. The application would be responsible
>> for scaling its output to the new canvas coordinate space size. Giving the
>> application control of the size change simplifies things for the browser
>> and gives applications maximum flexibility, e.g. resizing ancillary
>> resources as needed, or imposing constraints on the chosen size.
>>
>
> I like this new proposal. It works for both the canvas and WebGL use cases,
> and it puts the app in control of behaviour in a way that makes sense for
> immediate-mode APIs.
>
> I assume that the size change event would fire:
>
>- on browser page zoom
>- on pinch-zoom
>- when a CSS animation (e.g., scale) changes the canvas size in CSS
>pixels
>
> For browsers that implement the latter two off the main thread, perhaps
> they should only fire at end-of-gesture or end-of-animation, to avoid the
> rendered size being out-of-sync with scaled size by the time the canvas
> gets composited.
>
> I agree with Mark that the names need work. How about something that
> incorporates "device pixel" in some way, to reflect that this is roughly
> dpr * css scale * size?
>
> devicePixelWidth
> widthInDevicePixels
> pixelExactWidth
> exactPixelWidth

"widthInDevicePixels" and "exactPixelWidth" both sound clear.
"renderedPixelWidth" per
https://bugzilla.mozilla.org/show_bug.cgi?id=1024493 also sounds like
a good option. It would be great to spec these and their associated
change event. I'd be interested in adding support to Chrome.

It is a little unfortunate that a canvas-specific solution is needed.
Is it known whether document.getBoxQuads solves this problem in
Firefox?

-Ken


> pixelWidth
> pixelRatioExactWidth
> unscaledWidth
> unscaledPixelWidth
> nativeWidth
> nativePixelWidth
>
> Stephen
>
>
>>
>> Rob
>> --
>> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
>> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
>> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
>> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
>> waanndt  wyeonut  thoo mken.o w
>>

Re: [whatwg] WebGL and ImageBitmaps

[Kenneth Russell]

On Fri, May 16, 2014 at 9:27 AM, Ian Hickson  wrote:
> On Fri, 16 May 2014, Justin Novosad wrote:
>>
>> This is a longstanding issue with 2D canvas that several developers have
>> complained about over the years.  The color space of the canvas backing
>> store is currently unspecified.
>
> It's defined that the colour space must be the same as used for CSS and
> , but yeah the final colour space is left up to the UA:
>
>http://whatwg.org/html#color-spaces-and-color-correction
>
>
>> Blink/WebKit uses output-referred color space, which is bad for some
>> inter-op cases, but good for performance. Calling drawImage will produce
>> inter-operable behavior as far as visual output is concerned, but
>> getImageData will yield values that have the display color profile baked
>> in.
>
> I'm not quite sure what you mean here. If you mean that you can set
> 'fillStyle' to a colour, fillRect the canvas, and the get the data for
> that pixel and find that it's a different colour, then that's
> non-conforming. If you mean that you can take a bitmap image without
> colour-profile information and draw it on a canvas and then getImageData()
> will return different results, then again, that's non-conforming.
>
> If you mean that drawing an image with a color profile will result in
> getImageData() returning different colour pixels on different systems,
> then that's allowed, because the colour space of the canvas (and the rest
> of the Web platform, which must be the same colour space) is not defined.
>
>
>> Some web developers have worked around this by reverse-engineering the
>> client-specific canvas to sRGB colorspace transform by running a test
>> pattern through drawImage+getImageData.  It is horrible that we are
>> making devs resort to this.
>
> I'm not really sure what this work around achieves. Can you elaborate?
>
> If you just want to do everything in sRGB, then putting all your images in
> sRGB but without giving color space information (or setting the option to
> 'strip', if we add these createImageBitmap() options) would result in what
> you want, no?
>
> You'd have to manually (or on the server) convert images that were in
> other colour spaces, though.
>
>
>> Adding a colorspace option to createImageBitmap is not enough IMHO. I
>> think we need a more global color-management approach for canvas.
>
> If we need colour management, we need it for the Web as a whole, not just
> for canvas. So far, though, implementations have not been implementing the
> features that have been proposed, so...:
>
>http://www.w3.org/TR/css3-color/#dropped

Recently there has been activity in some implementations to finally
solve the color management problem. I can't speak for Safari, but have
heard that as of Mac OS X 10.9, that implementation is rendering web
pages into the sRGB color space per the CSS spec, and converting to
the display's color profile on the way to the screen. I was told that
there was a concern there would be a compatibility impact with web
apps implicitly expecting to work in the display's color space, but
surprisingly, everything seemed to work during this transition.

In Chrome there is also work ongoing to handle multiple monitors with
different display color profiles. Noel Gordon (CC'd) is driving this
and the list of active issues can be seen at
https://code.google.com/p/chromium/issues/list?q=owner%3Anoel%40chromium.org
. It's likely the implementation will be done in stages: first
rendering images correctly when dragging windows from monitor to
monitor, and then rendering all web page content correctly.

In sum, while it has taken a while for implementations to finally
start tackling the color management problem, it's happening now.

-Ken

Re: [whatwg] WebGL and ImageBitmaps

[Kenneth Russell]

On Thu, May 15, 2014 at 10:59 AM, Ian Hickson  wrote:
> On Wed, 14 May 2014, Kenneth Russell wrote:
>> >
>> > On the assumption that they are, I filed some bugs to cover this:
>> >
>> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642
>> >createImageBitmap() options: image data orientation
>> >
>> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643
>> >createImageBitmap() options: color space handling
>> >
>> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644
>> >createImageBitmap() options: alpha channel handling
>> >
>> > As usual with such bugs, if there is interest amongst browser vendors
>> > in implementing any of these, please comment on the bug.
>>
>> For the record, I've commented on the above bugs. I'd personally like to
>> see them all specified and implemented so that image loading for the
>> purpose of uploading to WebGL textures can be more efficient.
>
> I see that your comment was that the WebGL WG would like them specified.
>
> Does your saying that mean that Chrome wants to implement these also? Just
> having a working group want them specified doesn't help get us to multiple
> implementations...

To clarify the Chrome team's desire to implement these I added another
comment to each bug, and pointed to the relevant portion of the WebGL
spec in response to the latest comment on
https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 .

-Ken

Re: [whatwg] WebGL and ImageBitmaps

[Kenneth Russell]

On Fri, May 9, 2014 at 2:58 PM, Ian Hickson  wrote:
> On Fri, 9 May 2014, Ian Hickson wrote:
>> On Thu, 18 Jul 2013, Justin Novosad wrote:
>> >
>> > To help us iterate further, I've attempted to capture the essence of
>> > this thread on the whatwg wiki, using the problem solving template. I
>> > tried to capture the main ideas that we seem to agree on so far and I
>> > started to think about how to handle special cases.
>> >
>> > http://wiki.whatwg.org/wiki/ImageBitmap_Options
>>
>> Are the "strongly desired options" in the above wiki page still the
>> options we should be adding?
>
> On the assumption that they are, I filed some bugs to cover this:
>
>https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642
>createImageBitmap() options: image data orientation
>
>https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643
>createImageBitmap() options: color space handling
>
>https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644
>createImageBitmap() options: alpha channel handling
>
> As usual with such bugs, if there is interest amongst browser vendors in
> implementing any of these, please comment on the bug.

For the record, I've commented on the above bugs. I'd personally like
to see them all specified and implemented so that image loading for
the purpose of uploading to WebGL textures can be more efficient.

-Ken

Re: [whatwg] Proposal: navigator.cores

[Kenneth Russell]

Applications need this API in order to determine how many Web Workers
to instantiate in order to parallelize their work.

The problem of how to appropriately size web worker pools has arisen
many times in sophisticated web applications and frameworks. Google
Maps ran into it (I don't work on that product so can't provide
details, but hopefully some member of the team can, if necessary). The
three.js framework used to spawn model loaders in web workers, but had
to stop doing so when it was found that it was spawning too many and
it was difficult to figure out at run time how many would (probably)
be beneficial.

A prioritization API for Web Workers won't solve this problem, because
all of the workers an application requests must actually be spawned
for correctness purposes. There's no provision in the web worker
specification for allocation of a web worker to fail gracefully, or
for a worker to be suspended indefinitely. Even if a worker had its
priority designated as "low", it would still need to be started. On
32-bit systems, at least, spawning too many workers will cause the
user agent to run out of address space fairly quickly.

It would be great to design a new parallelism architecture for the
web, but from a practical standpoint, no progress has been made in
this area for a number of years, and web developers are hampered today
by the absence of this information. I think it should be exposed to
the platform.

-Ken


On Mon, May 5, 2014 at 4:09 PM, João Carlos Martins Eiras
 wrote:
> Hi.
>
> I'm just taking a peek at this topic. My first impression is: why
> would anyone want such a low level hardware information (CPU cores and
> whatnot) on something as high level and abstract as a browser? This
> strikes me initially a useful as having the CPU architecture exposed
> as well.
>
> However, I do see the use cases of somehow being more forgiving with
> CPU usage. But this would be much better achieved with an API like
> defining the priority of workers (high, normal, low) and other async
> tasks, and let the user agent do all the resource allocation and
> management, because that's what user agents are built to do. In any
> case the user agent would make those choices better than the web
> developer.
>
> The only use case I can think of where a web page might want explicit
> hardware information is for driver download pages, benchmarks or
> collecting statistics. But these cases would be better solved with a
> specialized API requiring user consent.

Re: [whatwg] Canvas in workers

[Kenneth Russell]

Great.


On Tue, Oct 22, 2013 at 2:54 PM, Robert O'Callahan  wrote:
> I got an account and I'm uploading the proposal now.
>
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 22, 2013 at 1:44 PM, Robert O'Callahan  wrote:
> On Tue, Oct 22, 2013 at 7:31 PM, Kenneth Russell  wrote:
>>
>> Robert, please don't remove those APIs from your proposal. They're
>> needed in order to address known use cases, and splitting them off
>> will make it difficult to understand how they interact with
>> WorkerCanvas later.
>
>
> Yes, I think it's a good idea to specify a complete set of APIs that fit
> together logically and if there are some we don't need, we can remove them
> or just delay implementing them until they're needed.
>
>> I would like to suggest changing the 'srcObject' property on
>> HTMLImageElement into some sort of method taking ImageBitmap as
>> argument. If an ImageBitmap had been previously set against the
>> HTMLImageElement, the method would automatically call 'close()'
>> against it. Fundamentally there should be some easy way to repeatedly
>> update the HTMLImageElement without having to query its previous
>> ImageBitmap and call close() against it before setting srcObject.
>
>
> Hmm. I'm not sure how this should work.
>
> Maybe instead we should use canvas elements and define
> ImageBitmap.transferToCanvas(HTMLCanvasElement). That would mitigate Glenn's
> argument about having to change element types too.

Using a Canvas as the target for displaying an ImageBitmap is fraught
with problems. Because Canvases are rendering targets themselves,
transferring in an ImageBitmap has to interoperate somehow with the
canvas's rendering context as well as other APIs like toDataURL. The
DrawingBuffer proposal in http://wiki.whatwg.org/wiki/CanvasInWorkers
was admittedly complex, but did properly handle fully detaching a
canvas's framebuffer and attaching it to another one. ImageBitmap,
being semantically read-only, isn't well suited for this task.

The idea to use an HTMLImageElement to display the ImageBitmaps is
elegant and I would like to see it explored, including prototyping it
behind an experimental flag and seeing how well it works.

>> Would you consider copying
>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal to
>> http://wiki.whatwg.org/wiki/Category:Proposals so that it's easier to
>> collaborate on it?
>
>
> No problem at all. Can you do it? I need to get a WHATWG account :-).

Yes, I'll take care of the initial draft. The whatwg was pretty
responsive when I got my account though.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Mon, Oct 21, 2013 at 8:03 AM, Glenn Maynard  wrote:
> On Sun, Oct 20, 2013 at 11:53 PM, Robert O'Callahan 
> wrote:
>>
>> Glenn, taking a step back for a bit, is there anything in
>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal that you would
>> actually object to? IOW, is there anything there that you would think is
>> completely superfluous to the platform if all your proposals were to be
>> adopted as well?
>
>
> I have no objection to the overall change from CanvasProxy to WorkerCanvas,
> eg. the stuff in Kyle's original mail to the thread.  (Being able to settle
> on that is one of the reasons I've tried to detach discussion for the other
> use cases.)
>
> I'd only recommend leaving out transferToImageBitmap, srcObject and
> ImageBitmap.close() parts.  I do think that would be redundant with with
> "present" proposal.  They can always be added later, and leaving them out
> keeps the WorkerCanvas proposal itself focused.

Robert, please don't remove those APIs from your proposal. They're
needed in order to address known use cases, and splitting them off
will make it difficult to understand how they interact with
WorkerCanvas later.

I would like to suggest changing the 'srcObject' property on
HTMLImageElement into some sort of method taking ImageBitmap as
argument. If an ImageBitmap had been previously set against the
HTMLImageElement, the method would automatically call 'close()'
against it. Fundamentally there should be some easy way to repeatedly
update the HTMLImageElement without having to query its previous
ImageBitmap and call close() against it before setting srcObject.

Would you consider copying
https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal to
http://wiki.whatwg.org/wiki/Category:Proposals so that it's easier to
collaborate on it?

Re: [whatwg] Synchronizing Canvas updates in a worker to DOM changes in the UI thread

[Kenneth Russell]

On Tue, Oct 22, 2013 at 7:37 AM, Glenn Maynard  wrote:
> I just noticed that Canvas already has a Canvas.setContext() method

That's there in support of CanvasProxy, which is a flawed API and
which this entire discussion is aiming to rectify.

> , which
> seems to do exactly what I'm proposing, even down to clearing the backbuffer
> on attach.  The only difference is that it lives on Canvas instead of the
> context--the only reason I put it there in my proposal was because this only
> seemed useful for WebGL.  Given that, I think this proposal can be
> simplified down to just: "put setContext on WorkerCanvas too".

Also, adding a present() method to Canvas.

> On Mon, Oct 21, 2013 at 9:03 PM, Kenneth Russell  wrote:
>>
>> There are some unexpected consequences of the attachToCanvas API
>> style. For example, what if two contexts use attachToCanvas to target
>> the same canvas?
>
>
> I left out these details in my initial post in order to see what people
> thought at a high level before delving into details.

At a high level I prefer the form of the WorkerCanvas API, including
transferToImageBitmap and the ability to transfer an ImageBitmap into
an HTMLImageElement for viewing, and removing the CanvasProxy concept
and associated APIs. I'd like to focus my own efforts in writing a
full draft for WorkerCanvas under
http://wiki.whatwg.org/wiki/Category:Proposals .

-Ken

Re: [whatwg] Synchronizing Canvas updates in a worker to DOM changes in the UI thread

[Kenneth Russell]

On Mon, Oct 21, 2013 at 4:34 PM, Glenn Maynard  wrote:
> On Mon, Oct 21, 2013 at 6:08 PM, Kenneth Russell  wrote:
>>
>> > Using ImageBitmap for this has a lot of issues.  It requires
>> > synchronizing
>> > with scripts in the UI thread.
>>
>> This isn't difficult, and amounts to a few additional lines of code in
>> the main thread's onmessage handler.
>
>
> Synchronization with the UI thread isn't bad because it's difficult.
> Avoiding synchronization with the main thread has been raised as a desirable
> goal:
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2013Oct/0152.html
> "including that it isn't possible to render from a worker without
> synchronizing with the main thread."
>
> (My previous comments on this are here:
> http://www.mail-archive.com/whatwg@lists.whatwg.org/msg35959.html)
>
>>
>> The ImageBitmap style proposal has another significant advantage in
>> that it allows a single canvas context to present results in multiple
>> output regions on the page.
>
>
> You can do that.  You just create a WorkerCanvas for each canvas you want to
> present to, hand them to the worker, then attachToCanvas in the worker to
> switch from canvas to canvas.  (That's orthogonal to explicitpresent.)

OK, I misunderstood that part of your attachToCanvas proposal.

There are some unexpected consequences of the attachToCanvas API
style. For example, what if two contexts use attachToCanvas to target
the same canvas? What if one of those contexts is 2D and the other is
WebGL? Currently it's illegal to try to fetch two different context
types for a single Canvas. The current CanvasProxy spec contains
several complex rules for these cases, and they're not easy to
understand.

Will it be guaranteed that if you have a WebGL context, attachToCanvas
to canvas1, do some rendering, and then attachToCanvas to canvas2,
that the only remaining buffer in canvas1 is its color buffer? No
depth buffers, multisample buffers, etc. will have to remain for some
reason?

How would WebGL's preserveDrawingBuffer attribute, which is a property
of the context, interact with directing its output to multiple
canvases?

Fundamentally I think the behavior is easier to spec, and the
implementation is easier to make correct, if the ultimate destination
is an image rather than a canvas, and the color buffer is transferred
out of the WorkerCanvas in an explicit step.

-Ken


>>
>> This sort of resource exhaustion is certainly possible, but I view
>> this downside as smaller than the upside of addressing both of the
>> above use cases.
>
>
> I can only find one thing above that you might be referring to as a use case
> (the one I replied to immediately above).  What was the other?
>
> --
> Glenn Maynard
>

Re: [whatwg] Synchronizing Canvas updates in a worker to DOM changes in the UI thread

[Kenneth Russell]

On Mon, Oct 21, 2013 at 3:39 PM, Glenn Maynard  wrote:
> On Sun, Oct 20, 2013 at 11:16 PM, Robert O'Callahan 
> wrote:
>>
>> With all these proposals I think it's OK to allow the main thread to do
>> (e.g.) a toDataURL and read what the current contents of the canvas is,
>
>
> We can defer this discussion, since it's not something new to this proposal
> (or any other proposal we're discussing).
>
>
> On Sun, Oct 20, 2013 at 11:33 PM, Robert O'Callahan 
> wrote:
>>
>> To me, passing the image data explicitly in an ImageBuffer along with the
>> "present" message seems like a better fit to the workers message-passing
>> model than this proposal, where the data is stored as hidden state in the
>> canvas element with (effectively) a setter in the worker and a getter in the
>> main thread, and that setting and getting has to be coordinated with
>> postMessage for synchronization. The relationship between a "commit" and its
>> "present" has to be deduced by reasoning about the timing of messages,
>> rather than by just reasoning about JS data flow through postMessage.
>
>
> Using ImageBitmap for this has a lot of issues.  It requires synchronizing
> with scripts in the UI thread.

This isn't difficult, and amounts to a few additional lines of code in
the main thread's onmessage handler.

The ImageBitmap style proposal has another significant advantage in
that it allows a single canvas context to present results in multiple
output regions on the page.


>  It requires manualling resize your canvas
> repeatedly to fit different destinations.  It also may potentially create
> lots of backbuffers. Here's an example of code using ImageBitmap
> incorrectly, leading to excess memory allocation:
>
> function render()
> {
> var canvas = myWorkerCanvas;
> renderTo(canvas);
> var buffer = canvas.transferToImageBitmap();
> postMessage(buffer);
> }
> setTimeout(render, 1);
>
> We start with one backbuffer available, render to it (renderTo), peel it off
> the canvas to be displayed somewhere, and toss it off to the main thread.
> (For the sake of the example, the main thread is busy and doesn't process it
> immediately.)  The worker enters render() again, and when it gets to
> renderTo, a new backbuffer has to be allocated, since the one buffer we have
> is still used by the ImageBuffer and can't be changed.  This happens
> repeatedly, creating new backbuffers each time, since none of them can be
> reused.
>
> This is an extreme example, but if this ever happens even once, it means
> potentially allocating an extra backbuffer.

This sort of resource exhaustion is certainly possible, but I view
this downside as smaller than the upside of addressing both of the
above use cases.

-Ken


>>
>> This proposal also requires that whenever a worker is going to return
>> image data to the main thread, the main thread must start things off by
>> creating a canvas element. It's also not possible for a worker to spawn off
>> sub-workers to do drawing (at least, not without some really ugly
>> coordination with the main thread.)
>
>
> Sure it is.  If you want an offscreen buffer, you just "new WorkerCanvas()".
> This is unrelated to offscreen drawing.
>
> --
> Glenn Maynard
>

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 15, 2013 at 5:30 PM, Kenneth Russell  wrote:
> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan  
> wrote:
>> On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell  wrote:
>>>
>>> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan 
>>> wrote:
>>> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>>> >>
>>> >> Would you mind looking at the proposal
>>> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>>> >
>>> >
>>> > Sure. Kyle and I looked at it while we were working on our proposal. The
>>> > main issues I have with it are that rearchitecting  to introduce
>>> > the
>>> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
>>> > doesn't handle direct presentation of frames from the worker, bypassing
>>> > the
>>> > main thread.
>>>
>>> Note that the CanvasInWorkers draft solves some other longstanding
>>> issues not addressed by the WorkerCanvas proposal. It provides the
>>> ability to render to multiple canvases from a single context, whether
>>> workers are involved or not.
>>
>>
>> That may be a useful feature, but I'd like to see it justified in its own
>> right.
>
> There has been a lot of developer feedback on the WebGL mailing lists
> over the past couple of years about exactly this feature. Web sites
> like Turbosquid want to present lots of little thumbnails of models --
> see for example http://www.turbosquid.com/Search/3D-Models/Vehicle/Car
> -- and have them be interactive. It's too resource-intensive to create
> a separate WebGL context for each. The most direct solution is to
> allow one context to render to multiple canvases.
>
>
>>> It achieves ideal memory utilization by
>>> being very explicit in the API, without the need for extensive and
>>> subtle optimizations behind the scenes.
>>
>>
>> We can be more explicit with ImageBitmaps. We could provide
>> WorkerCanvas.transferToImageBitmap which transfers the current canvas
>> contents to an ImageBitmap and clears the canvas. (Canvas implementations
>> are already optimized to support a zero-cost "cleared" state, because
>> existing benchmarks require it.) Sharing ImageBitmap contents across threads
>> during structured clone is not subtle. We can add an
>> HTMLImageElement.srcObject attribute which could take a Blob or an
>> ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
>> that be explicit enough for you?
>
> Yes, that generally sounds good.
>
>
>> Personally I think high-performance manipulation of ImageBitmaps would be
>> more generally useful than detachable DrawingBuffers, and would be easier
>> for authors to understand.
>>
>> If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
>> DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
>> further drawing. Do you?
>
> Not immediately. The ability to transfer out the canvas's contents,
> and render them in an HTMLImageElement without incurring an extra
> blit, should address the Maps team's requirements.
>
> Actually, adding transferToImageBitmap to HTMLCanvasElement as well
> would address the use case of rendering to multiple targets using one
> context. Instead of using multiple canvases as the targets, one would
> simply use multiple images. That sounds appealing.
>
> If WorkerCanvas is changed so that its width and height are mutable
> within the worker as you mentioned above, it sounds like it's
> addressing the known use cases.

Capturing Glenn Maynard's feedback from the other thread started by
Rik Cabanier, Glenn made a good point that there needs to be a way to
explicitly deallocate the ImageBitmap. Otherwise, the JavaScript
objects will have to be garbage collected before the GPU resource
(texture) it references can be freed, and that will not work -- GPU
resources will quickly pile up.

Would it be acceptable to say that setting the 'src' property of an
HTMLImageElement to an ImageBitmap neuters the incoming ImageBitmap?
If not, would it be feasible to add another method to HTMLImageElement
like setToImageBitmap which does this?

-Ken


>>> It's worth considering whether a change to the CanvasInWorkers
>>> proposal could support presenting frames directly from the worker.
>>
>>
>> Sure, by adding a commit() method to DrawingBuffer. Right?
>
> I'm not exactly sure how it would be done. In the proposal as written,
> the DrawingBuffer's not shared between threads, only transferred.
>
> -Ken
>
>
>> Rob
>> --
>> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
>> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
>> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
>> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
>> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Counterproposal for canvas in workers

[Kenneth Russell]

On Thu, Oct 17, 2013 at 2:08 PM, Rik Cabanier  wrote:
>
>
>
> On Thu, Oct 17, 2013 at 2:03 PM, Kenneth Russell  wrote:
>>
>> On Wed, Oct 16, 2013 at 10:26 PM, Robert O'Callahan
>>  wrote:
>> > On Thu, Oct 17, 2013 at 3:34 PM, Rik Cabanier 
>> > wrote:
>> >
>> >> The tasks themselves can also launch synchronized/unsynchronized
>> >> subtasks
>> >> with promises. A task is considered "done" if it exits and all its
>> >> promises
>> >> are fulfilled.
>> >>
>> >
>> > It seems that tasks are like workers, but different, and you'd have to
>> > do a
>> > lot of extra work to precisely define the execution environment of the
>> > task
>> > script.
>> >
>> > It also seems that you have to precisely define how different tasks
>> > interact. For example is the current path left in the canvas by task 1
>> > usable by the code in task 2? You also have to define how this works in
>> > WebGL.
>> >
>> > I don't think this supports a worker/task generating a steady stream of
>> > frames, e.g. for a 3D game. Does it?
>> >
>> > I'm not all that enthusiastic :-)
>>
>> Sorry, neither am I. OpenGL (and WebGL) applications do a lot of
>> one-time setup, and then repeatedly redraw using the previously
>> uploaded objects. This "stateless" drawing model isn't compatible with
>> that structure.
>
>
> Every task per ID and per canvas has access to its own state/VM.
> So, the first time a task is executed (or we could provide an 'init' phase),
> it could do setup which will be maintained between tasks.

OK, I see. Sorry for misinterpreting.

It seems to me that this proposal would restrict even further what the
worker executing the task can do, and be harder to program to than the
existing worker model. I'm interested in pursuing the other discussion
around a WorkerCanvas rather than this one.

-Ken

Re: [whatwg] Counterproposal for canvas in workers

[Kenneth Russell]

On Wed, Oct 16, 2013 at 10:26 PM, Robert O'Callahan
 wrote:
> On Thu, Oct 17, 2013 at 3:34 PM, Rik Cabanier  wrote:
>
>> The tasks themselves can also launch synchronized/unsynchronized subtasks
>> with promises. A task is considered "done" if it exits and all its promises
>> are fulfilled.
>>
>
> It seems that tasks are like workers, but different, and you'd have to do a
> lot of extra work to precisely define the execution environment of the task
> script.
>
> It also seems that you have to precisely define how different tasks
> interact. For example is the current path left in the canvas by task 1
> usable by the code in task 2? You also have to define how this works in
> WebGL.
>
> I don't think this supports a worker/task generating a steady stream of
> frames, e.g. for a 3D game. Does it?
>
> I'm not all that enthusiastic :-)

Sorry, neither am I. OpenGL (and WebGL) applications do a lot of
one-time setup, and then repeatedly redraw using the previously
uploaded objects. This "stateless" drawing model isn't compatible with
that structure.

-Ken

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Wed, Oct 16, 2013 at 1:13 PM, Justin Novosad  wrote:
>
>
>
> On Wed, Oct 16, 2013 at 3:53 PM, Robert O'Callahan 
> wrote:
>>
>> On Thu, Oct 17, 2013 at 6:35 AM, Kenneth Russell  wrote:
>>>
>>> Yes, right. That factory method is already spec'ed on the
>>> WorkerGlobalScope [1]. It actually returns a Promise, so presumably
>>> transferToImageBitmap would have to as well.
>>
>>
>> The whole point of transferToImageBitmap is that it's really fast, so I
>> don't see why it has to be async.
>
>
> True. I also wonder why all of the currently spec'ed ImageBitmap creation
> methods are async. I think asynchrony makes sense when creating ImageBitmaps
> from blobs, image elements and video elements, which may not be in an
> immediately accessible state, but creating an ImageBitmap from a Canvas or
> canvas context (or a WorkerCanvas) could be immediate.

It would be fine in my opinion for transferToImageBitmap to simply
return an ImageBitmap. The suggestion to return a Promise was merely
for symmetry with createImageBitmap.

While the Promise returned from createImageBitmap(HTMLCanvasElement)
can be fulfilled immediately, is it worth introducing a special
overload with a different return type?

-Ken

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Wed, Oct 16, 2013 at 5:39 AM, Justin Novosad  wrote:
> On Tue, Oct 15, 2013 at 8:30 PM, Kenneth Russell  wrote:
>>
>> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan 
>> wrote:
>> > If you squint, WorkerCanvas.transferToImageBitmap is similar to
>> > detaching a
>> > DrawingBuffer. But I don't see a need to reattach a buffer to a canvas
>> > for
>> > further drawing. Do you?
>>
>> Not immediately. The ability to transfer out the canvas's contents,
>> and render them in an HTMLImageElement without incurring an extra
>> blit, should address the Maps team's requirements.
>
>
> WorkerCanvas.copyToImageBitmap could be just as effective with a proper lazy
> copy-on-write mechanism. It would offer the same performance in cases where
> you would just need to transfer (as opposed to copy) the buffer, with the
> added flexibility that it reattaches a new buffer to the canvas, only if
> needed (at next draw).  Also the lazy copy can be skipped if the next draw
> operation to the canvas context is a clear, in which case the UA only needs
> to attach an uninitialized buffer.

I'm assuming that transferToImageBitmap will attach a new buffer to
the canvas as well. The semantic would be that the color buffer of the
canvas gets transferred to the ImageBitmap, and the canvas gets a new,
blank color buffer. (Any auxiliary buffers, like a depth buffer, would
also be implicitly cleared.)

It's easier to understand how to make the transfer operation efficient
than how to optimize the copy-on-write, which requires deeper analysis
of the calls made against the canvas and context in order to get peak
performance. If both are present then both the createImageBitmap and
transfer implementations can be really simple. Do you have a
particular objection to including the transfer API?

> Also, because ImageBitmaps are immutable objects, the API should probably be
> more like var imageBitmap = createImageBitmap(myWorkerCanvas);

Yes, right. That factory method is already spec'ed on the
WorkerGlobalScope [1]. It actually returns a Promise, so presumably
transferToImageBitmap would have to as well.

[1] 
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#imagebitmapfactories

-Ken

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan  wrote:
> On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell  wrote:
>>
>> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan 
>> wrote:
>> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>> >>
>> >> Would you mind looking at the proposal
>> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>> >
>> >
>> > Sure. Kyle and I looked at it while we were working on our proposal. The
>> > main issues I have with it are that rearchitecting  to introduce
>> > the
>> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
>> > doesn't handle direct presentation of frames from the worker, bypassing
>> > the
>> > main thread.
>>
>> Note that the CanvasInWorkers draft solves some other longstanding
>> issues not addressed by the WorkerCanvas proposal. It provides the
>> ability to render to multiple canvases from a single context, whether
>> workers are involved or not.
>
>
> That may be a useful feature, but I'd like to see it justified in its own
> right.

There has been a lot of developer feedback on the WebGL mailing lists
over the past couple of years about exactly this feature. Web sites
like Turbosquid want to present lots of little thumbnails of models --
see for example http://www.turbosquid.com/Search/3D-Models/Vehicle/Car
-- and have them be interactive. It's too resource-intensive to create
a separate WebGL context for each. The most direct solution is to
allow one context to render to multiple canvases.


>> It achieves ideal memory utilization by
>> being very explicit in the API, without the need for extensive and
>> subtle optimizations behind the scenes.
>
>
> We can be more explicit with ImageBitmaps. We could provide
> WorkerCanvas.transferToImageBitmap which transfers the current canvas
> contents to an ImageBitmap and clears the canvas. (Canvas implementations
> are already optimized to support a zero-cost "cleared" state, because
> existing benchmarks require it.) Sharing ImageBitmap contents across threads
> during structured clone is not subtle. We can add an
> HTMLImageElement.srcObject attribute which could take a Blob or an
> ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
> that be explicit enough for you?

Yes, that generally sounds good.


> Personally I think high-performance manipulation of ImageBitmaps would be
> more generally useful than detachable DrawingBuffers, and would be easier
> for authors to understand.
>
> If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
> DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
> further drawing. Do you?

Not immediately. The ability to transfer out the canvas's contents,
and render them in an HTMLImageElement without incurring an extra
blit, should address the Maps team's requirements.

Actually, adding transferToImageBitmap to HTMLCanvasElement as well
would address the use case of rendering to multiple targets using one
context. Instead of using multiple canvases as the targets, one would
simply use multiple images. That sounds appealing.

If WorkerCanvas is changed so that its width and height are mutable
within the worker as you mentioned above, it sounds like it's
addressing the known use cases.


>> It's worth considering whether a change to the CanvasInWorkers
>> proposal could support presenting frames directly from the worker.
>
>
> Sure, by adding a commit() method to DrawingBuffer. Right?

I'm not exactly sure how it would be done. In the proposal as written,
the DrawingBuffer's not shared between threads, only transferred.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Kenneth Russell]

On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan  wrote:
> On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell  wrote:
>>
>> Would you mind looking at the proposal
>> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>
>
> Sure. Kyle and I looked at it while we were working on our proposal. The
> main issues I have with it are that rearchitecting  to introduce the
> DrawingBuffer layer of abstraction seems unnecessarily complex, and it
> doesn't handle direct presentation of frames from the worker, bypassing the
> main thread.

Note that the CanvasInWorkers draft solves some other longstanding
issues not addressed by the WorkerCanvas proposal. It provides the
ability to render to multiple canvases from a single context, whether
workers are involved or not. It achieves ideal memory utilization by
being very explicit in the API, without the need for extensive and
subtle optimizations behind the scenes.

It's worth considering whether a change to the CanvasInWorkers
proposal could support presenting frames directly from the worker.


>> There's been some recent discussion in the WebGL WG on this topic and
>> concerns were raised from other parties at Mozilla about the
>> DrawingBuffer proposal above, including that it isn't possible to
>> render from a worker without synchronizing with the main thread.
>
>
> This is a huge limitation. For games and other animated content, achieving a
> stable frame rate is super important and a key motivation for adding canvas
> support to workers.
>
> My vision for handling the Maps use-cases based on our proposal is this: the
> worker renders to one or more WorkerCanvases, then does
> createImageBitmap(canvasContext), then ships the ImageBitmap(s) to the main
> thread using postMessage, and then renders those ImageBitmaps either by
> drawing them to a canvas or in some more direct way.
>
> This can all be implemented in a zero-copy way with the APIs currently in
> the spec, though it's not trivial. You'd need to do a few optimizations:
> -- createImageBitmap(canvasContext) would take a lazy snapshot of the canvas
> contents; i.e., further modifications to the canvas would trigger a copy (on
> the worker), but if the canvas is untouched no copy is required.
> -- structured clone of the ImageBitmap would not copy. This may actually
> require a small spec change.
> -- drawing an ImageBitmap to a 2D , if it covers the whole canvas,
> would simply replace the canvas buffer with the ImageBitmap contents (and
> perform copy-on-write if the script later makes changes to that canvas).
> These optimizations would all be useful in other contexts too. Whatever
> happens with canvas-in-a-worker, I bet we'll end up doing these
> optimizations for other reasons.

I agree that it's probably possible to make this work. It is still
worth considering whether these optimizations are going to be fragile,
and whether developers will fall off a performance cliff if something
subtle changes in their code or in the browser's behavior in the
future.


> It might make sense to create an API that renders an ImageBitmap more
> directly than drawing it to a canvas. For example we could create an API
> that allows an  element to render an ImageBitmap. It would be a bit
> simpler for authors and perhaps a bit simpler to implement than the final
> optimization in my list above.

That's an interesting possibility. It would probably be quite a bit
simpler to both specify and implement rather than using a canvas on
the receiving end.

-Ken


> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w

Re: [whatwg] Canvas in workers

[Kenneth Russell]

Would you mind looking at the proposal
http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it? This
was arrived at after extensive discussions with the Google Maps team,
and addresses their key use cases. Compared to the one below, it
solves the following problems:

  1) Rendering from a worker and displaying on the main thread with no
extra blits of the rendering results
  2) Allows one context to render to multiple canvases
  3) Supports resizing of the drawing buffer

The main difference in my mind is that in the DrawingBuffer proposal,
the back buffer of the canvas can be "detached", transferred via
postMessage to the other side, and attached to a Canvas, replacing its
previous back buffer. The semantics are simple, clear, avoid extra
blits of the rendered content, and support rendering into multiple
canvases from one context.

There's been some recent discussion in the WebGL WG on this topic and
concerns were raised from other parties at Mozilla about the
DrawingBuffer proposal above, including that it isn't possible to
render from a worker without synchronizing with the main thread.
Still, it seems to me it's worth considering all aspects of the
proposal, because it was motivated by a major existing web app which
is both using Canvas 2D and WebGL extensively, and desires to use
workers more heavily in its rendering pipeline.

-Ken



On Sat, Oct 12, 2013 at 9:12 PM, Kyle Huey  wrote:
> I talked at length with Robert O'Callahan about what the DOM API for
> supporting  in web workers should look like and we came up with the
> following modifications to the spec.
>
>1. Rename CanvasProxy to WorkerCanvas and only allow it to be
>transferred to workers.  I don't think we're interested in supporting
>cross-origin  via CanvasProxy (I would be curious to hear more
>about what the use cases are).
>2. Add a worker-only WorkerCanvas constructor that takes the desired
>width/height of the drawing surface.
>3. Remove the rendering context constructors and the setContext method
>on WorkerCanvas (née CanvasProxy).
>4. Copy all of the sensible non-node related things from
>HTMLCanvasElement to WorkerCanvas.  This would include
>- width and height as readonly attributes
>   - getContext (to replace what we removed in step 3).  roc prefers to
>   have getContext2D and getContextWebGL, and dispense with the string
>   argument version entirely, but I don't have strong feelings.
>   - toBlob.  We do not intend to implement toDataURL here.
>5. Add a "commit" method to WorkerCanvas.  For a WorkerCanvas obtained
>from a main thread  element, this would cause the buffer displayed
>on screen to swap.  For a WorkerCanvas created *de novo* on a worker
>thread, it would do nothing.  This commit method would also commit a minor
>violation of run-to-completion semantics, described below.
>6. We would rely on extracting ImageBitmaps from the WorkerCanvas and
>shipping them to the main thread via postMessage to allow synchronizing
>canvas updates with DOM updates.  We explored a couple other options but we
>didn't come up with anything else that allows synchronizing updates to
>multiple canvases from a worker.  This isn't really sketched out here.
>
> So the IDL would look something like:
>
>> [Constructor(unsigned long width, unsigned long height)]
>>
>> interface WorkerCanvas {
>>
>>   readonly attribute unsigned long width;
>>
>>   readonly attribute unsigned long height;
>>
>>
>>   CanvasRenderingContext2D? getContext2D(any... args);
>>
>>   WebGLRenderingContext? getContextWebGL(any... args);
>>
>>
>>   void toBlob(FileCallback? _callback, optional DOMString type, any...
>> arguments);
>>
>>
>>   bool commit();
>>
>> };
>>
>> WorkerCanvas implements Transferable;
>>
> Everything would be behave pretty much as one would expect, except perhaps
> for the commit method.  The width and height of the canvas can be modified
> on the main thread while the worker is drawing.  This would fire an event
> off to the worker to update the WorkerCanvas's dimensions that would be
> scheduled as if the main thread had postMessage()d something to the
> worker.  But it's possible that the worker would attempt to draw to the
>  before that update runs.  It's also possible that the worker would
> simply draw in a loop without yielding.  To solve this, if commit is called
> and the current dimensions on the main thread don't match the dimensions of
> the WorkerCanvas it would fail (return false) and update the dimensions of
> the WorkerCanvas before returning.  This is technically a violation of
> run-to-completion semantics, but is needed to support workers that do not
> yield.
>
> Thoughts?
>
> - Kyle

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

On Tue, Aug 13, 2013 at 11:57 AM, Ian Hickson  wrote:
> On Mon, 12 Aug 2013, Kenneth Russell wrote:
>>
>> The use case is the passing of ImageData objects back and forth to
>> workers to fill and refill with data.
>>
>> An ImageData is essentially a wrapper for the underlying
>> Uint8ClampedArray, providing an associated width and height. However,
>> the only way to draw raw pixels into a Canvas is using an ImageData.
>>
>> Currently a Unit8ClampedArray can be constructed, but the only way to
>> create an ImageData is to ask a Canvas for one, and there's no way to
>> associate it with a pre-allocated Uint8ClampedArray.
>
> Why can't you just send an ImageData over?
>
>
>> This means that if you want to pass an ImageData to a worker for
>> filling, transferring the underlying Uint8ClampedArray, you need to be
>> very careful about bookkeeping, and to not lose the reference to the
>> ImageData object.
>
> Sure. Just send the ImageData over. That seems relatively
> straight-forward. What am I missing?
>
>
>> IMO there ought to be a factory method for ImageData taking a
>> Uint8ClampedArray, width, height, and possibly resolution (or a
>> dictionary?), which validates the width, height and resolution against
>> the size of the Uint8ClampedArray, and makes a new ImageData object.
>> This would ease management of ImageData instances.
>
> We could have a constructor for ImageData objects, sure. That would be
> relatively easy to add, if it's really needed. I don't understand why it's
> hard to keep track of ImageData objects, though. Can you elaborate?

I have in mind new APIs for typed arrays which allow sharding of typed
arrays to workers and re-assembly of the component pieces when the
work is complete. This would involve multiple manipulations of the
ArrayBuffer and its views. It would be most convenient if the result
could be wrapped in an ImageData if it's destined to be drawn to a
Canvas. Otherwise it's likely that a data copy will need to be
incurred.

-Ken

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

On Fri, Aug 9, 2013 at 2:34 PM, Ian Hickson  wrote:
> On Fri, 9 Aug 2013, Rik Cabanier wrote:
>> On Fri, Aug 9, 2013 at 1:32 PM, Ian Hickson  wrote:
>> > On Mon, 11 Mar 2013, Kenneth Russell wrote:
>> > >
>> > > It would be useful to be able to create an ImageData [1] object with
>> > > preexisting data. The main use case is to display arbitrary data in
>> > > the 2D canvas context with no data copies.
>> >
>> > Doesn't ImageBitmap support this already? I'm not sure I understand
>> > the use case here. Where are you getting the image data from, that
>> > it's already in raw RGBA form rather than compressed e.g. as a PNG?
>> > (Presumably this isn't coming over the network, since I would imagine
>> > the time to compress and decompress an image is far smaller than the
>> > time to send uncompressed data. But maybe I'm wrong about that.)
>>
>> From re-reading the thread, it seems that this data comes from the
>> server (or a web worker?) as uncompressed data. The http protocol likely
>> did compression on the packets so the size difference is probably not
>> that great.
>>
>> I think the use-case is to avoid having to copy over the data pixel by
>> pixel from the arraybuffer.
>
> Could you elaborate on the use case?
>
> I'm happy to believe that there are times that a server or worker is
> generating lots of pixel data, but having never run into such a case
> myself, I would very much like to understand it better. It may be that
> there are better solutions to the real underlying problem.

The use case is the passing of ImageData objects back and forth to
workers to fill and refill with data.

An ImageData is essentially a wrapper for the underlying
Uint8ClampedArray, providing an associated width and height. However,
the only way to draw raw pixels into a Canvas is using an ImageData.

Currently a Unit8ClampedArray can be constructed, but the only way to
create an ImageData is to ask a Canvas for one, and there's no way to
associate it with a pre-allocated Uint8ClampedArray. This means that
if you want to pass an ImageData to a worker for filling, transferring
the underlying Uint8ClampedArray, you need to be very careful about
bookkeeping, and to not lose the reference to the ImageData object.

IMO there ought to be a factory method for ImageData taking a
Uint8ClampedArray, width, height, and possibly resolution (or a
dictionary?), which validates the width, height and resolution against
the size of the Uint8ClampedArray, and makes a new ImageData object.
This would ease management of ImageData instances.

-Ken

Re: [whatwg] BinaryEncoding for Typed Arrays using window.btoa and window.atob

[Kenneth Russell]

On Mon, Aug 5, 2013 at 2:04 PM, Simon Pieters  wrote:
> On Mon, 05 Aug 2013 22:39:22 +0200, Chang Shu  wrote:
>
>> I see your point now, Simon. Technically both approaches should work.
>> As you said, yours has the limitation that the implementation does not
>> know which view to return unless you provide an enum type of parameter
>> instead of boolean to atob. And mine has the performance issue. How
>> about we don't return the 'binary' string in case the 2nd parameter is
>> provided in my case?
>
>
> That works for me.

Chang, in your proposal for modifying atob, how does the developer
know how many characters were written into the outgoing
ArrayBufferView?

What happens if the ArrayBufferView argument isn't large enough to
hold the decoded string?

During the decoding process, is the type of the ArrayBufferView
significant? In your example, what would happen if an Int16Array were
passed instead of an Int32Array?

The Encoding spec at http://encoding.spec.whatwg.org/ seems to have
handled issues like these. Perhaps a better route would be to fold
this functionality into that spec.

-Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Tue, Jul 16, 2013 at 7:16 AM, Justin Novosad  wrote:
>
>
>
> On Tue, Jul 16, 2013 at 12:25 AM, Mark Callow 
> wrote:
>>
>> On 2013/07/15 10:46, Justin Novosad wrote:
>>
>> But to circle back to your point, I agree that an exception is a good idea
>> to avoid having to hold a triplicate copy in RAM, or having to redecode
>> all
>> the time. Better to force the dev to make additional copies explicitly if
>> needed than to make a potentially uselessly costly implementation.mf
>>
>> Maybe I am misunderstanding but the only reason I can see for 3 copies (2
>> if you ignore the undecoded copy) is if you propose to ignore the specified
>> parameters when drawing the image to a 2D canvas.
>
>
> Yes, that is what I was referring to because that was suggested earlier on
> this thread. But I think it is becoming clearer that that is not the right
> thing to do.
>
>>
>> I would expect to always draw the image decoded as indicated by the
>> proposed parameters so no additional copy would be necessary. Sure the image
>> might not be correct (colors off or image upside down) but that would be a
>> programmer error.
>
>
> Exactly.  That is what I am suggesting.  If the programmer wants several
> copies of the same image with different baked-in transformations, then the
> programmer should create several ImageBitmaps explicitly.  No under the hood
> magic. It's clearer that way.

This sounds good. Additionally, the WebGL spec can be updated to state
that the parameters UNPACK_FLIP_Y_WEBGL, etc. don't apply to
ImageBitmap, so the only way to affect the decoding is with the
dictionary of options.


>> I would like to see ImageBitmap fully support WebGL so WebGL apps can use
>> a Browser's built-in image decoders. And, if the rumors are true, come IE11,
>> WebGL will be supported by all major browsers so it should be treated as a
>> first-class citizen.
>
>
> Yes!

Agree with this sentiment also.

-Ken


>>
>> Regards
>>
>> -Mark
>>
>> --
>> 注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、
>> 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.
>>
>> NOTE: This electronic mail message may contain confidential and privileged
>> information from HI Corporation. If you are not the intended recipient, any
>> disclosure, photocopying, distribution or use of the contents of the
>> received information is prohibited. If you have received this e-mail in
>> error, please notify the sender immediately and permanently delete this
>> message and all related copies.
>
>

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Thu, Jul 11, 2013 at 8:29 AM, Justin Novosad  wrote:
>
>
> On Wed, Jul 10, 2013 at 9:37 PM, Rik Cabanier  wrote:
>>
>> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>
>> > On Wed, 10 Jul 2013, Kenneth Russell wrote:
>> > >
>> > > ImageBitmap can cleanly address all of the desired use cases simply by
>> > > adding an optional dictionary of options.
>> >
>> > I don't think that's true. The options only make sense for WebGL --
>> > flipping which pixel is the first pixel, for example, doesn't do
>> > anything
>> > to 2D canvas, which works at a higher level.
>> >
>> > (The other two options don't make much sense to me even for GL. If you
>> > don't want a color space, don't set one. If you don't want an alpha
>> > channel, don't set one. You control the image, after all.)
>> >
>> >
>> > > I suspect that in the future some options will be desired even for the
>> > > 2D canvas use case, and having the dictionary already specified will
>> > > make that easier. There is no need to invent a new primitive and means
>> > > of loading it.
>> >
>> > If options make sense for 2D canvas, then having ImageBitmap options
>> > would
>> > make sense, sure.
>> >
>> >
>> yeah, these options seem a bit puzzling.
>> From the spec:
>>
>> An ImageBitmap object represents a bitmap image that can be painted to a
>> canvas without undue latency.
>>
>> note: The exact judgement of what is undue latency of this is left up to
>> the implementer, but in general if making use of the bitmap requires
>> network I/O, or even local disk I/O, then the latency is probably undue;
>> whereas if it only requires a blocking read from a GPU or system RAM, the
>> latency is probably acceptable.
>>
>> It seems that people see the imageBitmap as something that doesn't just
>> represent in-memory pixels but that those pixels are also preprocessed so
>> they can be drawn quickly. The latter is not in the spec.
>>
>> I think authors will be very confused by these options. What would it mean
>> to pass a non-premultiplied ImageBitmap to a canvas object? Would the
>> browser have to add code to support it or is it illegal?
>> Maybe it's easier to add an optional parameter to createImageBitmap to
>> signal if the ImageBitmap is for WebGL or for Canvas and disallow a Canvas
>> ImageBitmap in WebGL and vice versa.
>
>
> You are implying a pretty heavy imposition as to what constitutes undue
> latency.
> I think the spec should stay away from forcing implementations to pin
> decoded image buffers in RAM (or on the GPU), so that the browser may have
> some latitude in preventing out of memory exceptions. In its current form,
> the spec implies that it would be acceptable for an implementation to
> discard the decoded buffer and only retain the resource in encoded form in
> RAM.  Do we really need to make further optimizations explicit? For example,
> an implementation could prepare the image data for use with WebGL the first
> time it is drawn to WebGL, and keep it cached in that state. If the same
> ImageBitmap is subsequently drawn to a 2D canvas, then it would use the
> non-WebGLified copy, which may be cached, or may require re-decoding the
> image. No big deal.

The step of preparing the image for use, either with WebGL or 2D
canvas, is expensive. Today, this step is necessarily done
synchronously when an HTMLImageElement is uploaded to WebGL. The
current ImageBitmap proposal would still require this synchronous
step, so for WebGL at least, it provides no improvement over the
current HTML5 APIs. A major goal of ImageBitmap was to allow Web
Workers to load them, and even this ability currently provides no
advantage over HTMLImageElement.

> Fundamental question: Do we really need the caller to be able to specify
> what treatments need to be applied to prepare an image for WebGL, or is it
> always possible to figure that out automatically?

It is never possible to figure out automatically how the image needs
to be treated when preparing it for use with WebGL. I'm not sure where
that idea came from. On the contrary, there are eight possibilities
(2^3), and different applications require different combinations.

-Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Wed, Jul 10, 2013 at 5:13 PM, Peter Kasting  wrote:
> On Wed, Jul 10, 2013 at 5:07 PM, Ian Hickson  wrote:
>>
>> (The other two options don't make much sense to me even for GL. If you
>> don't want a color space, don't set one. If you don't want an alpha
>> channel, don't set one. You control the image, after all.)
>
>
> I only have a small amount of graphics experience, but I don't think that
> latter comment is right, at least.
>
> At least for the alpha channel, as Gregg already wrote, a lot of GL
> algorithms use that data for something per-pixel that's not "alpha"
> (generally some other kind of per-pixel map).  It's not appropriate for the
> browser to assume that it's safe to muck with the values there.  Fixing this
> by instead trying to pass these values separate from the rest of the pixel
> data is inefficient as well as just weird from the perspective of anyone
> with significant experience in using these sorts of algorithms.

This is correct. Further, even if an image doesn't contain any color
space information, the browser may still incorrectly decide to adjust
the colorspace of decoded image data based on the client machine's
settings. It's required to be able to tell the browser to not do this.

I would find it really discouraging if the WebGL spec had to subsume
image loading functionality. It's a statement of fact that with a
dictionary of options, ImageBitmap can work as efficiently for the
WebGL canvas context as it's intended to for the 2D canvas context.

-Ken

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

On Wed, Jul 10, 2013 at 4:37 PM, Ian Hickson  wrote:
> On Wed, 10 Jul 2013, Kenneth Russell wrote:
>>
>> Some background: when uploading HTMLImageElements to WebGL it's required
>> to be able to specify certain options, such as whether to premultiply
>> the alpha channel, or perform colorspace conversion. Because it seemed
>> infeasible at the time to modify the HTML spec, these options are set
>> via the WebGL API. If they're set differently from the browser's
>> defaults (which are generally to do premultiplication, and do colorspace
>> conversion), then the WebGL implementation has to re-decode the image
>> when it's uploaded to a WebGL texture. (There's no way to know in
>> advance whether a given image is intended for upload to WebGL as opposed
>> to insertion into the document, and making image decoding lazier than it
>> currently is would introduce bad hiccups while scrolling.)
>
> It seems like the right solution is to create a primitive for WebGL that
> represents images that are going to be used in WebGL calls. Such a
> primitive could use the same sources for images as ImageBitmap, but would
> be specifically for use with WebGL, in the same way that ImageBitmap is
> used just by the 2D Canvas API.

That sounds like the wrong solution to me. The goal of HTML5 should be
good integration of all of the component APIs, not to treat some, like
WebGL, as bolt-on mechanisms.

ImageBitmap can cleanly address all of the desired use cases simply by
adding an optional dictionary of options. I suspect that in the future
some options will be desired even for the 2D canvas use case, and
having the dictionary already specified will make that easier. There
is no need to invent a new primitive and means of loading it.

-Ken


> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] Adding features needed for WebGL to ImageBitmap

[Kenneth Russell]

(Replying on behalf of Gregg, who unfortunately isn't at Google any more)

On Wed, Jul 10, 2013 at 3:17 PM, Ian Hickson  wrote:
> On Wed, 19 Jun 2013, Gregg Tavares wrote:
>>
>> In order for ImageBitmap to be useful for WebGL we need more options
>
> ImageBitmap is trying to just be a generic HTMLImageElement, that is, a
> bitmap image. It's not trying to be anything more than that.
>
> Based on some of these questions, though, maybe you mean ImageData?

Gregg meant ImageBitmap.

Some background: when uploading HTMLImageElements to WebGL it's
required to be able to specify certain options, such as whether to
premultiply the alpha channel, or perform colorspace conversion.
Because it seemed infeasible at the time to modify the HTML spec,
these options are set via the WebGL API. If they're set differently
from the browser's defaults (which are generally to do
premultiplication, and do colorspace conversion), then the WebGL
implementation has to re-decode the image when it's uploaded to a
WebGL texture. (There's no way to know in advance whether a given
image is intended for upload to WebGL as opposed to insertion into the
document, and making image decoding lazier than it currently is would
introduce bad hiccups while scrolling.)

We'd like to avoid the same problems with the new ImageBitmap concept.

The current ImageBitmap draft has the problem that when the callback
is called, image decoding will already have been done, just like
HTMLImageElement -- at least, this is almost surely how it'll be
implemented, in order to obey the rule "An ImageBitmap object
represents a bitmap image that can be painted to a canvas without
undue latency". Just like HTMLImageElement, these options need to be
set before decoding occurs, to avoid redundant work and rendering
pauses which would happen if operations like colorspace conversion
were done lazily. (By the way, colorspace conversion is typically
implemented inside the image decoder itself, and it would be a lot of
work to factor it out into code which can be applied to a
previously-decoded image. In fact from looking again at the code in
Blink which does this I'd say it's completely infeasible.)


>> premultipliedAlpha: true/false (default true)
>> Nearly all GL games use non-premultipiled alpha textures. So all those
>> games people want to port to WebGL will require non-premultipied textures.
>> Often in games the alpha might not even be used for alpha but rather for
>> glow maps or specular maps or the other kinds of data.
>
> How do you do this with  today?

Per above, by specifying the option via the WebGL API, and performing
a synchronous image re-decode. This re-decode is really expensive, and
a major pain point for WebGL developers. It's so bad that developers
are using pure JavaScript decoders for PNG and JPG formats just so
that they can do this on a worker thread.


>> flipY: true/false (default false)
>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>> provides this option but it takes time and memory to flip a large image
>> therefore it would be nice if that flip happened before the callback
>> from ImageBitmap
>
> No pixel is the first pixel in an ImageBitmap. I don't really understand
> what this means.

There's a longstanding difference between the coordinate systems used
by most 2D libraries, and 3D APIs. OpenGL in particular long ago
adopted the convention that the origin of a texture is its lower-left
corner, with the Y axis pointing up.

Every image loading library ever created for OpenGL has had an option
to flip (or not) loaded textures along the Y axis; the option is
required to support pipelines for loading artists' work.

The WebGL spec offers this option via the UNPACK_FLIP_Y_WEBGL state.
http://www.khronos.org/registry/webgl/specs/latest/#TEXIMAGE2D_HTML
defines that the upper left pixel of images is by default the first
pixel transferred to the GPU.

Flipping large images vertically is expensive, taking a significant
percentage of frame time. As with premultiplication of alpha, we want
to avoid doing it unnecessarily, redundantly, or synchronously with
respect to the application. For this reason we want to make it an
option on createImageBitmap so when the callback is called, the
decoded image data is already oriented properly for upload to the GPU.


>> colorspaceConversion: true/false (default true)
>> Some browsers apply color space conversion to match monitor settings.
>> That's fine for images with color but WebGL apps often load heightmaps,
>> normalmaps, lightmaps, global illumination maps and many other kinds of
>> data through images. If the browser applies a colorspace conversion the
>> data is not longer suitable for it's intended purpose therefore many WebGL
>> apps turn off color conversions. As it is now, when an image is uploaded to
>> WebGL, if colorspace conversion is
>> off

Re: [whatwg] Challenging canvas.supportsContext

[Kenneth Russell]

On Wed, Jun 19, 2013 at 3:39 PM, James Robinson  wrote:
>
>
>
> On Wed, Jun 19, 2013 at 3:24 PM, Kenneth Russell  wrote:
>>
>> On Wed, Jun 19, 2013 at 3:06 PM, James Robinson  wrote:
>> > On Wed, Jun 19, 2013 at 3:04 PM, Kenneth Russell  wrote:
>> >>
>> >> supportsContext() can give a much more accurate answer than
>> >> !!window.WebGLRenderingContext. I can only speak for Chromium, but in
>> >> that browser, it can take into account factors such as whether the GPU
>> >> sub-process was able to start, whether WebGL is blacklisted on the
>> >> current card, whether WebGL is disabled on the current domain due to
>> >> previous GPU resets, and whether WebGL initialization succeeded on any
>> >> other page. All of these checks can be done without the heavyweight
>> >> operation of actually creating an OpenGL context.
>> >
>> >
>> > That's true, but the answer still doesn't promise anything about what
>> > getContext() will do.  It may still return null and code will have to
>> > check
>> > for that.  What's the use case for calling supportsContext() without
>> > calling
>> > getContext()?
>>
>> Any application which has a complex set of fallback paths. For example,
>>
>>   - Preference 1: supportsContext('webgl', { softwareRendered: true })
>>   - Preference 2: supportsContext('2d', { gpuAccelerated: true })
>>   - Preference 3: supportsContext('webgl', { softwareRendered: false })
>>   - Fallback: 2D canvas
>
>
> I'm assuming you have (1) and (3) flipped here and both supportsContext()
> and getContext() support additional attributes to dictate whether a
> software-provided context can be supplied.  In that case, in order to write
> correct code I'd still have to attempt to fetch the contexts before using
> them, i.e.:
>
> var ctx = canvas.getContext('webgl', { 'allowSoftware': false});
> if (ctx) {
>   doPreference1(ctx);
>   return;
> }
> ctx = canvas.getContext('2d', {'allowSoftware': false});
> if (ctx) {
>   doPreference2(ctx);
> // etc
>
> how could I simplify this code using supportsContext() ?
>
>
>>
>> I agree that ideally, if supportsContext returns true then -- without
>> any other state changes that might affect supportsContext's result --
>> getContext should return a valid rendering context.
>
>
> It seems overwhelmingly likely that one of the state changes that might
> affect the result will be attempting to instantiate a real context.

In my experience, in Chromium, creation of the underlying OpenGL
context for a WebGLRenderingContext almost never fails in isolation.
Instead, more general failures happen such as the GPU process failing
to boot, or creation of all OpenGL contexts (including the
compositor's) failing. These failures would be detected before the app
calls supportsContext('webgl'). For this reason I believe
supportsContext's answer can be highly accurate in almost every
situation.

>>
>> It's simply
>> impossible to guarantee this correspondence 100% of the time, but if
>> supportsContext's spec were tightened somehow, and conformance tests
>> were added which asserted consistent results between supportsContext
>> and getContext, would that address your concern?
>
>
> I don't see how supportsContext() could be as accurate as getContext()
> without doing all of the work getContext() does.  If it's not 100% accurate,
> when is it useful?

This is a specious question. My point is that the answer can be
accurate enough to be useful, and I'm personally willing to sign up to
make the implementation follow a stricter spec.

-Ken


> - James
>
>
>>
>>
>> -Ken
>
>

Re: [whatwg] Challenging canvas.supportsContext

[Kenneth Russell]

On Wed, Jun 19, 2013 at 3:06 PM, James Robinson  wrote:
> On Wed, Jun 19, 2013 at 3:04 PM, Kenneth Russell  wrote:
>>
>> On Wed, Jun 19, 2013 at 2:20 PM, Brandon Benvie 
>> wrote:
>> > On 6/19/2013 2:05 PM, James Robinson wrote:
>> >>
>> >> What would a page using Modernizr (or other library) to feature detect
>> >> WebGL do if the supportsContext('webgl') call succeeds but the later
>> >> getContext('webgl') call fails?
>> >
>> >
>> > I don't have an example, I was just explaining how Mozernizr is often
>> > used.
>> >
>> >
>> >> I'm also failing to see the utility of the supportsContext() call.
>> >> It's
>> >> impossible for a browser to promise that supportsContext('webgl')
>> >> implies
>> >> that getContext('webgl') will succeed without doing all of the
>> >> expensive
>> >> work, so any correctly authored page will have to handle a
>> >> getContext('webgl') failure anyway.
>> >
>> >
>> > Given this, it would seem supportsContext is completely useless. The
>> > whole
>> > purpose of a feature detection check is to detect if a feature actually
>> > works or not. Accuracy is more important than cost.
>>
>> supportsContext() can give a much more accurate answer than
>> !!window.WebGLRenderingContext. I can only speak for Chromium, but in
>> that browser, it can take into account factors such as whether the GPU
>> sub-process was able to start, whether WebGL is blacklisted on the
>> current card, whether WebGL is disabled on the current domain due to
>> previous GPU resets, and whether WebGL initialization succeeded on any
>> other page. All of these checks can be done without the heavyweight
>> operation of actually creating an OpenGL context.
>
>
> That's true, but the answer still doesn't promise anything about what
> getContext() will do.  It may still return null and code will have to check
> for that.  What's the use case for calling supportsContext() without calling
> getContext()?

Any application which has a complex set of fallback paths. For example,

  - Preference 1: supportsContext('webgl', { softwareRendered: true })
  - Preference 2: supportsContext('2d', { gpuAccelerated: true })
  - Preference 3: supportsContext('webgl', { softwareRendered: false })
  - Fallback: 2D canvas

I agree that ideally, if supportsContext returns true then -- without
any other state changes that might affect supportsContext's result --
getContext should return a valid rendering context. It's simply
impossible to guarantee this correspondence 100% of the time, but if
supportsContext's spec were tightened somehow, and conformance tests
were added which asserted consistent results between supportsContext
and getContext, would that address your concern?

-Ken

Re: [whatwg] Challenging canvas.supportsContext

[Kenneth Russell]

On Wed, Jun 19, 2013 at 2:20 PM, Brandon Benvie  wrote:
> On 6/19/2013 2:05 PM, James Robinson wrote:
>>
>> What would a page using Modernizr (or other library) to feature detect
>> WebGL do if the supportsContext('webgl') call succeeds but the later
>> getContext('webgl') call fails?
>
>
> I don't have an example, I was just explaining how Mozernizr is often used.
>
>
>> I'm also failing to see the utility of the supportsContext() call.  It's
>> impossible for a browser to promise that supportsContext('webgl') implies
>> that getContext('webgl') will succeed without doing all of the expensive
>> work, so any correctly authored page will have to handle a
>> getContext('webgl') failure anyway.
>
>
> Given this, it would seem supportsContext is completely useless. The whole
> purpose of a feature detection check is to detect if a feature actually
> works or not. Accuracy is more important than cost.

supportsContext() can give a much more accurate answer than
!!window.WebGLRenderingContext. I can only speak for Chromium, but in
that browser, it can take into account factors such as whether the GPU
sub-process was able to start, whether WebGL is blacklisted on the
current card, whether WebGL is disabled on the current domain due to
previous GPU resets, and whether WebGL initialization succeeded on any
other page. All of these checks can be done without the heavyweight
operation of actually creating an OpenGL context.

-Ken

Re: [whatwg] Challenging canvas.supportsContext

[Kenneth Russell]

Looking back at the previous discussion:

  http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-September/037229.html
(and succeeding emails)
  http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-October/037693.html

Accurate feature detection in libraries like Modernizr was mentioned
as a key use case:

  http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-September/037249.html

Additionally, though, application developers wanted to be able to ask
questions like "is software rendering supported for WebGL":

  http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-September/037232.html

I think this second use case is valid. One could imagine a graphing or
charting library which would use low-power, software-rendered WebGL if
available, but otherwise would fall back to using
CanvasRenderingContext2D.

I'd like to see supportsContext remain in the spec.

-Ken



On Wed, Jun 19, 2013 at 12:34 PM, Tab Atkins Jr.  wrote:
> On Wed, Jun 19, 2013 at 11:29 AM, Boris Zbarsky  wrote:
>> On 6/19/13 2:17 PM, Benoit Jacob wrote:
>>> The closest thing that I could find being discussed, was use cases by JS
>>> frameworks or libraries that already expose similar feature-detection
>>> APIs.
>>> However, that only shifts the question to: what is the reason for them to
>>> expose such APIs?
>>
>>
>> I _think_ the issue is poorly-designed detection APIs that do the detection
>> even if the consumer of the framework/library doesn't care about that
>> particular feature.
>>
>> That means that right now those frameworks are doing a getContext() call but
>> then no one cares that they did.
>>
>>
>>> There is also the argument that supportsContext can be much cheaper than a
>>> getContext, given that it only has to guarantee that getContext must fail
>>> if supportsContext returned false. But this argument is overlooking that
>>> in
>>> the typical failure case, which is failure due to system/driver
>>> blacklisting, getContext returns just as fast as supportsContext
>>
>> I think the argument here is that the common case for getContext is in fact
>> more and more the success case.  So the framework/library is wasting time
>> successfully creating a context that no one actually cares about.
>>
>> If the above is correct, I agree with Benoit: the right fix is to fix the
>> libraries to do the getContext() lazily when someone actually asks whether
>> WebGL is enabled.
>>
>> If I'm wrong, then I'd like to understand what problem we _are_ trying to
>> solve.  That is, what the cases are that want to check that they can create
>> a context but not actually create one.
>
> This is missing the point.  You don't want to wait until it's actually
> time to create the context.  Unless you torture your code flow, by the
> time you're creating a context you should already know that the
> context is supported.  The knowledge of which context to use is most
> useful well before that, when you're first entering the app.
>
> Plus, it doesn't matter how late you do the detection - if you do a
> straight *detection* at all rather than an initialization (that is, if
> you throw away the context you've just created for testing), you'll
> still incur the start-up costs of spinning up a context.  Doing that
> early or late doesn't matter too much - it's bad either way.
>
> Like @supports, the supportsContext() method can be easy and reliable
> with a very simple definition for "supports" - it returns true if
> calling getContext() with the same arguments would return a context
> rather than erroring, and false otherwise.  No capacity for lying
> there without breaking sites.
>
> ~TJ

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

On Tue, Mar 12, 2013 at 4:54 PM, Rik Cabanier  wrote:
> On Tue, Mar 12, 2013 at 4:16 PM, Glenn Maynard  wrote:
>
>> On Tue, Mar 12, 2013 at 12:14 AM, Boris Zbarsky  wrote:
>>
>> > CSE can get rid of the redundant .data gets.  Similarly, .data gets can
>> be
>> > loop-hoisted in many cases.
>> >
>>
>> Doing COW based on page-faults is nicer anyway, but I don't know about the
>> data structures of JS engines to know whether this is feasible.  (For
>> example, if an object in JS is preceded by a header that gets written by
>> the engine now and then, it'll probably lie on the same page as the data,
>> which would trigger an expensive fault each time.)
>>
>> I suppose padding the backing store so it doesn't share pages with anything
>> else might be reasonable here: up to about 8k of waste on a system with 4kb
>> pages.  The cost of marking the pages read-only would only have to be paid
>> when the copy-on-write action (eg. a call to putImageData) is actually
>> made.  Very small buffers could simply disable copy-on-write and always
>> perform a copy, where the waste for padding is more significant and the
>> benefits of avoiding a copy are smaller.
>>
>> (For what it's worth, marking a 128 MB buffer read-only in Linux with
>> mprotect takes on the order of 3 microseconds on my typical desktop-class
>> system.  I don't know if Windows's VirtualProtect is slower.)
>>
>> On Tue, Mar 12, 2013 at 4:04 PM, Rik Cabanier  wrote:
>>
>> > > I don't think it is necessary to provide a createImageDataHD in this
>> >
>>  > interface. The caller will know the devicePixelRatio and determine
>> > > whether to generate high-DPI data.
>> >
>> > That probably won't work since it results in code that executes
>> differently
>> > on devices that are HD.
>> >
>>
>> The difference between getImageData(HD) and putImageData(HD) is in the
>> canvas operation, not the ImageData: it determines how pixels are scaled
>> when being read out of and written into the canvas backing store.  It
>> doesn't apply to this API; ImageData objects don't know anything beyond
>> their pixel data and dimensions.
>>
>> (Code executing differently on high-DPI devices is a bridge we've already
>> crossed.  getImageData scales pixels down from the device's pixel ratio;
>> getImageDataHD doesn't, copying backing store pixels one to one.)
>>
>> There is no real reason why you could use it in both HD and non-HD APIs.
>> >
>>
>> Rather, there's no reason you couldn't.  You can definitely create an
>> ImageData with getImageData and then pass it to putImageDataHD (which would
>> cause the image to be scaled on devices with a pixel ratio other than 1, of
>> course).
>
>
> It feels like something is missing. How does putImageDataHD know that the
> bitmap should be scaled? Width and height refer to the pixel dimensions and
> not the 'px' unit

putImageData measures the affected region of the scratch bitmap in CSS
pixels and putImageDataHD measures it in device pixels.
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-putimagedata

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

On Tue, Mar 12, 2013 at 3:49 PM, Rik Cabanier  wrote:
>
>
> On Tue, Mar 12, 2013 at 3:03 PM, Kenneth Russell  wrote:
>>
>> On Tue, Mar 12, 2013 at 2:04 PM, Rik Cabanier  wrote:
>> >
>> >
>> > On Tue, Mar 12, 2013 at 11:40 AM, Kenneth Russell 
>> > wrote:
>> >>
>> >> On Tue, Mar 12, 2013 at 11:15 AM, Rik Cabanier 
>> >> wrote:
>> >> > sounds good!
>> >> > I think this is a convenient and useful addition.
>> >>
>> >> Great.
>> >>
>> >> > do you want to keep doubles to define the dimensions instead of
>> >> > integers? If
>> >> > so, the size should probably  4 * ceil(sw) * ceil(sh)
>> >>
>> >> I would prefer to use integers, and only used doubles to be consistent
>> >> with the other APIs like getImageData and createImageData. In this
>> >> case it would make more sense to use integers, since the width and
>> >> height are simply being used to interpret preexisting data in the
>> >> Uint8ClampedArray.
>> >
>> >
>> > The current canvas spec doesn't specifically state what happens with
>> > partial
>> > pixels. What happens today?
>> > (Also is there a definition somewhere that states when a pixel is
>> > considered
>> > filled?)
>>
>> Safari, Firefox and Chrome all round the double arguments to
>> putImageData to integers using the "truncate" rounding mode and then
>> draw the source ImageData pixel-for-pixel. For example, passing 64.5
>> or 64.99 for the dx or dy arguments is equivalent to passing 64.
>> Here's a test case.
>>
>> 
>> 
>> var canvas = document.getElementById("canvas");
>> var ctx = canvas.getContext("2d");
>> var width = canvas.width;
>> var height = canvas.height;
>> ctx.fillRect(0, 0, width, height);
>> var imageData = ctx.createImageData(width / 2, height / 2);
>> for (var ii = 0; ii < imageData.data.length; ii += 4) {
>>   imageData.data[ii + 0] = 0;
>>   imageData.data[ii + 1] = 255;
>>   imageData.data[ii + 2] = 0;
>>   imageData.data[ii + 3] = 255;
>> }
>> // Try passing 64.5, 64.99, or 65 for one or both of these arguments
>> and see the results
>> ctx.putImageData(imageData, 64, 64);
>> 
>>
>> In other words, the source ImageData would not be rendered into the
>> canvas at a half-pixel offset if ctx.putImageData(imageData, 64.5,
>> 64.5) were called.
>
>
> Thanks for investigating this. The fact that 'truncate' is called, should
> probably go in the spec.
> Maybe we should change the IDL to integer.

I think that would be a good idea. I believe it would be backward
compatible and leave the definition of double -> long conversion to
the Web IDL and ECMAScript specs.


>> >> I don't think it is necessary to provide a createImageDataHD in this
>> >> interface. The caller will know the devicePixelRatio and determine
>> >> whether to generate high-DPI data.
>> >
>> > That probably won't work since it results in code that executes
>> > differently
>> > on devices that are HD.
>>
>> I think it works. The application will call the new ImageData
>> constructor and pass it to either putImageData or putImageDataHD.
>> These interpret the incoming ImageData differently depending on the
>> devicePixelRatio.
>>
>> In contrast, CanvasRenderingContext2D's existing createImageDataHD and
>> getImageDataHD methods will create an ImageData that may have a
>> different width and height from those passed in. The reason for this
>> is that these methods are referring to the canvas's backing store. For
>> this new constructor which simply wraps existing pixel data, the
>> application knows exactly how many pixels are contained in the array,
>> so it makes the most sense to take the incoming width and height
>> verbatim. I don't see any advantage to having an alternate "high-DPI"
>> constructor which would multiply the width and height by the
>> devicePixelRatio behind the scenes.
>>
>
> Your proposal is:
>
> createImageData would throw an exception if the length of the
> Uint8ClampedArray was not equal to 4 * floor(sw) * floor(sh), or at
> least, if the length of the array was less than this value.

Yes.

> So, if you create an imageData that is going to be used in putImageDataHD,
> the bounds checking happens when you pass it into putImageDataHD?
> It seems the imageData object should know if

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

On Tue, Mar 12, 2013 at 11:15 AM, Rik Cabanier  wrote:
> sounds good!
> I think this is a convenient and useful addition.

Great.

> do you want to keep doubles to define the dimensions instead of integers? If
> so, the size should probably  4 * ceil(sw) * ceil(sh)

I would prefer to use integers, and only used doubles to be consistent
with the other APIs like getImageData and createImageData. In this
case it would make more sense to use integers, since the width and
height are simply being used to interpret preexisting data in the
Uint8ClampedArray.

-Ken


> On Tue, Mar 12, 2013 at 10:50 AM, Kenneth Russell  wrote:
>>
>> It should simply reference the Uint8ClampedArray, not copy it or do
>> anything else esoteric. The only way to display an ImageData in the 2D
>> canvas context is via the putImageData API. I am not proposing
>> changing those semantics.
>>
>> -Ken
>>
>>
>>
>> On Mon, Mar 11, 2013 at 5:00 PM, Rik Cabanier  wrote:
>> > Do you expect that createImageData creates an internal copy of the
>> > Uint8ClampedArray object or is it live?
>> >
>> >
>> > On Mon, Mar 11, 2013 at 4:28 PM, Kenneth Russell  wrote:
>> >>
>> >> It would be useful to be able to create an ImageData [1] object with
>> >> preexisting data. The main use case is to display arbitrary data in
>> >> the 2D canvas context with no data copies.
>> >>
>> >> Proposed IDL:
>> >>
>> >> [NoInterfaceObject]
>> >> interface ImageDataFactories {
>> >>   ImageData createImageData(Uint8ClampedArray data, double sw, double
>> >> sh);
>> >> };
>> >> Window implements ImageDataFactories;
>> >> WorkerGlobalScope implements ImageDataFactories;
>> >>
>> >> createImageData would throw an exception if the length of the
>> >> Uint8ClampedArray was not equal to 4 * floor(sw) * floor(sh), or at
>> >> least, if the length of the array was less than this value. (Similar
>> >> wording would be used to that of CanvasRenderingContext2D's
>> >> createImageData.)
>> >>
>> >> I don't think it is necessary to provide a createImageDataHD in this
>> >> interface. The caller will know the devicePixelRatio and determine
>> >> whether to generate high-DPI data.
>> >>
>> >> [1]
>> >>
>> >> http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata
>> >>
>> >> Comments?
>> >>
>> >> Thanks,
>> >>
>> >> -Ken
>> >
>> >
>
>

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

I much prefer your suggestion to just add a constructor to ImageData.
I was not sure whether that style was preferred nowadays. ImageData is
already exposed in the global namespace, so making it a callable
constructor function seems like an easy change.

As mentioned in another reply, the intent here is to reference the
Uint8ClampedArray, not make a copy.

-Ken


On Mon, Mar 11, 2013 at 7:03 PM, Boris Zbarsky  wrote:
> On 3/11/13 7:28 PM, Kenneth Russell wrote:
>>
>> Proposed IDL:
>>
>> [NoInterfaceObject]
>> interface ImageDataFactories {
>>ImageData createImageData(Uint8ClampedArray data, double sw, double
>> sh);
>> };
>> Window implements ImageDataFactories;
>> WorkerGlobalScope implements ImageDataFactories;
>
>
> How about just:
>
>   [Constructor(Uint8ClampedArray data, double sw, double sh)]
>   interface ImageData {
> /* Whatever is currently there */
>   };
>
> and then you create one with:
>
>   new ImageData(someData, someWidth, someHeight);
>
> Other than needing to specify whether the array is copied or held on to by
> reference, and specifying that this interface should be exposed in workers,
> this seems fine to me.
>
> -Boris

Re: [whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

It should simply reference the Uint8ClampedArray, not copy it or do
anything else esoteric. The only way to display an ImageData in the 2D
canvas context is via the putImageData API. I am not proposing
changing those semantics.

-Ken



On Mon, Mar 11, 2013 at 5:00 PM, Rik Cabanier  wrote:
> Do you expect that createImageData creates an internal copy of the
> Uint8ClampedArray object or is it live?
>
>
> On Mon, Mar 11, 2013 at 4:28 PM, Kenneth Russell  wrote:
>>
>> It would be useful to be able to create an ImageData [1] object with
>> preexisting data. The main use case is to display arbitrary data in
>> the 2D canvas context with no data copies.
>>
>> Proposed IDL:
>>
>> [NoInterfaceObject]
>> interface ImageDataFactories {
>>   ImageData createImageData(Uint8ClampedArray data, double sw, double sh);
>> };
>> Window implements ImageDataFactories;
>> WorkerGlobalScope implements ImageDataFactories;
>>
>> createImageData would throw an exception if the length of the
>> Uint8ClampedArray was not equal to 4 * floor(sw) * floor(sh), or at
>> least, if the length of the array was less than this value. (Similar
>> wording would be used to that of CanvasRenderingContext2D's
>> createImageData.)
>>
>> I don't think it is necessary to provide a createImageDataHD in this
>> interface. The caller will know the devicePixelRatio and determine
>> whether to generate high-DPI data.
>>
>> [1]
>> http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata
>>
>> Comments?
>>
>> Thanks,
>>
>> -Ken
>
>

[whatwg] Proposal: ImageData constructor or factory method with preexisting data

[Kenneth Russell]

It would be useful to be able to create an ImageData [1] object with
preexisting data. The main use case is to display arbitrary data in
the 2D canvas context with no data copies.

Proposed IDL:

[NoInterfaceObject]
interface ImageDataFactories {
  ImageData createImageData(Uint8ClampedArray data, double sw, double sh);
};
Window implements ImageDataFactories;
WorkerGlobalScope implements ImageDataFactories;

createImageData would throw an exception if the length of the
Uint8ClampedArray was not equal to 4 * floor(sw) * floor(sh), or at
least, if the length of the array was less than this value. (Similar
wording would be used to that of CanvasRenderingContext2D's
createImageData.)

I don't think it is necessary to provide a createImageDataHD in this
interface. The caller will know the devicePixelRatio and determine
whether to generate high-DPI data.

[1] 
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata

Comments?

Thanks,

-Ken

Re: [whatwg] Adding a btoa overload that takes Uint8Array

[Kenneth Russell]

On Mon, Mar 4, 2013 at 10:04 AM, Joshua Bell  wrote:
> On Mon, Mar 4, 2013 at 9:09 AM, Boris Zbarsky  wrote:
>
>> The problem I'm trying to solve is sending Unicode text to consumers who
>> need base64-encoded input.  Right now the only "sane" way to do it (and I
>> quote "sane" for obvious reasons) is something like the example at
>> https://developer.mozilla.org/**en-US/docs/DOM/window.btoa#**
>> Unicode_Strings
>>
>> It seems like it would be better if the output of a TextEncoder could be
>> passed directly to btoa.  But for that we need an overload of btoa that
>> takes a Uint8Array.
>>
>
> FYI, I believe the last iteration on this topic ended with this message:
>
> http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-June/036372.html
>
> i.e. consensus that base64 should stay out of the Encoding API, but that it
> would be nice to have some form of base64 / Typed Array conversion API. But
> there were no concrete proposals beyond my strawman in that post.
>
> So: agreed, have at it!

Yes, adding an overload of btoa taking Uint8Array sounds good.

Re: [whatwg] Fetch: crossorigin="anonymous" and XMLHttpRequest

[Kenneth Russell]

Are you referring to the crossOrigin attribute on HTMLImageElement and
HTMLMediaElement? Those are implemented in WebKit. It should be fine
to change crossOrigin="anonymous" requests to satisfy (a) and (b). Any
server that satisfies these anonymous requests in a way compatible
with UAs' caching will ignore the incoming origin and the referrer.

-Ken


On Tue, Feb 26, 2013 at 2:52 PM, Adam Barth  wrote:
> WebKit hasn't implemented either, so we don't have any implementation
> constraints in this area.
>
> Adam
>
>
> On Tue, Feb 26, 2013 at 3:35 AM, Anne van Kesteren  wrote:
>> There's an unfortunate mismatch currently. new
>> XMLHttpRequest({anon:true}) will generate a request where a) origin is
>> a globally unique identifier b) referrer source is the URL
>> about:blank, and c) credentials are omitted. From those
>> crossorigin="anonymous" only does c. Can we still change
>> crossorigin="anonymous" to match the anonymous flag semantics of
>> XMLHttpRequest or is it too late?
>>
>>
>> --
>> http://annevankesteren.nl/

[whatwg] Reporting errors during Web Worker startup

[Kenneth Russell]

http://www.whatwg.org/specs/web-apps/current-work/multipage/workers.html#creating-workers
doesn't seem to define what happens if there aren't enough resources
to create a separate parallel execution environment.

Would it be legal for a UA to consider this as violating a policy
decision and throw SecurityError? Or  is that step intended to reflect
a static decision, such as whether the UA allows workers to run at
all?

If this behavior isn't specified, could some graceful failure mode be
specified? Currently some UAs terminate the execution of the page
attempting to start the worker, which is obviously undesirable.

Thanks,

-Ken

Re: [whatwg] Endianness of typed arrays

[Kenneth Russell]

On Wed, Mar 28, 2012 at 2:27 PM, Brandon Jones  wrote:
> I was initially on the "just make it little endian and don't make me worry
> about it" side of the fence, but on further careful consideration I've
> changed my mind: I think having typed arrays use the platform endianness is
> the right call.
>
> As Ken pointed out, if you are populating your arrays from javascript or a
> JSON file or something similar this is a non-issue. The problem only occurs
> when you are attempting to load a binary blob directly into a typed array.
> Unless that blob is entirely homogenous (ie: all Float32's or all Int16's,
> etc) it's impossible to trivially swap endianness without being provided a
> detailed breakdown of the data patterns contained within the blob.
>
> Consider this example (using WebGL, but the same could apply elsewhere): I
> download a binary file containing tightly packed interleaved vertices that I
> want to pass directly to a WebGL buffer. The data contains little endian
> vertex positions, texture coordinates, texture ID's and a 32 bit color per
> vertex, so the data looks something like this:
>
> struct {
>     Float32[3] pos,
>     Float32[4] uv,
>     Uint16 textureId,
>     Uint32 color
> };
>
> I will receive this data from XHR as an opaque TypedArray, and if the
> platform is little endian I can pass it directly to the GPU. But on big
> endian systems, a translation needs to be done somewhere:
>
> xhr.responseType = "arraybuffer";
> xhr.onload = function() {
>     var vertBuffer = gl.createBuffer();
>     gl.bindBuffer(gl.ARRAY_BUFFER, vertBuffer);
>
>     // If bigEndian then... magic!
>
>     gl.bufferData(gl.ARRAY_BUFFER, this.response, gl.STATIC_DRAW);
> }
>
> So the question is: What exactly are we expecting that "magic" to be? We
> can't just swizzle every 4 bytes. Either the graphics driver must do the
> endian swap as it processes the buffer, which is possible but entirely out
> of the browsers control, or we would have to provide data packing
> information to the browser so that it could do the appropriate swap for us.
> And if I'm going to have to build up a data definition and pass that through
> to the browser anyway... well I've just destroyed the whole "don't make me
> care about endianness" ideal, haven't I? I might as well just do the swap in
> my own code via a DataView, or better yet cache a big endian version of the
> same file on the server side if I'm worried about performance.

I would suggest that you pass down the schema of the data to the
client application along with the raw binary file, and always iterate
down it with DataView, reading each individual value and storing it
into one of multiple typed array views of a new ArrayBuffer. Then
upload the new ArrayBuffer to WebGL. This way, if you get the code
working on one platform, you are guaranteed that it will work on all
platforms.

As one simple concrete example, please look at
http://code.google.com/p/webglsamples/source/browse/hdr/hdr.js#235 .
This demo downloads high dynamic range textures as binary files
containing floating-point values. The data is copied from the XHR's
ArrayBuffer using a DataView, knowing that the source data is in
little endian format, and stored into a Float32Array for upload to
WebGL. This code works identically on big-endian and little-endian
architectures.

> So yeah, it sucks that we have to plan for devices that are practically
> non-existant and difficult to test for, but I don't really see a nicer
> (practical) solution.
>
> That said, one thing that DataView doesn't handle too nicely right now is
> arrays. You're basically stuck for-looping over your data, even if it's all
> the same type. I would fully support having new DataView methods available
> like:
>
> Int32Array getInt32Array(unsigned long byteOffset, unsigned long elements,
> optional boolean littleEndian);
>
> Which would be a nice, sensible optimization since I'm pretty sure the
> browser backend could do that faster than a JS loop.

Definitely agree that adding array readers and writers to DataView is
worth considering; it's even mentioned in the typed array spec at
http://www.khronos.org/registry/typedarray/specs/latest/#11 . I would
however like to work on optimizing DataView's single-element accessors
first so that we could do a good measurement of the potential speedup.
Right now DataView is completely unoptimized in WebKit's
implementation, but the typed array views have had the benefit of
months of optimization work in both the JavaScriptCore and V8 engines.

-Ken

> --Brandon
>
> On Wed, Mar 28, 2012 at 1:39 PM, Kenneth Russell  wrote:
>>
>> On Wed, Mar 28, 20

Re: [whatwg] Endianness of typed arrays

[Kenneth Russell]

On Wed, Mar 28, 2012 at 12:34 PM, Benoit Jacob  wrote:
> Before I joined this mailign list, Boris Zbarsky wrote:
>> C)  Try to guess based on where the array buffer came from and have
>> different behavior for different array buffers.  With enough luck (or
>> good enough heuristics), would make at least some WebGL work, while also
>> making non-WebGL things loaded over XHR work.
>
> FWIW, here is a way to do this that will always work and won't rely on 
> "luck". The key idea is that by the time one draws stuff, all the information 
> about how vertex attributes use buffer data must be known.
>
> 1. In webgl.bufferData implementation, don't call glBufferData, instead just 
> cache the buffer data.
>
> 2. In webgl.vertexAttribPointer, record the attributes structure (their 
> types, how they use buffer data). Do not convert/upload buffers yet.
>
> 3. In the first WebGL draw call (like webgl.drawArrays) since the last 
> bufferData/vertexAttribPointer call, do the conversion of buffers and the 
> glBufferData calls. Use some heuristics to drop the buffer data cache, as 
> most WebGL apps will not have a use for it anymore.

It would never be possible to drop the CPU side buffer data cache. A
subsequent draw call may set up the vertex attribute pointers
differently for the same buffer object, which would necessitate going
back through the buffer's data and generating new, appropriately
byte-swapped data for the GPU.

>> In practice, if forced to implement a UA on a big-endian system today, I
>> would likely pick option (C)  I wouldn't classify that as a victory
>> for standardization, but I'm also not sure what we can do at this point
>> to fix the brokenness.
>
> I agree that seems to be the only way to support universal webgl content on 
> big-endian UAs. It's not great due to the memory overhead, but at least it 
> shouldn't incur a significant performance overhead, and it typically only 
> incurs a temporary memory overhead as we should be able to drop the buffer 
> data caches quickly in most cases. Also, buffers are typically 10x smaller 
> than textures, so the memory overhead would typically be ~ 10% in corner 
> cases where we couldn't drop the caches.

Our emails certainly crossed, but please refer to my other email.
WebGL applications that assemble vertex data for the GPU using typed
arrays will already work correctly on big-endian architectures. This
was a key consideration when these APIs were being designed. The
problems occur when binary data is loaded via XHR and uploaded to
WebGL directly. DataView is supposed to be used in such cases to load
the binary data, because the endianness of the file format must
necessarily be known.

The possibility of forcing little-endian semantics was considered when
typed arrays were originally being designed. I don't have absolute
performance numbers to quote you, but based on previous experience
with Java's NIO Buffer classes, I am positive that the performance
impact for WebGL applications on big-endian architectures would be
very large. It would prevent applications which manipulate vertices in
JavaScript from running acceptably on big-endian machines.

-Ken

> In conclusion: WebGL is not the worst here, there is a pretty reasonable 
> avenue for big-endian UAs to implement it in a way that allows running the 
> same unmodified content as little-endian UAs.
>
> Benoit

Re: [whatwg] Endianness of typed arrays

[Kenneth Russell]

On Wed, Mar 28, 2012 at 3:46 AM, Boris Zbarsky  wrote:
> On 3/28/12 3:14 AM, Mark Callow wrote:
>
>> vertexAttribPointer lets you specifiy to WebGL the layout and type of
>> the data in the buffer object.
>
> Sure.  That works for the GPU, but it doesn't allow for the sort of
> on-the-fly endianness conversion that would be needed to make webgl still
> work on big-endian platforms if the JS-visible typed arrays were always
> little-endian.
>
>
>> The API follows OpenGL {,ES} for familiarity and reflects its
>> heritage of a C API avoiding use of structures.
>
> Yep.  I know the history.  I think this was a mistake, if we care about the
> web ever being usable on big-endian hardware.  Whether we do is a separate
> question.
>
>> But it works.
>
> Sort of, but maybe not; see below.
>
>
>> OpenGL {,ES} developers typically load data from a serialized form and
>> perform endianness conversion during deserialization. The serialized
>> form is what would be loaded into an ArrayBuffer via XHR. It is then
>> deserialized into 1 or more additional ArrayBuffers.
>
>
> The point is that developers are:
>
> 1)  Loading data in serialized forms that has nothing to do with WebGL
>    via XHR and then reading it using typed array views on the
>    resulting array buffer.
> 2)  Not doing endianness conversions, either for the use case in point
>    1 or indeed for WebGL.
>
> Again, I think we all agree how this would work if everyone using the typed
> array APIs were perfect in every way and had infinite resources. But they're
> not and they don't... The question is where we go from here.
>
> In practice, it sounds like a UA on a big-endian system has a few options:
>
> A)  Native-endianness typed arrays.  Breaks anyone loading data via XHR
> arraybuffer responses (whether for WebGL or not) and not doing manual
> endianness conversions.
>
> B)  Little-endian typed arrays.  Breaks WebGL, unless developers switch to a
> more "struct-based" API.  Makes the non-WebGL cases of XHR arraybuffer
> responses work.
>
> C)  Try to guess based on where the array buffer came from and have
> different behavior for different array buffers.  With enough luck (or good
> enough heuristics), would make at least some WebGL work, while also making
> non-WebGL things loaded over XHR work.
>
> In practice, if forced to implement a UA on a big-endian system today, I
> would likely pick option (C)  I wouldn't classify that as a victory for
> standardization, but I'm also not sure what we can do at this point to fix
> the brokenness.

The top priority should be to implement DataView universally. DataView
is specifically designed for correct, portable manipulation of binary
data coming from or going to files or the network. Fortunately,
DataView is supported in nearly every actively developed UA; once
https://bugzilla.mozilla.org/show_bug.cgi?id=575688 is fixed, it
should be present in every major UA -- even the forthcoming IE 10! See
http://blogs.msdn.com/b/ie/archive/2011/12/01/working-with-binary-data-using-typed-arrays.aspx
.

Once DataView is available everywhere then the top priority should be
to write educational materials regarding binary I/O. It should be
possible to educate the web development community about correct
practices with only a few high profile articles.

Changing the endianness of Uint16Array and the other multi-byte typed
arrays is not a feasible solution. Existing WebGL programs already
work correctly on big-endian architectures specifically because the
typed array views use the host's endianness. If the typed array views
were changed to be explicitly little-endian, it would be a requirement
to introduce new big-endian views, and all applications using typed
arrays would have to be rewritten, not just those which use WebGL.

Finally, to reiterate one point: the typed array design was informed
by prior experience with the design and performance characteristics of
a similar API, specifically Java's New I/O (NIO) Buffer classes. NIO
merged the two distinct use cases of file and network I/O, and
interaction with graphics and audio devices, into one API. The result
was increased polymorphism at call sites, which defeated the Java VM's
optimizing compiler and led to 10x slowdowns in many common
situations. It was so difficult to fix these performance pitfalls that
they remained for many years, and I don't know how robust the
solutions are in current Java VMs. To avoid these issues the typed
array spec consciously treats these use cases separately. It is
possible to make incorrect assumptions leading to non-portable code,
but at some level this is possible with nearly any API that extends
beyond a small, closed world. I believe the focus should be on
educating developers about correct use of the APIs, developing
supporting libraries to ease development, and advancing the ECMAScript
language with constructs like struct types
(http://wiki.ecmascript.org/doku.php?id=harmony:binary_data).

-Ken

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

[Kenneth Russell]

On Tue, Mar 27, 2012 at 6:44 PM, Glenn Maynard  wrote:
> On Tue, Mar 27, 2012 at 7:12 PM, Kenneth Russell  wrote:
>>
>>   - I think it should reference DataView directly rather than
>> ArrayBufferView. The typed array spec was specifically designed with
>> two use cases in mind: in-memory assembly of data to be sent to the
>> graphics card or audio device, where the byte order must be that of
>> the host architecture;
>
>
> This is wrong, broken, won't be implemented this way by any production
> browser, isn't how it's used in practice, and needs to be fixed in the
> spec.  It violates the most basic web API requirement: interoperability.
> Please see earlier in the thread; the views affected by endianness need to
> be specced as little endian.  That's what everyone is going to implement,
> and what everyone's pages are going to depend on, so it's what the spec
> needs to say.  Separate types should be added for big-endian (eg.
> Int16BEArray).

Thanks for your input.

The design of the typed array classes was informed by requirements
about how the OpenGL, and therefore WebGL, API work; and from prior
experience with the design and implementation of Java's New I/O Buffer
classes, which suffered from horrible performance pitfalls because of
a design similar to that which you suggest.

Production browsers already implement typed arrays with their current
semantics. It is not possible to change them and have WebGL continue
to function. I will go so far as to say that the semantics will not be
changed.

In the typed array specification, unlike Java's New I/O specification,
the API was split between two use cases: in-memory data construction
(for consumption by APIs like WebGL and Web Audio), and file and
network I/O. The API was carefully designed to avoid roadblocks that
would prevent maximum performance from being achieved for these use
cases. Experience has shown that the moment an artificial performance
barrier is imposed, it becomes impossible to build certain kinds of
programs. I consider it unacceptable to prevent developers from
achieving their goals.


> I also disagree that it should use DataView.  Views are used to access
> arrays (including strings) within larger data structures.  DataView is used
> to access packed data structures, where constructing a view for each
> variable in the struct is unwieldy.  It might be useful to have a helper in
> DataView, but the core API should work on views.

This is one point of view. The true design goal of DataView is to
supply the primitives for fast file and network input/output, where
the endianness is explicitly specified in the file format. Converting
strings to and from binary encodings is obviously an operation
associated with transfer of data to or from files or the network.
According to this taxonomy, the string encoding and decoding
operations should only be associated with DataView, and not the other
typed array types, which are designed for in-memory data assembly for
consumption by other hardware on the system.


>>  - It would be preferable if the encoding API had a way to avoid
>> memory allocation, for example to encode into a passed-in DataView.
>
>
> This was an earlier design, and discussion led to it being removed as a
> premature optimization, to simplify the API.  I'd recommend reading the rest
> of the thread.

I do apologize for not being fully caught up on the thread, but hope
that the input above was still useful.

-Ken

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

[Kenneth Russell]

On Mon, Mar 26, 2012 at 10:28 PM, Jonas Sicking  wrote:
> On Mon, Mar 26, 2012 at 6:11 PM, Kenneth Russell  wrote:
>> On Mon, Mar 26, 2012 at 5:33 PM, Jonas Sicking  wrote:
>>> On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell  wrote:
>>>>> * We lost the ability to decode from a arraybuffer and see how many
>>>>> bytes were consumed before a null-terminator was hit. One not terribly
>>>>> elegant solution would be to add a TextDecoder.decodeWithLength method
>>>>> which return a DOMString+length tuple.
>>>>
>>>> Agreed, but of course see above - there was consensus earlier in the thread
>>>> that searching for null terminators should be done outside the API,
>>>> therefore the caller will have the length handy already. Yes, this would be
>>>> a big flaw since decoding at tightly packed data structure (e.g. array of
>>>> null terminated strings w/o length) would be impossible with just the
>>>> nullTerminator flag.
>>>
>>> Requiring callers to find the null character first, and then use that
>>> will require one additional pass over the encoded binary data though.
>>> Also, if we put the API for finding the null character on the Decoder
>>> object it doesn't seem like we're creating an API which is easier to
>>> use, just one that has moved some of the logic from the API to every
>>> caller.
>>>
>>> Though I guess the best solution would be to add methods to DataView
>>> which allows consuming an ArrayBuffer up to a null terminated point
>>> and returns the decoded string. Potentially such a method could take a
>>> Decoder object as argument.
>>
>> The rationale for specifying the string encoding and decoding
>> functionality outside the typed array specification is to keep the
>> typed array spec small and easily implementable. The indexed property
>> getters and setters on the typed array views, and methods on DataView,
>> are designed to be implementable with a small amount of assembly code
>> in JavaScript engines. I'd strongly prefer to continue to design the
>> encoding/decoding functionality separately from the typed array views.
>
> Is there a reason you couldn't keep the current set of functions on
> DataView implemented using a small amount of assembly code, and let
> the new functions fall back to slower C++ functions?

That's possible.

Another motivation for keeping encoding/decoding functionality
separate is that it is likely that it will require a lot of spec text,
which would dramatically increase the size of the typed array spec.
Perhaps once all of the details have been hammered out on this thread
it will be more obvious whether these methods would be much clearer if
added directly to DataView.

A couple of comments on the current StringEncoding proposal:

  - I think it should reference DataView directly rather than
ArrayBufferView. The typed array spec was specifically designed with
two use cases in mind: in-memory assembly of data to be sent to the
graphics card or audio device, where the byte order must be that of
the host architecture; and assembly of data for network transmission,
where the byte order needs to be explicit. DataView covers the latter
case.

  - It would be preferable if the encoding API had a way to avoid
memory allocation, for example to encode into a passed-in DataView.

-Ken

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

[Kenneth Russell]

On Mon, Mar 26, 2012 at 5:33 PM, Jonas Sicking  wrote:
> On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell  wrote:
>>> * We lost the ability to decode from a arraybuffer and see how many
>>> bytes were consumed before a null-terminator was hit. One not terribly
>>> elegant solution would be to add a TextDecoder.decodeWithLength method
>>> which return a DOMString+length tuple.
>>
>> Agreed, but of course see above - there was consensus earlier in the thread
>> that searching for null terminators should be done outside the API,
>> therefore the caller will have the length handy already. Yes, this would be
>> a big flaw since decoding at tightly packed data structure (e.g. array of
>> null terminated strings w/o length) would be impossible with just the
>> nullTerminator flag.
>
> Requiring callers to find the null character first, and then use that
> will require one additional pass over the encoded binary data though.
> Also, if we put the API for finding the null character on the Decoder
> object it doesn't seem like we're creating an API which is easier to
> use, just one that has moved some of the logic from the API to every
> caller.
>
> Though I guess the best solution would be to add methods to DataView
> which allows consuming an ArrayBuffer up to a null terminated point
> and returns the decoded string. Potentially such a method could take a
> Decoder object as argument.

The rationale for specifying the string encoding and decoding
functionality outside the typed array specification is to keep the
typed array spec small and easily implementable. The indexed property
getters and setters on the typed array views, and methods on DataView,
are designed to be implementable with a small amount of assembly code
in JavaScript engines. I'd strongly prefer to continue to design the
encoding/decoding functionality separately from the typed array views.

-Ken

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

[Kenneth Russell]

On Tue, Mar 13, 2012 at 6:10 PM, Jonas Sicking  wrote:
> On Tue, Mar 13, 2012 at 4:08 PM, Kenneth Russell  wrote:
>> Joshua Bell has been working on a string encoding and decoding API
>> that supports the needed encodings, and which is separable from the
>> core typed array API:
>>
>> http://wiki.whatwg.org/wiki/StringEncoding
>>
>> This is the direction I prefer. String encoding and decoding seems to
>> be a complex enough problem that it should be expressed separately
>> from the typed array spec itself.
>
> Very cool. Where do I provide feedback to this? Here?

This list seems like a good place to discuss it.

-Ken

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

[Kenneth Russell]

Joshua Bell has been working on a string encoding and decoding API
that supports the needed encodings, and which is separable from the
core typed array API:

http://wiki.whatwg.org/wiki/StringEncoding

This is the direction I prefer. String encoding and decoding seems to
be a complex enough problem that it should be expressed separately
from the typed array spec itself.

-Ken


On Tue, Mar 13, 2012 at 5:59 PM, Ian Hickson  wrote:
> On Tue, 13 Mar 2012, Jonas Sicking wrote:
>>
>> Something that has come up a couple of times with content authors
>> lately has been the desire to convert an ArrayBuffer (or part thereof)
>> into a decoded string. Similarly being able to encode a string into an
>> ArrayBuffer (or part thereof).
>>
>> Something as simple as
>>
>> DOMString decode(ArrayBufferView source, DOMString encoding);
>> ArrayBufferView encode(DOMString source, DOMString encoding,
>> [optional] ArrayBufferView destination);
>>
>> would go a very long way. The question is where to stick these
>> functions. Internationalization doesn't have a obvious object we can
>> hang functions off of (unlike, for example crypto), and the above
>> names are much too generic to turn into global functions.
>
> Shouldn't this just be another ArrayBufferView type with special
> semantics, like Uint8ClampedArray? DOMStringArray or some such? And/or a
> getString()/setString() method pair on DataView?
>
> Incidentally I _strongly_ suggest we only support UTF-8 here.
>
> --
> Ian Hickson               U+1047E                )\._.,--,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: [whatwg] [CORS] WebKit tainting image instead of throwing error

[Kenneth Russell]

On Tue, Oct 4, 2011 at 12:11 PM, Boris Zbarsky  wrote:
> "default origin behavior" is only relevant when the mode is "No CORS". See
>
http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html#potentially-cors-enabled-fetch
>
> So for images it only applies when the "crossorigin" attribute is not set.
>
> So no, WebKit's implementation is not correct if you were trying to
> implement the spec.
>
> In particular, if "crossorigin" is set, you end up at
>
http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html#potentially-cors-enabled-fetch
> step 3 item 1 in the 'If mode is "Anonymous" or "Use Credentials"'
section,
> which is exactly what was cited in the mail that started this thread.

Right, I see now. Sorry.

On Tue, Oct 4, 2011 at 12:10 PM, Ian Hickson  wrote:

> On Tue, 4 Oct 2011, Anne van Kesteren wrote:
> > On Tue, 04 Oct 2011 20:32:02 +0200, Ian Hickson  wrote:
> > > The idea is that if the server explicitly rejected the CORS request,
> > > then the image should not be usable at all.
> >
> > FWIW, from a CORS-perspective both scenarios are fine. CORS only cares
> > about whether data gets shared in the end. One advantage I can see about
> >  still displaying the image is that the request does
> > not use cookies. Not displaying the image probably makes debugging
> > easier however.
>
> On Tue, 4 Oct 2011, Boris Zbarsky wrote:
> >
> > Displaying images involves sharing data, basically.  That's why we're
> > having to jump through all these hoops
>
> On Tue, 4 Oct 2011, Anne van Kesteren wrote:
> >
> > Sure, but not more than per usual. Note that if you do not specify the
> > crossorigin attribute the image can still get untainted. And if it does
> > not you would still display the image (as always).
>
> The thing is that you can grab the image data (at least the alpha channel
> using 2D canvas) from these images, even with tainting enabled, so if the
> server says no, we really should honour it.
>

I don't think that this is a good argument for the currently specified
behavior. The server only has the option of declining cross-origin access if
the application specified the crossorigin attribute. A hostile application
would simply not specify that attribute, would receive the tainted image,
and would use the timing attack I assume you're referring to to infer the
alpha channel.

The far more common case today is that the server doesn't understand the
CORS request, not that it explicitly forbids cross-origin access to the
resource.

It seems to me that tainting the image if the CORS check fails is more
graceful behavior, but I also see the advantages in early error reporting.

Odin, if you file a bug on bugs.webkit.org, would you CC my email address?

-Ken

On Tue, 4 Oct 2011, Boris Zbarsky wrote:
> >
> > And in particular an  that's in the DOM and fails the
> > CORS checks should not render the image on the page.  Anything else is
> > just broken.
>
> Agreed.
>
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>

Re: [whatwg] [CORS] WebKit tainting image instead of throwing error

[Kenneth Russell]

On Tue, Oct 4, 2011 at 11:55 AM, Boris Zbarsky  wrote:
> On 10/4/11 2:44 PM, Anne van Kesteren wrote:
>>
>> On Tue, 04 Oct 2011 20:32:02 +0200, Ian Hickson  wrote:
>>>
>>> The idea is that if the server explicitly rejected the CORS request, then
>>> the image should not be usable at all.
>>
>> FWIW, from a CORS-perspective both scenarios are fine. CORS only cares
>> about whether data gets shared in the end.
>
> Displaying images involves sharing data, basically.  That's why we're having
> to jump through all these hoops

As far as I can tell the tainting behavior WebKit implements is
correct, and is specified by the text in
http://www.whatwg.org/specs/web-apps/current-work/multipage/embedded-content-1.html#the-img-element
. Scroll down to step 6 in the algorithm for "When the user agent is
to update the image data...". Note that the "default origin behaviour"
is set to "taint" when fetching images.

-Ken

Re: [whatwg] ArrayBuffer and the structured clone algorithm

[Kenneth Russell]

On Mon, Jan 31, 2011 at 10:47 PM, Kenneth Russell  wrote:
> On Mon, Jan 31, 2011 at 3:10 PM, Ian Hickson  wrote:
>> On Fri, 7 Jan 2011, David Flanagan wrote:
>>>
>>> The structured clone algorithm currently allows ImageData and Blob
>>> objects to be cloned but doesn't mention ArrayBuffer.  Is this
>>> intentional?  I assume there are no security issues involved, since one
>>> could copy the bytes of an ArrayBuffer into either a Blob or an
>>> ImageData object in order to clone them.
>>
>> It's intentional in that I'm waiting for ArrayBuffer to be more stable
>> before I add it throughout the spec. (Same with CORS and the various
>> places that might support cross-origin communication, e.g. Web Workers,
>> Server-Sent Events, +, etc.)
>
> There's been some preliminary discussion within the WebGL working
> group (where ArrayBuffer / Typed Arrays originated) about using
> ArrayBuffer with Web Workers in particular. There is a strong desire
> to support handoff of an ArrayBuffer from the main thread to a worker
> and vice versa; this would allow efficient producer/consumer queues to
> be built without violating ECMAScript's shared-nothing semantics.
>
> All of the parties involved are pretty busy getting WebGL 1.0 out the
> door; once that happens, we aim to make one more revision to the Typed
> Array spec to support (1) read-only arrays for more efficient XHRs and
> (2) handoff of ArrayBuffers. Expect public discussions to start in
> about six to eight weeks.

Apologies for not following up to this thread.

Some strawman proposals for the typed array specification were posted
about a month ago and updated just now based on initial feedback. The
primary goal is to define interactions between the structured cloning
algorithm and ArrayBuffers, so that zero-copy data transfer between
Web Workers can be achieved. These proposals, if agreed upon, would
involve changes to the structured cloning algorithm and Web Messaging
spec. Please see:

http://www.khronos.org/registry/typedarray/specs/latest/

Feedback would be greatly appreciated. There is an ongoing thread on
the public WebGL mailing list ( see
https://www.khronos.org/webgl/public-mailing-list/ ), but I'll take
care of coalescing feedback from mailing lists.

-Ken

Re: [whatwg] CORS requests for image and video elements

[Kenneth Russell]

On Thu, May 19, 2011 at 12:27 AM, Henri Sivonen  wrote:
> On Tue, 2011-05-17 at 14:25 -0700, Kenneth Russell wrote:
>> Unfortunately, experimentation indicates that it is
>> not possible to simply send CORS' Origin header with every HTTP GET
>> request for images; some servers do not behave properly when this is
>> done.
>
> How do they behave? Which servers? Why? Has evangelism been attempted?

One service I tested returned a 403 when the Origin: header was
specified and the request originated from certain networks. I
contacted the team running the server and it sounded like this was a
bug that had already been fixed but not deployed. It does indicate
that other servers might mis-handle a CORS request.

In general evangelism is definitely needed to encourage popular media
hosting sites to support CORS. The WebGL working group is certainly
going to perform some outreach since that specification is most
directly impacted by these changes.

-Ken

Re: [whatwg] CORS requests for image and video elements

[Kenneth Russell]

On Tue, May 17, 2011 at 6:11 PM, Ian Hickson  wrote:
> On Tue, 17 May 2011, Kenneth Russell wrote:
>>
>> Last week, a proof of concept of a previously theoretical timing attack
>> against WebGL was published which allows theft of cross-domain images'
>> content.
>>
>> To address this vulnerability it appears to be necessary to ban the use
>> of cross-domain images and videos in WebGL. Unfortunately, doing so will
>> prevent entire classes of applications from being written, and break a
>> not insignificant percentage of current applications.
>>
>> We would like to use CORS to solve this problem; if the server grants
>> access to the image or video, WebGL can use it. Initial discussions with
>> image hosting services have been positive, and it seems that CORS
>> support could be enabled fairly quickly. Many such services already
>> support other access control mechanisms such as Flash's crossdomain.xml.
>> Unfortunately, experimentation indicates that it is not possible to
>> simply send CORS' Origin header with every HTTP GET request for images;
>> some servers do not behave properly when this is done.
>>
>> We would like to propose adding a new Boolean property, useCORS, to
>> HTMLImageElement and HTMLMediaElement, defaulting to false. If set to
>> true, then HTTP requests sent for these elements will set the Origin
>> header from the page's URL. If the Access-Control-Allow-Origin header in
>> the response grants access, then the content's origin will be treated as
>> the same as the page's.
>
> On Tue, 17 May 2011, Jonas Sicking wrote:
>>
>> Does setting "useCORS" make the CORS implementation execute with the
>> "supports credentials" flag set to true or false?
>>
>> When set to true, the request to the server will contain the normal
>> cookies which the user has set for that domain. However, the response
>> from the server will have to contain "Access-Control-Allow-Origin:
>> ". In particular "Access-Control-Allow-Origin:*" will not be
>> treated as a valid response.
>>
>> If the "supports credentials" flag is set to false, the request will be
>> made without cookies, and the server may respond with either
>> "Access-Control-Allow-Origin:*" or "Access-Control-Allow-Origin:
>> ".
>>
>> I propose that the latter mode is used as it will make servers easier to
>> configure as they can just add a static header to all their responses.
>
> On Tue, 17 May 2011, Glenn Maynard wrote:
>>
>> This could be specified, eg.  without credentials and > cors="credentials"> with.  I don't know if there are use cases to
>> justify it.
>
> On Tue, 17 May 2011, Kenneth Russell wrote:
>>
>> In general I think we need to enable as close behavior to the normal
>> image fetching code path as possible. For example, a mashup might
>> require you to be logged in to a site in order to display thumbnails of
>> movie trailers. If normal image fetches send cookies, then it has to be
>> possible to send them when doing a CORS request. I like the idea of > cors> vs. .
>
> I've added a content attribute to , , and  that makes
> the image or media resource be fetched with CORS And have the origin of
> the page if CORS succeeded.
>
> The attribute is "cross-origin" and it has two allowed values,
> "use-credentials" and "anonymous". The latter is the default, so you can
> just say .
>
> This is only a first draft, I'm not sure it's perfect. In particular,
> right now cross-origin media is not allowed at all without this attribute
> (this is not a new change, but I'm not sure it's what implementations do).
> Also, right now as specced if you give a local URL that redirects to a
> remote URL, I don't have CORS kick in, even if you specified cross-origin.
> (This is mostly an editorial thing, I'm going to wait for Anne to get back
> and then see if he can help me out with some editorial changes to CORS to
> make it easier to make that work generally.)
>
> Implementation and author experience feedback would be very welcome on
> this.

Thanks very much for your prompt attention. This sounds like a great
first step. I'll personally try to implement this in WebKit ASAP, and
encourage other browser vendors who are members of the WebGL working
group to do so as well and provide feedback.

> On Tue, 17 May 2011, Kenneth Russell wrote:
>>
>> Perhaps an API could also be added to find out whether the server
>> granted CORS access to the resulting media, tho

Re: [whatwg] CORS requests for image and video elements

[Kenneth Russell]

On Tue, May 17, 2011 at 2:52 PM, Glenn Maynard  wrote:
> On Tue, May 17, 2011 at 5:40 PM, Jonas Sicking  wrote:
>>
>> If the "supports credentials" flag is set to false, the request will
>> be made without cookies, and the server may respond with either
>> "Access-Control-Allow-Origin:*" or "Access-Control-Allow-Origin:
>> ".
>>
>> I propose that the latter mode is used as it will make servers easier
>> to configure as they can just add a static header to all their
>> responses.
>
> This could be specified, eg.  without credentials and  cors="credentials"> with.  I don't know if there are use cases to justify
> it.

In general I think we need to enable as close behavior to the normal
image fetching code path as possible. For example, a mashup might
require you to be logged in to a site in order to display thumbnails
of movie trailers. If normal image fetches send cookies, then it has
to be possible to send them when doing a CORS request. I like the idea
of  vs. .

-Ken

[whatwg] CORS requests for image and video elements

[Kenneth Russell]

Last week, a proof of concept of a previously theoretical timing
attack against WebGL was published which allows theft of cross-domain
images' content.

To address this vulnerability it appears to be necessary to ban the
use of cross-domain images and videos in WebGL. Unfortunately, doing
so will prevent entire classes of applications from being written, and
break a not insignificant percentage of current applications.

We would like to use CORS to solve this problem; if the server grants
access to the image or video, WebGL can use it. Initial discussions
with image hosting services have been positive, and it seems that CORS
support could be enabled fairly quickly. Many such services already
support other access control mechanisms such as Flash's
crossdomain.xml. Unfortunately, experimentation indicates that it is
not possible to simply send CORS' Origin header with every HTTP GET
request for images; some servers do not behave properly when this is
done.

We would like to propose adding a new Boolean property, useCORS, to
HTMLImageElement and HTMLMediaElement, defaulting to false. If set to
true, then HTTP requests sent for these elements will set the Origin
header from the page's URL. If the Access-Control-Allow-Origin header
in the response grants access, then the content's origin will be
treated as the same as the page's.

Perhaps an API could also be added to find out whether the server
granted CORS access to the resulting media, though this is less
important. (Note that the canvas element does not have an explicit API
for querying the origin-clean flag.)

Thoughts on this proposal? We would like to decide on a path quickly
so that we can update both specs and implementations.

Thanks,

-Ken

Re: [whatwg] Canvas.getContext error handling

[Kenneth Russell]

On Wed, Apr 13, 2011 at 4:43 PM, Glenn Maynard  wrote:
> On Wed, Apr 13, 2011 at 4:21 PM, Kenneth Russell  wrote:
>>
>> It's essential to be able to report more detail about why context
>> creation failed. We have already received a lot of feedback from users
>> and developers of popular projects like Google Body that doing so will
>> reduce end user frustration and provide them a path toward getting the
>> content to work.
>
> Hixie says this is a bad idea, for security reasons, and that the UA should
> just tell the user directly:
> http://krijnhoetmer.nl/irc-logs/whatwg/20110413#l-1056
>
> That said, the discussion lead to another approach:
>
> Calling canvas.getContext("webgl", {async: true}) will cause it to *always*
> return an object immediately, without attempting to initialize the
> underlying drawing context.  This context starts out in the "lost" state.
> As long as WebGL is supported by the browser, getContext will never return
> null, even for blacklisted GPUs.  The context is initialized
> asynchronously.  On success, webglcontextrestored is fired, as if the
> context had just come back from a normal context loss.  On failure,
> webglcontextcreationerror is fired with a statusMessage, and possibly a flag
> indicating whether it's a permanent failure (GPU blacklisted) or a
> recoverable one (insufficient resources).
>
> If {async: true} isn't specified, then an initial context failure returns
> null (using the "unsupported contextId" approach), and there's no interface
> to get an error message--people should be strongly discouraged from using
> this API (deprecating it if possible).
>
> (If it's possible to make the backwards-incompatible change to remove sync
> initialization entirely, that would be good to do, but I'm assuming it's
> not.)
>
> There are other fine details (such as feature detection, and possibly
> distinguishing "initializing" from "lost"), but I'll wait for people to give
> their thoughts before delving in deeper.  Aside from giving a consistent way
> to report errors, this allows browsers to initialize WebGL contexts in the
> background.

Providing a programmatic status message about why WebGL initialization
failed (for example, that the user's card or driver is blacklisted) is
not a security issue. First, there would be no way to issue work to
the GPU to exploit any vulnerabilities that might exist, since the app
couldn't get a WebGLRenderingContext. Second, there wouldn't be
detailed enough information in the error message to find out what
graphics card is in use and attempt any other kind of targeted attacks
using other web rendering mechanisms.

Adding support for asynchronous initialization of WebGL is a good
idea, and should be proposed on public_webgl, but this discussion
should focus solely on improving the specification of the existing
synchronous initialization path, and its error conditions.

Given that the proposed asynchronous initialization path above uses
webglcontextcreationerror and provides a status message, I think that
should continue to be the error reporting mechanism for the current
initialization path. Then the introduction of any asynchronous
initialization path would be very simple: the application should
anticipate that it will receive a context lost event immediately,
rather than assuming it can immediately do its initialization. Error
reporting would be identical in the two scenarios.

-Ken

Re: [whatwg] Canvas.getContext error handling

[Kenneth Russell]

On Tue, Apr 12, 2011 at 4:32 PM, Glenn Maynard  wrote:
> Based on some discussion[1], it looks like a clean way to handle the
> "permanent failure" case is: If the GPU is blacklisted, or any other
> permanent error occurs, treat "webgl" as an unsupported context.  This means
> instead of WebGL's context creation algorithm executing and returning null,
> it would never be run at all; instead, step 2 of getContext[2] would return
> null.
>
> For transient errors, eg. too many open contexts, return a WebGL context in
> the "lost" state as Kenneth described.
>
> It was mentioned that the GPU blacklist can change as the browser runs.
> That's supported with this method, since whether a context type is
> "supported" or not can change over time.
>
> Are there any cases where this wouldn't work?
>
> (I'm not sure if or how webglcontextcreationerror fits in this.  It would
> either go away entirely, or be wedged between steps 1 and 2 of getContext; I
> don't know how WebGL would specify that.)

Thanks for the pointer to the IRC logs. It looks like it was a useful
discussion.

It's essential to be able to report more detail about why context
creation failed. We have already received a lot of feedback from users
and developers of popular projects like Google Body that doing so will
reduce end user frustration and provide them a path toward getting the
content to work.

At a minimum, we need to either continue to allow the generation of
webglcontextcreationerror at some point during the getContext() call,
throw an exception from getContext() in this case, or do something
else. Do you have a suggestion on which way to proceed?

-Ken

> [1] http://krijnhoetmer.nl/irc-logs/whatwg/20110413#l-77
> [2]
> http://dev.w3.org/html5/spec/the-canvas-element.html#dom-canvas-getcontext
>
> --
> Glenn Maynard
>

Re: [whatwg] Canvas.getContext error handling

[Kenneth Russell]

On Tue, Apr 12, 2011 at 2:21 AM, Boris Zbarsky  wrote:
> On 4/12/11 12:06 AM, Ian Hickson wrote:
>>>
>>> Now, that's a problem for WebGL, because it's not possible to tell in
>>> advance whether the underlying rendering context can be created.
>>
>> It would be helpful if someone could explain what conditions could lead to
>> a situation where getContext() could fail to return a WebGL context.
>
> In at least Gecko's implementation, creating of an actual GLContext to back
> the WebGL context could fail.  Unfortunately, this can happen any time too
> many WebGL contexts are live; what "too many" means depends on the exact GPU
> resources available.
>
> I think we consider the fact that part a quality of implementation issue,
> though we haven't figured out how to do the "high quality" think here yet.
>  ;)
>
>> Is it something the author can do anything about
>
> Still in Gecko's case, drop references to some GL contexts and canvases, and
> hope for GC to happen.

In Chromium the same basic issue is present. We are close to being
able to forcibly evict "old" OpenGL contexts in response, so that
creation of the current one could proceed. In this case the author
wouldn't need to do anything.

There are two more cases I can think of. The first is when the
graphics card or driver version is blacklisted. In this case
getContext() will always return null, and there's nothing the
developer can do. Currently this would be reported via
webglcontextcreationerror. We could consider throwing an exception
containing the detail message.

The second is if the graphics card is in a powered-off state when the
app calls Canvas.getContext("webgl") -- for example, if a notebook is
awakening from sleep. In this case a good quality WebGL implementation
would like to notify the app when the graphics card is available. In
order for this to work, WebGL would actually need to return a non-null
WebGLRenderingContext, but immediately dispatch a webglcontextlost
event to the canvas.

To sum up, in general I think that whenever getContext("webgl")
returns null, it's unrecoverable in a high quality WebGL
implementation.

-Ken

Re: [whatwg] Canvas.getContext error handling

[Kenneth Russell]

On Sat, Apr 9, 2011 at 7:55 PM, Glenn Maynard  wrote:
> getContext doesn't specify error handling.  WebGL solves this oddly: if an
> error occurs, it dispatches an event with error details at the canvas.  It's
> odd for a synchronous API to report error information with an event; it
> would make a lot more sense to raise an exception.  However, getContext
> doesn't specify error handling for the "Return a new object for contextId"
> algorithms.
>
> The "primary context" should only be set by getContext after "Return a new
> object for contextId" completes successfully; it shouldn't be set on error.
> The cached return value used in step 5 should also only happen after
> success; don't cache a null response.  This way, you can retry getContext on
> failure, and getContext is a straightforward no-op after an error.
>
> (I don't know if the WebGL folks could be convinced to change to throwing an
> exception or if they want to continue returning null and firing an event.)
>
> Related thread:
> https://www.khronos.org/webgl/public-mailing-list/archives/1104/msg00027.html

Just for the record, I'm sure the WebGL working group would be
amenable to making changes in this area. However, there is a general
problem: if getContext() throws an exception, how does the caller know
whether a later call to getContext() might succeed, or will always
fail?

I don't remember all of the discussions in the WebGL working group
which led to the currently defined behavior, but I think that the fact
that 3D graphics contexts can be spontaneously lost, and recovered,
factored into the current design.

-Ken

Re: [whatwg] ArrayBuffer and the structured clone algorithm

[Kenneth Russell]

On Tue, Feb 1, 2011 at 11:08 AM, Tab Atkins Jr.  wrote:
> On Tue, Feb 1, 2011 at 10:04 AM, Anne van Kesteren  wrote:
>> On Tue, 01 Feb 2011 18:36:19 +0100, Boris Zbarsky  wrote:
>>>
>>> On 2/1/11 5:19 AM, Simon Pieters wrote:

 While you're discussing efficient handoff of ArrayBuffer, do you also
 keep in mind efficient handoff of other objects (e.g. ImageData) as
 discussed in this thread?:

 http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029885.html
>>>
>>> For what it's worth, in Gecko that's the same thing, since imagedata is
>>> just a typed array in our implementation.
>>
>> ImageData.data you mean? I wonder if we can still remove CanvasPixelArray.
>
> Only if the out-of-bounds behavior for entries in Typed Arrays matches
> the current clamping behavior for CanvasPixelArray.  I don't see any
> explicit indication of what should be done in the Typed Array spec,
> which I suppose means that they're relying on WebIDL's coercion algos
> to keep things in-range for the given view.  WebIDL has the wrong
> behavior here right now (it wraps), though I think heycan is receptive
> to changing it.

For this reason I think we need to keep CanvasPixelArray distinct. I
certainly hope that Web IDL does not change its conversion rules to
mimic the clamping behavior in CanvasPixelArray. Right now Web IDL
delegates to the ECMA-262 specification for primitive conversions,
which have the wrapping behavior of C-style casts rather than clamping
behavior. Forcing clamping for out-of-range integer values would
impose a significant negative performance constraint on typed arrays.

-Ken

Re: [whatwg] ArrayBuffer and the structured clone algorithm

[Kenneth Russell]

On Mon, Jan 31, 2011 at 3:10 PM, Ian Hickson  wrote:
> On Fri, 7 Jan 2011, David Flanagan wrote:
>>
>> The structured clone algorithm currently allows ImageData and Blob
>> objects to be cloned but doesn't mention ArrayBuffer.  Is this
>> intentional?  I assume there are no security issues involved, since one
>> could copy the bytes of an ArrayBuffer into either a Blob or an
>> ImageData object in order to clone them.
>
> It's intentional in that I'm waiting for ArrayBuffer to be more stable
> before I add it throughout the spec. (Same with CORS and the various
> places that might support cross-origin communication, e.g. Web Workers,
> Server-Sent Events, +, etc.)

There's been some preliminary discussion within the WebGL working
group (where ArrayBuffer / Typed Arrays originated) about using
ArrayBuffer with Web Workers in particular. There is a strong desire
to support handoff of an ArrayBuffer from the main thread to a worker
and vice versa; this would allow efficient producer/consumer queues to
be built without violating ECMAScript's shared-nothing semantics.

All of the parties involved are pretty busy getting WebGL 1.0 out the
door; once that happens, we aim to make one more revision to the Typed
Array spec to support (1) read-only arrays for more efficient XHRs and
(2) handoff of ArrayBuffers. Expect public discussions to start in
about six to eight weeks.

-Ken

Re: [whatwg] ArrayBuffer and ByteArray questions

[Kenneth Russell]

On Wed, Sep 8, 2010 at 5:04 PM, Silvia Pfeiffer
 wrote:
> On Thu, Sep 9, 2010 at 4:37 AM, Chris Marrin  wrote:
>>
>> On Sep 8, 2010, at 11:21 AM, Oliver Hunt wrote:
>>
>> >
>> > On Sep 8, 2010, at 11:13 AM, Chris Marrin wrote:
>> >
>> >> Web Sockets is certainly another candidate, but I meant Web Workers.
>> >> There have been informal discussions on using ArrayBuffers as a way to
>> >> safely share binary data between threads. I don't believe anything has 
>> >> been
>> >> formalized here.
>> >
>> > You can't share data between workers.  There is no (and there cannot be)
>> > any shared state between multiple threads of JS execution.
>>
>> Right. I didn't mean literal sharing. But you can imagine some
>> copy-on-write semantics which would make it more efficient to pass data this
>> way.
>>
>
> Is this then similar to posting ImageData with Web Workers?
> (http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata)
> . I know that these can already be put into a postMessage and they are
> effectively arrays.

It's similar, but we want to define different semantics to achieve
higher performance. Copy-on-write does not work because in a
producer/consumer scenario the producer will always overwrite the same
buffer passed to the consumer, leading to a copy each time. We want to
make the source ArrayBuffer, and any ArrayBufferViews, zero length
upon posting them to a worker or back to the main thread. By
ping-ponging the same ArrayBuffer back and forth you can avoid
allocating new backing store each iteration.

-Ken

Re: [whatwg] ArrayBuffer and ByteArray questions

[Kenneth Russell]

On Wed, Sep 8, 2010 at 11:21 AM, Oliver Hunt  wrote:
>
> On Sep 8, 2010, at 11:13 AM, Chris Marrin wrote:
>
>> Web Sockets is certainly another candidate, but I meant Web Workers. There 
>> have been informal discussions on using ArrayBuffers as a way to safely 
>> share binary data between threads. I don't believe anything has been 
>> formalized here.
>
> You can't share data between workers.  There is no (and there cannot be) any 
> shared state between multiple threads of JS execution.

Let's say "efficiently send" rather than "share". The current thinking
has been around a way to post one ArrayBuffer to a worker which would
"close" that ArrayBuffer and all views on the main thread. The way to
get the same backing store from the worker back to the main thread
would be to post the ArrayBuffer from the worker to the main thread,
at which point the ArrayBuffer and all views on the worker would be
"closed". This ping-ponging would allow efficient implementation of
producer/consumer queues without allocating new backing store each
time the worker wants to produce something for the main thread.

This would require some small API additions to the typed array spec,
and a prototype so we can convince ourselves of its effectiveness.

-Ken

Re: [whatwg] api for fullscreen()

[Kenneth Russell]

On Mon, Feb 1, 2010 at 11:05 AM, Robert O'Callahan  wrote:
> On Tue, Feb 2, 2010 at 7:39 AM, Kenneth Russell  wrote:
>>
>> When you say that the DOM viewport of the element is aligned with the
>> screen when it goes fullscreen, does that mean that the .width and
>> .height properties are changed? Or does it mean that the element's
>> size is changed by a CSS style?
>
> The latter. The window's viewport is aligned with the screen bounds, and by
> default the element is styled with position:fixed; left:0; right:0; top:0;
> bottom:0, which resizes it in CSS to fill the viewport.
>
>> The case I'm thinking about is when a Canvas element is taken
>> fullscreen; on that element changing the .width and .height properties
>> changes the size of the backing store, but applying a CSS style to
>> change its width and height causes the backing store to be scaled to
>> fit. The desired behavior is for the backing store to be resized.
>
>
> The author would have to handle the beginfullscreen event and manually set
> the canvas width/height attributes, e.g. to
> getBoundingClientRect().width/height. I don't think we should change
> width/height attributes automatically, since that has the side effect of
> clearing the canvas.

OK, that sounds reasonable. Thanks.

-Ken

> Rob
> --
> "He was pierced for our transgressions, he was crushed for our iniquities;
> the punishment that brought us peace was upon him, and by his wounds we are
> healed. We all, like sheep, have gone astray, each of us has turned to his
> own way; and the LORD has laid on him the iniquity of us all." [Isaiah
> 53:5-6]
>

Re: [whatwg] api for fullscreen()

[Kenneth Russell]

On Thu, Jan 28, 2010 at 8:55 PM, Robert O'Callahan  wrote:
> On Fri, Jan 29, 2010 at 5:06 PM, Geoff Stearns 
> wrote:
>>>
>>> enterFullscreen always returns immediately. If fullscreen mode is
>>> currently supported and permitted, enterFullscreen dispatches a task that a)
>>> imposes the fullscreen style, b) fires the beginfullscreen event on the
>>> element and c) actually initiates fullscreen display of the element. The UA
>>> may asynchronously display confirmation UI and dispatch the task when the
>>> user has confirmed (or never).
>>
>> Don't you think it would make more sense to dispatch the enterFullscreen
>> event only when the element actually goes fullscreen? If the user clicks the
>> fullscreen button, but then doesn't accept whatever options (likely a
>> security dialog or something) then it doesn't make sense to broadcast an
>> enterFullscreen event, as you'd just have to broadcast an exitFullscreen
>> event right away to show that the user isn't actually in fullscreen.
>
> That was my intent in the last sentence of the paragraph you quoted.
>
>>
>>>
>>> The enableKeys parameter to enterFullscreen is a hint to the UA that the
>>> application would like to be able to receive arbitrary keyboard input.
>>> Otherwise the UA is likely to disable alphanumeric keyboard input. If
>>> enableKeys is specified, the UA might require more severe confirmation UI.
>>
>> This seems overly complicated. I think it would suffice to simply show a
>> dialog the first time a user wants to go fullscreen within a domain with an
>> option to "remember this choice for this domain." Then the user won't have
>> to jump through the hoops again when they return, but will still protect
>> them from random websites going fullscreen and trying to phish things. This
>> way blocking or restricting keyboard events isn't needed.
>
> Those kinds of dialogs are dangerous because users tend to just dismiss them
> without reading. Passive (ignorable and asynchronous) confirmation works
> better.
>
> The enableKeys option would let authors who don't need alphanumeric input
> (video playback) go fullscreen with a low confirmation bar (perhaps none at
> all, if the fullscreen request is in a click event handler).
>
>> Also consider what happens if the user focuses something on another
>> display. Do you then drop out of fullscreen, or just blur() the fullscreen
>> window? (I'd vote to leave it and just blur() it, so you can do things like
>> watch fullscreen video on one display and continue working in the other).
>
> That sounds like a good idea, but I don't think it needs to be in the spec.
> It's up to the UA.
>
>> Another thing to add in here I haven't seen discussed yet is what to show
>> as the background to the fullscreen element. Consider the example of a 16:9
>> video going fullscreen on a 4:3 display. How do you tell the browser to fill
>> in the extra space around the video with black (or whatever other color you
>> want). Is this a custom css element?
>
>
> The  element already letterboxes. So you'd do something like this:
> 
>    src=...>
>   ... controls ...
> 
>
> Making the  fullscreen would override the author geometry and produce
> the effect you want.

When you say that the DOM viewport of the element is aligned with the
screen when it goes fullscreen, does that mean that the .width and
.height properties are changed? Or does it mean that the element's
size is changed by a CSS style?

The case I'm thinking about is when a Canvas element is taken
fullscreen; on that element changing the .width and .height properties
changes the size of the backing store, but applying a CSS style to
change its width and height causes the backing store to be scaled to
fit. The desired behavior is for the backing store to be resized.

-Ken

> Rob
> --
> "He was pierced for our transgressions, he was crushed for our iniquities;
> the punishment that brought us peace was upon him, and by his wounds we are
> healed. We all, like sheep, have gone astray, each of us has turned to his
> own way; and the LORD has laid on him the iniquity of us all." [Isaiah
> 53:5-6]
>

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Fri, Dec 4, 2009 at 9:30 AM, Jason Oster  wrote:
> I guess this suggestion to access the full pixel data in a single array
> element has fallen by the wayside.  Are there any direct objections to
> including additional API to allow this kind of behavior?  It seems most
> developers believe it would be unnecessary, but I haven't heard much in the
> way of reasoning (technical nor personal).
>
> I cannot comment on the "typical uses" of accessing pixel data from script;
> if it is [in general] more important to have each of the R,G,B,A components
> separated for script access, or not.  But for cases involving indexed
> palettes, having the ability to directly treat each pixel as a single
> property is very much desired.
>
> It is not going to provide a huge boost in performance.  At worst, it will
> help make code cleaner.  But at best, it will do that and [slightly?] reduce
> the performance penalty of reading/writing 3 superfluous (in my eyes) array
> accesses.  The only negative aspect I can think of with additional API
> functions is the introduction of new developer confusion; "Which one do I
> use?"

I think you'd get more traction if you had performance measurements;
minimally, profiles showing that this is hot in your current
application. Ideally, you could do a prototype in one of the browsers
supporting WebGL which exposes the ImageData's backing store as a
WebGLUnsignedIntArray. If this showed a significant speedup it would
provide strong motivation.

-Ken

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Sun, Nov 29, 2009 at 8:22 PM, Oliver Hunt  wrote:
>
> On Nov 29, 2009, at 10:59 AM, Kenneth Russell wrote:
>
>> On Sat, Nov 28, 2009 at 9:47 PM, Boris Zbarsky  wrote:
>>> On 11/29/09 12:15 AM, Kenneth Russell wrote:
>>>>>
>>>>> I assume you meant JS bitwise operators?  Do we have any indication that
>>>>> this would be faster than four array property sets?  The bitwise ops in
>>>>> JS
>>>>> are not necessarily particulary fast.
>>>>
>>>> Yes, that's what I meant. I don't have any data on whether this would
>>>> currently be faster than the four separate byte stores.
>>>
>>> Are they even byte stores, necessarily?  I know in Gecko imagedata is just a
>>> JS array at the moment; it stores each of R,G,B,A as a JS Number (with the
>>> usual "if it's an integer store as an integer" optimization arrays do).
>>>  That might well change in the future, and I hope it does, but that's the
>>> current code.
>>>
>>> I can't speak to what the behavior is in Webkit, and in particular whether
>>> it's even the same when using V8 vs Nitro.
>>
>> In Chromium (WebKit + V8), CanvasPixelArray property stores write
>> individual bytes to memory. WebGLByteArray and WebGLUnsignedByteArray
>> behave similarly but have simpler clamping semantics.
>
> I don't know where you're getting that idea from -- the clamping semantics 
> for CanvasPixelArray and WebGLUnsignedByteArray are identical.

CanvasPixelArray specifies that values greater than 255, including
+inf, are clamped to 255 and values less than 0, including -inf, are
clamped to zero. WebGLUnsignedByteArray (as people will see in the
WebGL draft spec this week or next) specifies that the conversion is
done with a C-style cast. The results are different for out-of-range
values.

-Ken

> The CanvasPixelArray implementation in WebKit has always matched the spec and 
> been a clamping bytearray, eg. one byte per channel, per pixel.
>
> Just for future reference for all who are interested: in WebKit the JS 
> interface to a DOM object is merely a binding to a C++ implementation, eg. 
> there's no reason to be concerned about different DOM object behaviour 
> dependent on the JS engine - if there were it a difference would imply a bug 
> rather than a design choice.
>
> --Oliver
>
>>
>> -Ken
>
>

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Sun, Nov 29, 2009 at 11:05 AM, Philip Taylor  wrote:
> On Sun, Nov 29, 2009 at 6:59 PM, Kenneth Russell  wrote:
>> On Sat, Nov 28, 2009 at 9:47 PM, Boris Zbarsky  wrote:
>>> Are they even byte stores, necessarily?  I know in Gecko imagedata is just a
>>> JS array at the moment; it stores each of R,G,B,A as a JS Number (with the
>>> usual "if it's an integer store as an integer" optimization arrays do).
>>>  That might well change in the future, and I hope it does, but that's the
>>> current code.
>>>
>>> I can't speak to what the behavior is in Webkit, and in particular whether
>>> it's even the same when using V8 vs Nitro.
>>
>> In Chromium (WebKit + V8), CanvasPixelArray property stores write
>> individual bytes to memory. WebGLByteArray and WebGLUnsignedByteArray
>> behave similarly but have simpler clamping semantics.
>
> Would it be helpful (for simplicity or performance or consistency etc)
> to change the specification of CanvasPixelArray to have those "simpler
> clamping semantics"? (I don't expect there would be compatibility
> problems with changing it now, particularly since Firefox doesn't
> implement clamping at all in CPA.)

It would. Vladimir Vukicevic from Mozilla was planning to raise this
issue with the whatwg upon release of the first public draft of the
WebGL spec.

-Ken

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Sat, Nov 28, 2009 at 9:47 PM, Boris Zbarsky  wrote:
> On 11/29/09 12:15 AM, Kenneth Russell wrote:
>>>
>>> I assume you meant JS bitwise operators?  Do we have any indication that
>>> this would be faster than four array property sets?  The bitwise ops in
>>> JS
>>> are not necessarily particulary fast.
>>
>> Yes, that's what I meant. I don't have any data on whether this would
>> currently be faster than the four separate byte stores.
>
> Are they even byte stores, necessarily?  I know in Gecko imagedata is just a
> JS array at the moment; it stores each of R,G,B,A as a JS Number (with the
> usual "if it's an integer store as an integer" optimization arrays do).
>  That might well change in the future, and I hope it does, but that's the
> current code.
>
> I can't speak to what the behavior is in Webkit, and in particular whether
> it's even the same when using V8 vs Nitro.

In Chromium (WebKit + V8), CanvasPixelArray property stores write
individual bytes to memory. WebGLByteArray and WebGLUnsignedByteArray
behave similarly but have simpler clamping semantics.

-Ken

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Sat, Nov 28, 2009 at 9:00 PM, Boris Zbarsky  wrote:
> On 11/28/09 11:42 PM, Kenneth Russell wrote:
>>
>> From a technical standpoint, it would be feasible to use the
>> "WebGLUnsignedIntArray" to access the Canvas's pixel data, and
>> assemble RGBA pixels into integer values using just JavaScript logical
>> operators.
>
> I assume you meant JS bitwise operators?  Do we have any indication that
> this would be faster than four array property sets?  The bitwise ops in JS
> are not necessarily particulary fast.

Yes, that's what I meant. I don't have any data on whether this would
currently be faster than the four separate byte stores.

-Ken

Re: [whatwg] Canvas pixel manipulation and performance

[Kenneth Russell]

On Sat, Nov 28, 2009 at 12:44 PM, Jason Oster  wrote:
>
> Once again, I agree.  My confusion on the type-specific arrays for WebGL is
> that they were specific and general enough to use in other cases.  If they
> should not be used in 2D canvas implementations (or elsewhere) then a
> 2D-canvas-specific array or object would be the way forward.

I and other members of the WebGL working group are hoping that the new
array-like types being introduced with this specification will be
general enough to repurpose in other areas. The first public draft of
the spec will be released in the next week or two, and we're hoping
that will enable discussion with the broader web community.

>From a technical standpoint, it would be feasible to use the
"WebGLUnsignedIntArray" to access the Canvas's pixel data, and
assemble RGBA pixels into integer values using just JavaScript logical
operators. To keep things sane, the specification would need to state
something along the lines that the high (logical, not addressing) 8
bits are the red bits and the low 8 bits are the alpha bits. This
means that implementations on big-endian and little-endian machines
would need to store the data differently internally so that the
behavior at the JavaScript level is identical; the WebGL array types
currently deliberately do no byte swapping.

-Ken

Re: [whatwg] An BinaryArchive API for HTML5?

[Kenneth Russell]

On Thu, Jul 30, 2009 at 6:13 AM, Sebastian
Markbåge wrote:
> This suggestion seems similar to Digg's Stream project that uses multipart
> documents: http://github.com/digg/stream
>
> While it would be nice to have a way to parse and handle this in JavaScript,
> it shouldn't be JavaScript's responsibility to work with large object data
> and duplicating it as in-memory data strings.
> The real issue here is the overhead of each additional HTTP request for
> those thousands of objects. But that's useful for all parts of the spec if
> you can download it as a single package even without JavaScript. Images,
> CSS, background-images, JavaScript, etc. Currently you can include graphics
> as data URLs in CSS. Using a package you could package whole widgets (or
> apps) as a single request.
> I'd suggest that this belongs in a lower level API such as the URIs and
> network stack for the tags. You could specify a file within an archive by
> adding an hash with the filename to the URI:
> http://someplace.com/somearchive.tgz#myimage.jpg"; />
> 
> #id { background-image:
> url(http://someplace.com/somearchive.tgz#mybackgroundimage.jpg); }
> 
> http://someplace.com/somearchive.tgz#myscript.js";
> type="text/javascript">
> var img = new Image();
> img.src = "http://someplace.com/somearchive.tgz#myimage.png";;
> Now which packaging format to use would be a discussion on it's own. An easy
> route would be to use multipart/mixed that is already used for this in
> e-mails and can also be gzipped using Content-Encoding.

In the context of the 3d canvas discussions, it looks like there is a
need to load binary blobs of vertex data and feed them to the graphics
card via a JavaScript call. Here is some hypothetical IDL similar to
what is being considered:

[IndexGetter, IndexSetter]
interface CanvasFloatArray {
readonly attribute unsigned long length;
};

interface CanvasRenderingContextGL {
...
typedef unsigned long GLenum;
void glBufferData(in GLenum target, in CanvasFloatArray data,
in GLenum usage);
...
};

Do you have some suggestions for how the data could be transferred
most efficiently to the glBufferData call? As far as I know there is
no tag which could be used to refer to the binary file within the
archive. If there were then presumably it could provide its contents
as a CanvasFloatArray or other type.

-Ken

> On Thu, Jul 30, 2009 at 11:41 AM, Anne van Kesteren 
> wrote:
>>
>> On Thu, 30 Jul 2009 08:49:12 +0200, Gregg Tavares  wrote:
>> > What are people's feelings on adding a Binary Archive API to HTML5?
>>
>> I think it makes more sense to build functionality like this on top of the
>> File API rather than add more things into HTML5.
>>
>>
>> > It seems like it would be useful if there was browser API that let you
>> > download something like gzipped tar files.
>>
>> We already have that: XMLHttpRequest.
>>
>>
>> > The API would look something like
>> >
>> > var request = createArchiveRequest();
>> > request.open("GET", "http://someplace.com/somearchive.tgz";);
>> > request.onfileavailable = doSomethingWithEachFileAsItArrives;
>> > request.send();
>>
>> I don't think we should introduce a new HTTP API.
>>
>>
>> > function doSomethingWithEachFileAsItArrives(binaryBlob) {
>> >   // Load every image in archive
>> >   if (binaryBlob.url.substr(-3) == ".jpg") {
>> >      var image = new Image();
>> >      image.src = binaryBlob.toDataURL();  // or something;
>> >      ...
>> >   }
>> >   // Look for a specific text file
>> >   else if (binaryBlog.url === "myspecial.txt") {
>> >     // getText only works if binaryBlob is valid utf-8 text.
>> >     var text = binaryBlob.getText();
>> >     document.getElementById("content").innerHTML = text;
>> >   }
>> > }
>>
>> Having dedicated support for a subset of archiving formats in within the
>> API for File objects makes sense to me. Latest draft of the File API I know
>> of is
>>
>>  http://dev.w3.org/2006/webapi/FileUpload/publish/FileAPI.xhtml
>>
>> and the mailing list would be public-weba...@w3.org.
>>
>>
>> --
>> Anne van Kesteren
>> http://annevankesteren.nl/
>
>