Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Fri, Aug 8, 2014 at 7:54 PM, Katelyn Gadd wrote: > A multiply blend mode by itself is not sufficient because the image > being rgba multiplied typically has alpha transparency. The closest > approximation is to generate (offline, in software with getImageData) > an image per channel - rgbk - and to source-over blend the 'k' channel > and then additive blend the r/g/b channels with individual alpha. This > approximates the per-channel alpha values with nearly equivalent > results (I say nearly equivalent because some browsers do weird stuff > with gamma/colorspace conversion that completely breaks this.) > > If it helps, you could think of this operation as a layer group in > photoshop. The first layer in the group is the source image, the > second layer is a solid color filled layer containing the rgba > 'multiplier', in multiply mode, and then the layer group has a mask on > it that contains the source image's alpha information. Note that this > representation (a layer group with two layers and a mask) implies that > drawing an image this way requires multiple passes, which is highly > undesirable. My current fallback requires 4 passes, along with 4 > texture changes and two blend state changes. Not wonderful. > I see; you're asking for a feature like Photoshop Color Overlay layer effect. Is that correct? > RGBA multiplication dates back to early fixed-function graphics > pipelines. If a blend with globalAlpha and a premultiplied source is > represented like this: > > result(r, g, b) = ( source-premultiplied(r, g, b) * globalAlpha ) + ( > dest(r, g, b) * (1 - (source(a) * globalAlpha)) ) > > Then if you take a premultiplied color constant and use that as the > multiplier for your image (instead of a global alpha value - this is > the input to rgba multiplication, i.e. a 'vertex color'): > > result(r, g, b) = ( source-premultiplied(r, g, b) * > rgba-multiplier-premultiplied(r, g, b) ) + ( dest(r, g, b) * (1 - > (source(a) * rgba-multiplier-premultiplied(a))) ) > > (Sorry if this is unclear, I don't have a math education) > > So you basically take the global alpha multiplier and you go from that > to a per-channel multiplier. If you're using premultiplied alpha > already, this ends up being pretty straightforward... you just take a > color (premultiplied, like everything else) and use that as your > multiplier. You can multiply directly by each channel since the global > 'alpha' part of the multiplier is already baked in by the > premultiplication step. > > This is a really common primitive since it's so easy to implement, if > not entirely free - you're already doing that global alpha > multiplication, so you just introduce a different multiplier > per-channel, which is really trivial in a SIMD model like the ones > used in computer graphics. You go from vec4 * scalar to vec4 * vec4. > > > Text rendering is the most compelling reason to support this, IMO. > With this feature you can build glyph atlases inside 2d canvases > (using something like freetype, etc), then trivially draw colored > glyphs out of them without having to drop down into getImageData or > use WebGL. It is trivially expressed in most graphics APIs since it > uses the same machinery as a global alpha multiplier - if you're > drawing a premultiplied image with an alpha multiplier in hardware, > you're almost certainly doing vec4 * scalar in your shader. If you're > using the fixed-function pipeline from bad old 3d graphics, vec4 * > scalar didn't even exist - the right hand side was *always* another > vec4 so this feature literally just changed the constant on the right > hand side. > > > I harp on this feature since nearly every 2d game I encounter uses it, > and JSIL has to do it in software. If not for this one feature it > would be very easy to make the vast majority of ported titles Just > Work against canvas, which makes them more likely to run correctly on > mobile. Maybe it would be best to bring this up as a separate topic on this mailing list. (just copy/paste most of your message) > On Fri, Aug 8, 2014 at 5:28 PM, Rik Cabanier wrote: > > > > > > > > On Thu, Aug 7, 2014 at 7:11 PM, Katelyn Gadd wrote: > >> > >> Sorry, in this context rgba multiplication refers to per-channel > >> multipliers (instead of only one multiplier for the alpha channel), so > >> that you can color tint images when drawing them. As mentioned, it's > >> used for fades, drawing colored text, and similar effects. > > > > > > I see. Any reason that this couldn't be done with a 'multiply' blend > mode? > > > >> > >> Premultiplication is a different subject, sorry if I confused you with > >> the similar language. There are past discussions about both in the > >> list archives. > >> >
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
A multiply blend mode by itself is not sufficient because the image being rgba multiplied typically has alpha transparency. The closest approximation is to generate (offline, in software with getImageData) an image per channel - rgbk - and to source-over blend the 'k' channel and then additive blend the r/g/b channels with individual alpha. This approximates the per-channel alpha values with nearly equivalent results (I say nearly equivalent because some browsers do weird stuff with gamma/colorspace conversion that completely breaks this.) If it helps, you could think of this operation as a layer group in photoshop. The first layer in the group is the source image, the second layer is a solid color filled layer containing the rgba 'multiplier', in multiply mode, and then the layer group has a mask on it that contains the source image's alpha information. Note that this representation (a layer group with two layers and a mask) implies that drawing an image this way requires multiple passes, which is highly undesirable. My current fallback requires 4 passes, along with 4 texture changes and two blend state changes. Not wonderful. RGBA multiplication dates back to early fixed-function graphics pipelines. If a blend with globalAlpha and a premultiplied source is represented like this: result(r, g, b) = ( source-premultiplied(r, g, b) * globalAlpha ) + ( dest(r, g, b) * (1 - (source(a) * globalAlpha)) ) Then if you take a premultiplied color constant and use that as the multiplier for your image (instead of a global alpha value - this is the input to rgba multiplication, i.e. a 'vertex color'): result(r, g, b) = ( source-premultiplied(r, g, b) * rgba-multiplier-premultiplied(r, g, b) ) + ( dest(r, g, b) * (1 - (source(a) * rgba-multiplier-premultiplied(a))) ) (Sorry if this is unclear, I don't have a math education) So you basically take the global alpha multiplier and you go from that to a per-channel multiplier. If you're using premultiplied alpha already, this ends up being pretty straightforward... you just take a color (premultiplied, like everything else) and use that as your multiplier. You can multiply directly by each channel since the global 'alpha' part of the multiplier is already baked in by the premultiplication step. This is a really common primitive since it's so easy to implement, if not entirely free - you're already doing that global alpha multiplication, so you just introduce a different multiplier per-channel, which is really trivial in a SIMD model like the ones used in computer graphics. You go from vec4 * scalar to vec4 * vec4. Text rendering is the most compelling reason to support this, IMO. With this feature you can build glyph atlases inside 2d canvases (using something like freetype, etc), then trivially draw colored glyphs out of them without having to drop down into getImageData or use WebGL. It is trivially expressed in most graphics APIs since it uses the same machinery as a global alpha multiplier - if you're drawing a premultiplied image with an alpha multiplier in hardware, you're almost certainly doing vec4 * scalar in your shader. If you're using the fixed-function pipeline from bad old 3d graphics, vec4 * scalar didn't even exist - the right hand side was *always* another vec4 so this feature literally just changed the constant on the right hand side. I harp on this feature since nearly every 2d game I encounter uses it, and JSIL has to do it in software. If not for this one feature it would be very easy to make the vast majority of ported titles Just Work against canvas, which makes them more likely to run correctly on mobile. On Fri, Aug 8, 2014 at 5:28 PM, Rik Cabanier wrote: > > > > On Thu, Aug 7, 2014 at 7:11 PM, Katelyn Gadd wrote: >> >> Sorry, in this context rgba multiplication refers to per-channel >> multipliers (instead of only one multiplier for the alpha channel), so >> that you can color tint images when drawing them. As mentioned, it's >> used for fades, drawing colored text, and similar effects. > > > I see. Any reason that this couldn't be done with a 'multiply' blend mode? > >> >> Premultiplication is a different subject, sorry if I confused you with >> the similar language. There are past discussions about both in the >> list archives. >>
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On 8/8/14, 9:46 PM, Justin Novosad wrote: The way you measured the JS crossing time does not include parameter validations if I am not mistaken. It includes the validation Web IDL does (e.g. "this is an HTMLImageElement") but not the specific validation this method does, correct. If you have 1000 sprite draws that draw from the same sprite sheet, that is 1000 times you are verifying the same image parameter (verifying that the image is complete, that its security origin does not taint the canvas, fetch the decoded image data, etc.), even tough most of this stuff is cached, it is being looked up N times instead of just once. True. I just tested this in a Firefox nightly: I put the early return after we have looked up the decoded image data in our internal cache, or gotten it from the image if not cached, but before we've started doing anything with the position passed to drawImage. The internal cache lookup already includes checks for things like tainting, completeness verification, etc, since there is nothing in the cache if the image is not complete and the canvas is already tainted as needed if we have cached decoded data for this (image,canvas) pair. This version of drawImage takes about 47ns per call when averaged over 1e5 calls (of which, recall, 17ns is the JS call overhead; the other 30ns is the cache lookup, which hits). That's starting to creep up into the 1.5% range of the time I see drawImage taking in the fastest drawImage implementation I have on hand. I will prototype this and come back with some hard data. That would be awesome! -Boris
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
The way you measured the JS crossing time does not include parameter
validations if I am not mistaken. If you have 1000 sprite draws that draw
from the same sprite sheet, that is 1000 times you are verifying the same
image parameter (verifying that the image is complete, that its security
origin does not taint the canvas, fetch the decoded image data, etc.), even
tough most of this stuff is cached, it is being looked up N times instead
of just once.
I will prototype this and come back with some hard data.
On Fri, Aug 8, 2014 at 9:26 PM, Boris Zbarsky wrote:
> On 8/8/14, 8:43 PM, Rik Cabanier wrote:
>
>> The problem is that a large number of drawImage calls have a lot of
>> overhead due to JS crossings and housekeeping.
>>
>
> Could we please quantify this?
>
> I measured the "JS crossings" part of this, because it's easy: Just have
> the C++ side of drawImage return immediately. What I see if I do that is
> that a drawImage call that passes an HTMLImageElement and two integers
> takes about 17ns on my hardware in a current Firefox nightly on Mac.
>
> For scale, a full-up drawImage call that actually does something takes the
> following amounts of time in various browsers I have [1]:
>
> Firefox nightly: ~9500ns/call
> Chrome dev: ~4300ns/call
> Safari 7.0.5 and WebKit nightly: ~3000ns/call
>
> all with noise (when averaged across 10 calls) that's way more than
> 17ns.
>
> So I'm not sure "JS crossings" is a significant performance cost here. I'd
> be interested in which parts of "housekeeping" would be shareable across
> the multiple images in the proposal and how much time those take in
> practice.
>
> -Boris
>
> [1] The specific testcase I used:
>
>
> http://www.mozilla.org/images/mozilla-banner.gif";>
>
> onload = function() {
> var c = document.createElement("canvas").getContext("2d");
> var count = 100;
> var img = document.querySelector("img");
> var start = new Date;
> for (var i = 0; i < count; ++i) c.drawImage(img, 0, 0);
> var stop = new Date;
> console.log((stop - start) / count * 1e6 + "ns per call");
> }
>
>
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On 8/8/14, 8:43 PM, Rik Cabanier wrote:
The problem is that a large number of drawImage calls have a lot of
overhead due to JS crossings and housekeeping.
Could we please quantify this?
I measured the "JS crossings" part of this, because it's easy: Just have
the C++ side of drawImage return immediately. What I see if I do that
is that a drawImage call that passes an HTMLImageElement and two
integers takes about 17ns on my hardware in a current Firefox nightly on
Mac.
For scale, a full-up drawImage call that actually does something takes
the following amounts of time in various browsers I have [1]:
Firefox nightly: ~9500ns/call
Chrome dev: ~4300ns/call
Safari 7.0.5 and WebKit nightly: ~3000ns/call
all with noise (when averaged across 10 calls) that's way more than
17ns.
So I'm not sure "JS crossings" is a significant performance cost here.
I'd be interested in which parts of "housekeeping" would be shareable
across the multiple images in the proposal and how much time those take
in practice.
-Boris
[1] The specific testcase I used:
http://www.mozilla.org/images/mozilla-banner.gif";>
onload = function() {
var c = document.createElement("canvas").getContext("2d");
var count = 100;
var img = document.querySelector("img");
var start = new Date;
for (var i = 0; i < count; ++i) c.drawImage(img, 0, 0);
var stop = new Date;
console.log((stop - start) / count * 1e6 + "ns per call");
}
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Fri, Aug 8, 2014 at 7:25 AM, Ashley Gullen wrote: > As Justin stated, 20% of current Chrome users currently fall back to >> canvas 2d. >> > > 1. What fraction of those 20% actually still get a GPU accelerated canvas > vs. software rendered? Batching will be of very little use to the software > rendered audience, making it an even smaller target market. > There will still be a noticeable gain since we wouldn't have to cross the JS boundary as much. More importantly though, things will still work for non-accelerated canvas while WebGL won't. > 2. In Firefox's case, that number has reduced from 67% to 15% over a > couple of years. Surely in time this will fall even further to a negligible > amount. Why standardise a feature whose target market is disappearing? > How long will it be before we're at a reliable 100%? Graphics drivers are still flaky and it's not as if they only came out a couple of years ago. Note that the supported % for graphics "layers" (accelerated compositing) is much lower. > Small developers that don't have the resources to develop for concurrent >> WebGL and Canvas2D code paths >> > > They don't have to: there are free, high-quality open-source libraries > like Pixi.js that do this already, so even "small" developers have an easy > way to make use of a WebGL renderer without much extra effort. > > >> When do you envision that OpenGL drivers are bug free everywhere? History >> is not on your side here... >> I would much rather have something short term that can be implemented >> with low effort and improves performance. >> > > No software is ever bug-free, but this is irrelevant. To not be > blacklisted, drivers don't need to be perfect, they just need to meet a > reasonable threshold of security and reliability. If a driver is insecure > or crashes constantly it is blacklisted. Drivers are being improved so they > are no longer this poor, and it is not unrealistic to imagine 99%+ of > drivers meeting this threshold in the near future, even if none of them are > completely bug-free. > I think Justin can better fill you in on this. Chrome has to jump through many hoops to make canvas reliable on top of OpenGL and it still suffers from random crashes when you stress the system. Both Safari and Firefox use higher level system calls and are more reliable (albeit slower) than Chrome. > I don't really understand why you and Brian are so opposed to improving >> the performance of canvas 2D. >> > > I see it as a feature targeted at a rapidly disappearing segment of the > market that will disappear in the long run, leaving the web platform with > unnecessary API cruft. > > >> Following your logic, why work on new canvas or SVG features as they can >> theoretically be emulated in WebGL? >> Or now that we have asm.js, why even bother with new JavaScript features? >> > > I am in general against duplication on the web platform, but new features > deserve to be implemented if they have a valid use case or solve a real > problem. > The problem is that a large number of drawImage calls have a lot of overhead due to JS crossings and housekeeping. This proposal solves that. > In this case I don't see that any real problem is being solved, since > widely available frameworks and engines already solve it with WebGL in a > way accessible even to individual developers, and this solution is already > production-grade and widely deployed. > Sure, but that is in WebGL which not everyone wants to use and is less widely supported. > On further thought this particular proposal doesn't even appear to solve > the batching problem very well. Many games consist of large numbers of > rotated sprites. If a canvas2d batching facility needs to break the batch > every time it needs to call rotate(), this will revert back to individual > draw-calls for many kinds of game. WebGL does not have this limitation and > can batch in to single calls objects of a variety of scales, angles, > tiling, opacity and more. This is done by control over individual vertex > positions and texture co-ordinates, which is a fundamental break from the > style of the canvas2d API. Therefore even with the proposed batching > facility, for maximum performance it is still necessary to use WebGL. This > proposal solves a very narrowly defined performance problem. > I'm unsure if I follow. The point of Justin's proposal is to do just that under the hood. Why do you think the batching needs to be broken up? Did you see that the proposal has a matrix per draw? > An alternate solution is for browser vendors to implement canvas2d > entirely in JS on top of WebGL. This reduces per-call overhead by staying > in JS land, while not needing to add any new API surface. In fact it looks > like this has already been attempted here: > https://github.com/corbanbrook/webgl-2d - > Implementing Canvas on top of WebGL is not realistic. Please look into Chrome's implementation to make canvas reliable and fast. This can not be achieved today. I tota
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Thu, Aug 7, 2014 at 7:11 PM, Katelyn Gadd wrote: > Sorry, in this context rgba multiplication refers to per-channel > multipliers (instead of only one multiplier for the alpha channel), so > that you can color tint images when drawing them. As mentioned, it's > used for fades, drawing colored text, and similar effects. > I see. Any reason that this couldn't be done with a 'multiply' blend mode? > Premultiplication is a different subject, sorry if I confused you with > the similar language. There are past discussions about both in the > list archives. > > On Thu, Aug 7, 2014 at 10:59 AM, Rik Cabanier wrote: > > > > > > > > On Mon, Aug 4, 2014 at 4:35 PM, Katelyn Gadd wrote: > >> > >> Many, many uses of drawImage involve transform and/or other state > >> changes per-blit (composite mode, global alpha). > >> > >> I think some of those state changes could be viably batched for most > >> games (composite mode) but others absolutely cannot (global alpha, > >> transform). I see that you handle transform with > >> source-rectangle-and-transform (nice!) but you do not currently handle > >> the others. I'd suggest that this needs to at least handle > >> globalAlpha. > >> > >> Replacing the overloading with individual named methods is something > >> I'm also in favor of. I think it would be ideal if the format-enum > >> argument were not there so that it's easier to feature-detect what > >> formats are available (for example, if globalAlpha data is added later > >> instead of in the '1.0' version of this feature). > > > > > > We can define the functions so they throw a type error if an unknown > enum is > > passed. That way you can feature detect future additions to the enum. > > > > What should be do about error detection in general? If we require the > float > > array to be well formed before drawing, we need an extra pass to make > sure > > that they are correct. > > If we don't require it, we can skip that pass but content could be > partially > > drawn to the canvas before the exception is thrown. > > > >> > >> I get the impression that ordering is implicit for this call - the > >> batch's drawing operations occur in exact order. It might be > >> worthwhile to have a way to indicate to the implementation that you > >> don't care about order, so that it is free to rearrange the draw > >> operations by image and reduce state changes. Doing that in userspace > >> js is made difficult since you can't easily do efficient table lookup > >> for images. > >> > >> if rgba multiplication were to make it into canvas2d sometime in the > >> next decade, that would nicely replace globalAlpha as a per-draw > >> value. This is an analogue to per-vertex colors in 3d graphics and is > >> used in virtually every hardware-accelerated 2d game out there, > >> whether to tint characters when drawing text, fade things in and out, > >> or flash the screen various colors. That would be another reason to > >> make feature detection easier. > >> > >> Would it be possible to sneak rgba multiplication in under the guise > >> of this feature? ;) Without it, I'm forced to use WebGL and reduce > >> compatibility just for something relatively trivial on the > >> implementer's side. (I should note that from what I've heard, Direct2D > >> actually makes this hard to implement. > > > > > > Is this the other proposal to control the format of the canvas buffer > that > > is passed to WebGL? > > > >> > >> On the bright side there's a workaround for RGBA multiplication based > >> on generating per-channel bitmaps from the source bitmap (k, r/g/b), > >> then blending them source-over/add/add/add. drawImageBatch would > >> improve perf for the r/g/b part of it, so it's still an improvement. > >> > >> On Mon, Aug 4, 2014 at 3:39 PM, Robert O'Callahan > > >> wrote: > >> > It looks reasonable to me. > >> > > >> > How do these calls interact with globalAlpha etc? You talk about > >> > decomposing them to individual drawImage calls; does that mean each > >> > image > >> > draw is treated as a separate composite operation? > >> > > >> > Currently you have to choose between using a single image or passing > an > >> > array with one element per image-draw. It seems to me it would be more > >> > flexible to always pass an array but allow the parameters array to > refer > >> > to > >> > an image by index. Did you consider that approach? > >> > > >> > Rob > >> > -- > >> > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso > oaonogoroyo > >> > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo > >> > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo > >> > owohooo > >> > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o > >> > o‘oRoaocoao,o’o > >> > oioso > >> > oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo > oaonoyooonoeo > >> > owohooo > >> > osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono > odoaonogoeoro > >> > ooofo > >> > otohoeo ofoioroeo ooofo ohoeololo. > > > > >
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
> > 1. What fraction of those 20% actually still get a GPU accelerated canvas >> vs. software rendered? Batching will be of very little use to the software >> rendered audience, making it an even smaller target market. >> > > ~25% of Chrome users that do not have gpu-accelerated WebGL do have gpu > accelerated 2d canvas. > So 25% of 20%... you are speccing a feature for just 5% of users, correct? (Since batching likely makes no difference when software rendered) > The problem is the long tail of old devices. > These will still disappear with time, just like old Windows 95 machines have. Is your intent to spec a feature that will no longer be necessary in future? > I will post an update to this thread as soon as I have more compelling > performance data. My goal is to demonstrate that a 2d canvas (with batched > drawImage calls) can yield performance characteristics that are > significantly superior to WebGL's for typical 2D sprite-based game use > cases, particularly on mobile platforms. > Here's some existing data: comparing a WebGL renderer ( http://www.scirra.com/demos/c2/renderperfgl/) with canvas2d ( http://www.scirra.com/demos/c2/renderperf2d) in Chrome on a Nexus 5: canvas2d: 360 objects @ 30 FPS webgl: ~17,500 objects @ 30 FPS WebGL is nearly 50x (fifty!) times faster than canvas2d in these results. Do you really consider this not fast enough? And by just how much further do you hope to improve the result?
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Fri, Aug 8, 2014 at 10:25 AM, Ashley Gullen wrote: > As Justin stated, 20% of current Chrome users currently fall back to >> canvas 2d. >> > > 1. What fraction of those 20% actually still get a GPU accelerated canvas > vs. software rendered? Batching will be of very little use to the software > rendered audience, making it an even smaller target market. > ~25% of Chrome users that do not have gpu-accelerated WebGL do have gpu accelerated 2d canvas. Nonetheless, the software implementation of 2D canvas is way faster than software-emulated WebGL, so it makes sens to fallback to 2D canvas anytime accelerated WebGL is unavailable. > No software is ever bug-free, but this is irrelevant. To not be > blacklisted, drivers don't need to be perfect, they just need to meet a > reasonable threshold of security and reliability. If a driver is insecure > or crashes constantly it is blacklisted. Drivers are being improved so they > are no longer this poor, and it is not unrealistic to imagine 99%+ of > drivers meeting this threshold in the near future, even if none of them are > completely bug-free. > The problem is the long tail of old devices. There is an astonishingly large number of machines in the world running outdated OSes that no longer receive updates, or graphics driver updates. Also, AFAIK, display drivers tend to not have auto-updates the way OSes and browsers do. So even if there is an updated driver out there, most users are unlikely to install it until they try to use some software that requires it explicitly. > > Many games consist of large numbers of rotated sprites. > The proposal includes the possibility of specifying a per draw transformation matrix. And as Katelyn suggested, we could add a variant that takes an alpha value. I will post an update to this thread as soon as I have more compelling performance data. My goal is to demonstrate that a 2d canvas (with batched drawImage calls) can yield performance characteristics that are significantly superior to WebGL's for typical 2D sprite-based game use cases, particularly on mobile platforms. This will be possible by leveraging the browser's compositing framework in ways that are not possible with WebGL. Stay tuned. -Justin
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
> > As Justin stated, 20% of current Chrome users currently fall back to > canvas 2d. > 1. What fraction of those 20% actually still get a GPU accelerated canvas vs. software rendered? Batching will be of very little use to the software rendered audience, making it an even smaller target market. 2. In Firefox's case, that number has reduced from 67% to 15% over a couple of years. Surely in time this will fall even further to a negligible amount. Why standardise a feature whose target market is disappearing? > Small developers that don't have the resources to develop for concurrent > WebGL and Canvas2D code paths > They don't have to: there are free, high-quality open-source libraries like Pixi.js that do this already, so even "small" developers have an easy way to make use of a WebGL renderer without much extra effort. > When do you envision that OpenGL drivers are bug free everywhere? History > is not on your side here... > I would much rather have something short term that can be implemented with > low effort and improves performance. > No software is ever bug-free, but this is irrelevant. To not be blacklisted, drivers don't need to be perfect, they just need to meet a reasonable threshold of security and reliability. If a driver is insecure or crashes constantly it is blacklisted. Drivers are being improved so they are no longer this poor, and it is not unrealistic to imagine 99%+ of drivers meeting this threshold in the near future, even if none of them are completely bug-free. > I don't really understand why you and Brian are so opposed to improving > the performance of canvas 2D. > I see it as a feature targeted at a rapidly disappearing segment of the market that will disappear in the long run, leaving the web platform with unnecessary API cruft. > Following your logic, why work on new canvas or SVG features as they can > theoretically be emulated in WebGL? > Or now that we have asm.js, why even bother with new JavaScript features? > I am in general against duplication on the web platform, but new features deserve to be implemented if they have a valid use case or solve a real problem. In this case I don't see that any real problem is being solved, since widely available frameworks and engines already solve it with WebGL in a way accessible even to individual developers, and this solution is already production-grade and widely deployed. On further thought this particular proposal doesn't even appear to solve the batching problem very well. Many games consist of large numbers of rotated sprites. If a canvas2d batching facility needs to break the batch every time it needs to call rotate(), this will revert back to individual draw-calls for many kinds of game. WebGL does not have this limitation and can batch in to single calls objects of a variety of scales, angles, tiling, opacity and more. This is done by control over individual vertex positions and texture co-ordinates, which is a fundamental break from the style of the canvas2d API. Therefore even with the proposed batching facility, for maximum performance it is still necessary to use WebGL. This proposal solves a very narrowly defined performance problem. An alternate solution is for browser vendors to implement canvas2d entirely in JS on top of WebGL. This reduces per-call overhead by staying in JS land, while not needing to add any new API surface. In fact it looks like this has already been attempted here: https://github.com/corbanbrook/webgl-2d - I'd suggest before speccing this feature that it is researched whether the same performance goals can be achieved with a well-batched JS implementation which reduces the number of calls in to the browser, which is exactly how existing WebGL renderers outperform canvas2d despite both being GPU accelerated. Ashley
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
Sorry, in this context rgba multiplication refers to per-channel multipliers (instead of only one multiplier for the alpha channel), so that you can color tint images when drawing them. As mentioned, it's used for fades, drawing colored text, and similar effects. Premultiplication is a different subject, sorry if I confused you with the similar language. There are past discussions about both in the list archives. On Thu, Aug 7, 2014 at 10:59 AM, Rik Cabanier wrote: > > > > On Mon, Aug 4, 2014 at 4:35 PM, Katelyn Gadd wrote: >> >> Many, many uses of drawImage involve transform and/or other state >> changes per-blit (composite mode, global alpha). >> >> I think some of those state changes could be viably batched for most >> games (composite mode) but others absolutely cannot (global alpha, >> transform). I see that you handle transform with >> source-rectangle-and-transform (nice!) but you do not currently handle >> the others. I'd suggest that this needs to at least handle >> globalAlpha. >> >> Replacing the overloading with individual named methods is something >> I'm also in favor of. I think it would be ideal if the format-enum >> argument were not there so that it's easier to feature-detect what >> formats are available (for example, if globalAlpha data is added later >> instead of in the '1.0' version of this feature). > > > We can define the functions so they throw a type error if an unknown enum is > passed. That way you can feature detect future additions to the enum. > > What should be do about error detection in general? If we require the float > array to be well formed before drawing, we need an extra pass to make sure > that they are correct. > If we don't require it, we can skip that pass but content could be partially > drawn to the canvas before the exception is thrown. > >> >> I get the impression that ordering is implicit for this call - the >> batch's drawing operations occur in exact order. It might be >> worthwhile to have a way to indicate to the implementation that you >> don't care about order, so that it is free to rearrange the draw >> operations by image and reduce state changes. Doing that in userspace >> js is made difficult since you can't easily do efficient table lookup >> for images. >> >> if rgba multiplication were to make it into canvas2d sometime in the >> next decade, that would nicely replace globalAlpha as a per-draw >> value. This is an analogue to per-vertex colors in 3d graphics and is >> used in virtually every hardware-accelerated 2d game out there, >> whether to tint characters when drawing text, fade things in and out, >> or flash the screen various colors. That would be another reason to >> make feature detection easier. >> >> Would it be possible to sneak rgba multiplication in under the guise >> of this feature? ;) Without it, I'm forced to use WebGL and reduce >> compatibility just for something relatively trivial on the >> implementer's side. (I should note that from what I've heard, Direct2D >> actually makes this hard to implement. > > > Is this the other proposal to control the format of the canvas buffer that > is passed to WebGL? > >> >> On the bright side there's a workaround for RGBA multiplication based >> on generating per-channel bitmaps from the source bitmap (k, r/g/b), >> then blending them source-over/add/add/add. drawImageBatch would >> improve perf for the r/g/b part of it, so it's still an improvement. >> >> On Mon, Aug 4, 2014 at 3:39 PM, Robert O'Callahan >> wrote: >> > It looks reasonable to me. >> > >> > How do these calls interact with globalAlpha etc? You talk about >> > decomposing them to individual drawImage calls; does that mean each >> > image >> > draw is treated as a separate composite operation? >> > >> > Currently you have to choose between using a single image or passing an >> > array with one element per image-draw. It seems to me it would be more >> > flexible to always pass an array but allow the parameters array to refer >> > to >> > an image by index. Did you consider that approach? >> > >> > Rob >> > -- >> > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo >> > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo >> > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo >> > owohooo >> > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o >> > o‘oRoaocoao,o’o >> > oioso >> > oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo >> > owohooo >> > osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro >> > ooofo >> > otohoeo ofoioroeo ooofo ohoeololo. > >
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Mon, Aug 4, 2014 at 4:35 PM, Katelyn Gadd wrote: > Many, many uses of drawImage involve transform and/or other state > changes per-blit (composite mode, global alpha). > > I think some of those state changes could be viably batched for most > games (composite mode) but others absolutely cannot (global alpha, > transform). I see that you handle transform with > source-rectangle-and-transform (nice!) but you do not currently handle > the others. I'd suggest that this needs to at least handle > globalAlpha. > > Replacing the overloading with individual named methods is something > I'm also in favor of. I think it would be ideal if the format-enum > argument were not there so that it's easier to feature-detect what > formats are available (for example, if globalAlpha data is added later > instead of in the '1.0' version of this feature). > We can define the functions so they throw a type error if an unknown enum is passed. That way you can feature detect future additions to the enum. What should be do about error detection in general? If we require the float array to be well formed before drawing, we need an extra pass to make sure that they are correct. If we don't require it, we can skip that pass but content could be partially drawn to the canvas before the exception is thrown. > I get the impression that ordering is implicit for this call - the > batch's drawing operations occur in exact order. It might be > worthwhile to have a way to indicate to the implementation that you > don't care about order, so that it is free to rearrange the draw > operations by image and reduce state changes. Doing that in userspace > js is made difficult since you can't easily do efficient table lookup > for images. > > if rgba multiplication were to make it into canvas2d sometime in the > next decade, that would nicely replace globalAlpha as a per-draw > value. This is an analogue to per-vertex colors in 3d graphics and is > used in virtually every hardware-accelerated 2d game out there, > whether to tint characters when drawing text, fade things in and out, > or flash the screen various colors. That would be another reason to > make feature detection easier. > > Would it be possible to sneak rgba multiplication in under the guise > of this feature? ;) Without it, I'm forced to use WebGL and reduce > compatibility just for something relatively trivial on the > implementer's side. (I should note that from what I've heard, Direct2D > actually makes this hard to implement. > Is this the other proposal to control the format of the canvas buffer that is passed to WebGL? > On the bright side there's a workaround for RGBA multiplication based > on generating per-channel bitmaps from the source bitmap (k, r/g/b), > then blending them source-over/add/add/add. drawImageBatch would > improve perf for the r/g/b part of it, so it's still an improvement. > > On Mon, Aug 4, 2014 at 3:39 PM, Robert O'Callahan > wrote: > > It looks reasonable to me. > > > > How do these calls interact with globalAlpha etc? You talk about > > decomposing them to individual drawImage calls; does that mean each image > > draw is treated as a separate composite operation? > > > > Currently you have to choose between using a single image or passing an > > array with one element per image-draw. It seems to me it would be more > > flexible to always pass an array but allow the parameters array to refer > to > > an image by index. Did you consider that approach? > > > > Rob > > -- > > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo > > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo > > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo > > owohooo > > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o > > oioso > > oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo > > owohooo > > osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro > > ooofo > > otohoeo ofoioroeo ooofo ohoeololo. >
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Tue, Aug 5, 2014 at 5:55 PM, Ashley Gullen wrote: > If your argument is that WebGL sometimes falls back to canvas2d, this > generally only happens when the system has crappy drivers that are > blacklisted for being insecure/unstable. The solution to this is to develop > and distribute better drivers that are not blacklisted. This is already > happening and making good progress - according to Mozilla's stats, Firefox > users who get WebGL support has increased from 33% in 2011 to 85% in 2014 ( > http://people.mozilla.org/~bjacob/gfx_features_stats/). As Justin stated, 20% of current Chrome users currently fall back to canvas 2d. This is a large chunk of the market. Small developers that don't have the resources to develop for concurrent WebGL and Canvas2D code paths will certainly code for just Canvas since that will give them close to 100%. > I feel it is likely > to continue to approach ubiquitous WebGL support, making fallbacks > unnecessary. This also solves the problem of having to have dual renderer > implementations: only the WebGL renderer will be necessary, and this is far > more compelling than a souped-up canvas2d, since WebGL can use shader > effects, have advanced control over textures and co-ordinates, also do 3D, > and so on. This cannot all be brought to canvas2d without simply > reinventing WebGL. Further, crappy drivers can also cause software-rendered > canvas2d as well, which is likely so slow to begin with that batching will > have no important performance improvement. Software-rendered WebGL is just > another workaround to crappy drivers (or in rare cases systems without > GPUs, but then who's going to be gunning for high performance there?) and > there is still no guarantee falling back to canvas2d will be > GPU-accelerated, especially since the system already has such poor drivers > that the browser has blacklisted it for WebGL support. > > The real problem is that there is not 100% WebGL support everywhere, but > with drivers improving and Apple and Microsoft on board I'm sure that will > fix itself eventually. Please don't spec features to improve canvas2d > performance in the mean time; I don't see it having any long-term utility > for the web platform. When do you envision that OpenGL drivers are bug free everywhere? History is not on your side here... I would much rather have something short term that can be implemented with low effort and improves performance. I don't really understand why you and Brian are so opposed to improving the performance of canvas 2D. There are a lot of people that use and like its API. WebGL on the other hand, has a very steep learning curve and problems are not always obvious. Following your logic, why work on new canvas or SVG features as they can theoretically be emulated in WebGL? Or now that we have asm.js, why even bother with new JavaScript features? > On 5 August 2014 16:21, Justin Novosad wrote: > > > On Mon, Aug 4, 2014 at 6:39 PM, Robert O'Callahan > > wrote: > > > > > It looks reasonable to me. > > > > > > How do these calls interact with globalAlpha etc? You talk about > > > decomposing them to individual drawImage calls; does that mean each > image > > > draw is treated as a separate composite operation? > > > > > > > Composited separately is the intent. A possible internal optimization: > the > > implementation could group non-overlapping draw and composite them > > together. > > > > > > > Currently you have to choose between using a single image or passing an > > > array with one element per image-draw. It seems to me it would be more > > > flexible to always pass an array but allow the parameters array to > refer > > to > > > an image by index. Did you consider that approach? > > > > > > > Had not thought of that. Good idea. > > > > On Mon, Aug 4, 2014 at 7:35 PM, Katelyn Gadd wrote: > > > > > I'd suggest that this needs to at least handle > > > globalAlpha. > > > > > > > It would be trivial to add a an addition format that includes alpha. > > > > > > > Replacing the overloading with individual named methods is something > > > I'm also in favor of. > > > > > > That's something I pondered and was not sure about. Eliminating the > > parameter format argument would be nice. Your feature-detection argument > is > > a really good reason. > > > > > > > > I get the impression that ordering is implicit for this call - the > > > batch's drawing operations occur in exact order. It might be > > > worthwhile to have a way to indicate to the implementation that you > > > don't care about order, so that it is free to rearrange the draw > > > operations by image and reduce state changes. Doing that in userspace > > > js is made difficult since you can't easily do efficient table lookup > > > for images. > > > > > > > I am not sure exposing that in the API is a good idea because it opens > the > > door to undefined behavior. It could result in different implementations > > producing drastically different yet compliant results. > > P
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Tue, Aug 5, 2014 at 10:04 AM, Brian Blakely wrote: > On Tue, Aug 5, 2014 at 11:21 AM, Justin Novosad wrote: > > > On Tue, Aug 5, 2014 at 7:47 AM, Ashley Gullen wrote: > > > > > I am against this suggestion. If you are serious about performance then > > > you should use WebGL and implement your own batching system, which is > > what > > > every major 2D HTML5 game framework I'm aware of does already. Adding > > > batching features to canvas2d has three disadvantages in my view: > > > > > > 1. Major 2D engines already support WebGL, so even if this new feature > > was > > > supported, in practice it would not be used. > > > 2. There is opportunity cost in speccing something that is unlikely to > be > > > used and already well-covered by another part of the web platform. We > > could > > > be speccing something else more useful. > > > 3. canvas2d should not end up being specced closer and closer to WebGL: > > > canvas2d should be kept as a high-level easy-to-use API even with > > > performance cost, whereas WebGL is the low-level high-performance API. > > > These are two different use cases and it's good to have two different > > APIs > > > to cover them. If you want to keep improving canvas2d performance I > would > > > worry you will simply end up reinventing WebGL. > > > > > > > > These are good points. The only counter argument I have to that is that a > > fallback from WebGL to canvas2d is unfortunately necessary for a > > significant fraction of users. Even on web-browsers that do support > WebGL, > > gl may be emulated in software, which can be detected by web apps and > > warrants falling back to canvas2d (approx. 20% of Chrome users, for > > example). I realize that there is currently a clear ease of use vs. > > performance dichotomy between 2d and webgl, and this proposal blurs that > > boundary. Nonetheless, there is developer-driven demand for this based > on a > > real-world problem. Also, if 2D canvas had better performance > > characteristics, it would not be necessary for some game engines to have > > dual (2d/webgl) implementations. > > > > -Justin > > > > My take is similar to Ashley's, and I wonder how buffing up the toy API > (2D) compensates for the fact that the performance API (GL) has > compatibility problems, even on platforms that support it. If the goal is > to solve the latter, why not introduce more direct proposals? > Can you explain what you're asking for? Are you asking for a proposal that fixes compatibility?
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
If your argument is that WebGL sometimes falls back to canvas2d, this generally only happens when the system has crappy drivers that are blacklisted for being insecure/unstable. The solution to this is to develop and distribute better drivers that are not blacklisted. This is already happening and making good progress - according to Mozilla's stats, Firefox users who get WebGL support has increased from 33% in 2011 to 85% in 2014 ( http://people.mozilla.org/~bjacob/gfx_features_stats/). I feel it is likely to continue to approach ubiquitous WebGL support, making fallbacks unnecessary. This also solves the problem of having to have dual renderer implementations: only the WebGL renderer will be necessary, and this is far more compelling than a souped-up canvas2d, since WebGL can use shader effects, have advanced control over textures and co-ordinates, also do 3D, and so on. This cannot all be brought to canvas2d without simply reinventing WebGL. Further, crappy drivers can also cause software-rendered canvas2d as well, which is likely so slow to begin with that batching will have no important performance improvement. Software-rendered WebGL is just another workaround to crappy drivers (or in rare cases systems without GPUs, but then who's going to be gunning for high performance there?) and there is still no guarantee falling back to canvas2d will be GPU-accelerated, especially since the system already has such poor drivers that the browser has blacklisted it for WebGL support. The real problem is that there is not 100% WebGL support everywhere, but with drivers improving and Apple and Microsoft on board I'm sure that will fix itself eventually. Please don't spec features to improve canvas2d performance in the mean time; I don't see it having any long-term utility for the web platform. Ashley On 5 August 2014 16:21, Justin Novosad wrote: > On Mon, Aug 4, 2014 at 6:39 PM, Robert O'Callahan > wrote: > > > It looks reasonable to me. > > > > How do these calls interact with globalAlpha etc? You talk about > > decomposing them to individual drawImage calls; does that mean each image > > draw is treated as a separate composite operation? > > > > Composited separately is the intent. A possible internal optimization: the > implementation could group non-overlapping draw and composite them > together. > > > > Currently you have to choose between using a single image or passing an > > array with one element per image-draw. It seems to me it would be more > > flexible to always pass an array but allow the parameters array to refer > to > > an image by index. Did you consider that approach? > > > > Had not thought of that. Good idea. > > On Mon, Aug 4, 2014 at 7:35 PM, Katelyn Gadd wrote: > > > I'd suggest that this needs to at least handle > > globalAlpha. > > > > It would be trivial to add a an addition format that includes alpha. > > > > Replacing the overloading with individual named methods is something > > I'm also in favor of. > > > That's something I pondered and was not sure about. Eliminating the > parameter format argument would be nice. Your feature-detection argument is > a really good reason. > > > > > I get the impression that ordering is implicit for this call - the > > batch's drawing operations occur in exact order. It might be > > worthwhile to have a way to indicate to the implementation that you > > don't care about order, so that it is free to rearrange the draw > > operations by image and reduce state changes. Doing that in userspace > > js is made difficult since you can't easily do efficient table lookup > > for images. > > > > I am not sure exposing that in the API is a good idea because it opens the > door to undefined behavior. It could result in different implementations > producing drastically different yet compliant results. > Perhaps implementations could auto-detect draw operations that are > commutative based on a quick overlap analysis, and use that knowledge to > automatically group draw calls that use similar drawing state (e.g. the > same source GPU texture) > > > > > > Would it be possible to sneak rgba multiplication in under the guise > > of this feature? ;) Without it, I'm forced to use WebGL and reduce > > compatibility just for something relatively trivial on the > > implementer's side. (I should note that from what I've heard, Direct2D > > actually makes this hard to implement. > > > > I think that would make this feature significantly more complex to spec, > and to implement. It really should be treated as an orthogonal feature > request. Your suggestion is very use-case specific. A more general > incarnation of your request would be to have a blendMode parameter that > that determines how the source image gets blended with the fillStyle. With > that, you resolve your specific use case by setling the fill style to an > rgba color, and using a multiply blend op to use it to modulate images. > > On Tue, Aug 5, 2014 at 7:47 AM, Ashley Gullen wrote: > > > I am against
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Tue, Aug 5, 2014 at 11:21 AM, Justin Novosad wrote: > On Tue, Aug 5, 2014 at 7:47 AM, Ashley Gullen wrote: > > > I am against this suggestion. If you are serious about performance then > > you should use WebGL and implement your own batching system, which is > what > > every major 2D HTML5 game framework I'm aware of does already. Adding > > batching features to canvas2d has three disadvantages in my view: > > > > 1. Major 2D engines already support WebGL, so even if this new feature > was > > supported, in practice it would not be used. > > 2. There is opportunity cost in speccing something that is unlikely to be > > used and already well-covered by another part of the web platform. We > could > > be speccing something else more useful. > > 3. canvas2d should not end up being specced closer and closer to WebGL: > > canvas2d should be kept as a high-level easy-to-use API even with > > performance cost, whereas WebGL is the low-level high-performance API. > > These are two different use cases and it's good to have two different > APIs > > to cover them. If you want to keep improving canvas2d performance I would > > worry you will simply end up reinventing WebGL. > > > > > These are good points. The only counter argument I have to that is that a > fallback from WebGL to canvas2d is unfortunately necessary for a > significant fraction of users. Even on web-browsers that do support WebGL, > gl may be emulated in software, which can be detected by web apps and > warrants falling back to canvas2d (approx. 20% of Chrome users, for > example). I realize that there is currently a clear ease of use vs. > performance dichotomy between 2d and webgl, and this proposal blurs that > boundary. Nonetheless, there is developer-driven demand for this based on a > real-world problem. Also, if 2D canvas had better performance > characteristics, it would not be necessary for some game engines to have > dual (2d/webgl) implementations. > > -Justin > My take is similar to Ashley's, and I wonder how buffing up the toy API (2D) compensates for the fact that the performance API (GL) has compatibility problems, even on platforms that support it. If the goal is to solve the latter, why not introduce more direct proposals?
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
On Mon, Aug 4, 2014 at 6:39 PM, Robert O'Callahan wrote: > It looks reasonable to me. > > How do these calls interact with globalAlpha etc? You talk about > decomposing them to individual drawImage calls; does that mean each image > draw is treated as a separate composite operation? > Composited separately is the intent. A possible internal optimization: the implementation could group non-overlapping draw and composite them together. > Currently you have to choose between using a single image or passing an > array with one element per image-draw. It seems to me it would be more > flexible to always pass an array but allow the parameters array to refer to > an image by index. Did you consider that approach? > Had not thought of that. Good idea. On Mon, Aug 4, 2014 at 7:35 PM, Katelyn Gadd wrote: > I'd suggest that this needs to at least handle > globalAlpha. > It would be trivial to add a an addition format that includes alpha. > Replacing the overloading with individual named methods is something > I'm also in favor of. That's something I pondered and was not sure about. Eliminating the parameter format argument would be nice. Your feature-detection argument is a really good reason. > > I get the impression that ordering is implicit for this call - the > batch's drawing operations occur in exact order. It might be > worthwhile to have a way to indicate to the implementation that you > don't care about order, so that it is free to rearrange the draw > operations by image and reduce state changes. Doing that in userspace > js is made difficult since you can't easily do efficient table lookup > for images. > I am not sure exposing that in the API is a good idea because it opens the door to undefined behavior. It could result in different implementations producing drastically different yet compliant results. Perhaps implementations could auto-detect draw operations that are commutative based on a quick overlap analysis, and use that knowledge to automatically group draw calls that use similar drawing state (e.g. the same source GPU texture) > > Would it be possible to sneak rgba multiplication in under the guise > of this feature? ;) Without it, I'm forced to use WebGL and reduce > compatibility just for something relatively trivial on the > implementer's side. (I should note that from what I've heard, Direct2D > actually makes this hard to implement. > I think that would make this feature significantly more complex to spec, and to implement. It really should be treated as an orthogonal feature request. Your suggestion is very use-case specific. A more general incarnation of your request would be to have a blendMode parameter that that determines how the source image gets blended with the fillStyle. With that, you resolve your specific use case by setling the fill style to an rgba color, and using a multiply blend op to use it to modulate images. On Tue, Aug 5, 2014 at 7:47 AM, Ashley Gullen wrote: > I am against this suggestion. If you are serious about performance then > you should use WebGL and implement your own batching system, which is what > every major 2D HTML5 game framework I'm aware of does already. Adding > batching features to canvas2d has three disadvantages in my view: > > 1. Major 2D engines already support WebGL, so even if this new feature was > supported, in practice it would not be used. > 2. There is opportunity cost in speccing something that is unlikely to be > used and already well-covered by another part of the web platform. We could > be speccing something else more useful. > 3. canvas2d should not end up being specced closer and closer to WebGL: > canvas2d should be kept as a high-level easy-to-use API even with > performance cost, whereas WebGL is the low-level high-performance API. > These are two different use cases and it's good to have two different APIs > to cover them. If you want to keep improving canvas2d performance I would > worry you will simply end up reinventing WebGL. > > These are good points. The only counter argument I have to that is that a fallback from WebGL to canvas2d is unfortunately necessary for a significant fraction of users. Even on web-browsers that do support WebGL, gl may be emulated in software, which can be detected by web apps and warrants falling back to canvas2d (approx. 20% of Chrome users, for example). I realize that there is currently a clear ease of use vs. performance dichotomy between 2d and webgl, and this proposal blurs that boundary. Nonetheless, there is developer-driven demand for this based on a real-world problem. Also, if 2D canvas had better performance characteristics, it would not be necessary for some game engines to have dual (2d/webgl) implementations. -Justin
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
I am against this suggestion. If you are serious about performance then you should use WebGL and implement your own batching system, which is what every major 2D HTML5 game framework I'm aware of does already. Adding batching features to canvas2d has three disadvantages in my view: 1. Major 2D engines already support WebGL, so even if this new feature was supported, in practice it would not be used. 2. There is opportunity cost in speccing something that is unlikely to be used and already well-covered by another part of the web platform. We could be speccing something else more useful. 3. canvas2d should not end up being specced closer and closer to WebGL: canvas2d should be kept as a high-level easy-to-use API even with performance cost, whereas WebGL is the low-level high-performance API. These are two different use cases and it's good to have two different APIs to cover them. If you want to keep improving canvas2d performance I would worry you will simply end up reinventing WebGL. As developers of a HTML5 game engine with WebGL support and fallback to canvas2d, this has no utility to us at all. It would be more useful to work on WebGL 2. Ashley Gullen Scirra.com On 5 August 2014 00:35, Katelyn Gadd wrote: > Many, many uses of drawImage involve transform and/or other state > changes per-blit (composite mode, global alpha). > > I think some of those state changes could be viably batched for most > games (composite mode) but others absolutely cannot (global alpha, > transform). I see that you handle transform with > source-rectangle-and-transform (nice!) but you do not currently handle > the others. I'd suggest that this needs to at least handle > globalAlpha. > > Replacing the overloading with individual named methods is something > I'm also in favor of. I think it would be ideal if the format-enum > argument were not there so that it's easier to feature-detect what > formats are available (for example, if globalAlpha data is added later > instead of in the '1.0' version of this feature). > > I get the impression that ordering is implicit for this call - the > batch's drawing operations occur in exact order. It might be > worthwhile to have a way to indicate to the implementation that you > don't care about order, so that it is free to rearrange the draw > operations by image and reduce state changes. Doing that in userspace > js is made difficult since you can't easily do efficient table lookup > for images. > > if rgba multiplication were to make it into canvas2d sometime in the > next decade, that would nicely replace globalAlpha as a per-draw > value. This is an analogue to per-vertex colors in 3d graphics and is > used in virtually every hardware-accelerated 2d game out there, > whether to tint characters when drawing text, fade things in and out, > or flash the screen various colors. That would be another reason to > make feature detection easier. > > Would it be possible to sneak rgba multiplication in under the guise > of this feature? ;) Without it, I'm forced to use WebGL and reduce > compatibility just for something relatively trivial on the > implementer's side. (I should note that from what I've heard, Direct2D > actually makes this hard to implement. > > On the bright side there's a workaround for RGBA multiplication based > on generating per-channel bitmaps from the source bitmap (k, r/g/b), > then blending them source-over/add/add/add. drawImageBatch would > improve perf for the r/g/b part of it, so it's still an improvement. > > On Mon, Aug 4, 2014 at 3:39 PM, Robert O'Callahan > wrote: > > It looks reasonable to me. > > > > How do these calls interact with globalAlpha etc? You talk about > > decomposing them to individual drawImage calls; does that mean each image > > draw is treated as a separate composite operation? > > > > Currently you have to choose between using a single image or passing an > > array with one element per image-draw. It seems to me it would be more > > flexible to always pass an array but allow the parameters array to refer > to > > an image by index. Did you consider that approach? > > > > Rob > > -- > > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo > > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo > > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo > > owohooo > > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o > > oioso > > oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo > > owohooo > > osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro > > ooofo > > otohoeo ofoioroeo ooofo ohoeololo. >
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
Many, many uses of drawImage involve transform and/or other state changes per-blit (composite mode, global alpha). I think some of those state changes could be viably batched for most games (composite mode) but others absolutely cannot (global alpha, transform). I see that you handle transform with source-rectangle-and-transform (nice!) but you do not currently handle the others. I'd suggest that this needs to at least handle globalAlpha. Replacing the overloading with individual named methods is something I'm also in favor of. I think it would be ideal if the format-enum argument were not there so that it's easier to feature-detect what formats are available (for example, if globalAlpha data is added later instead of in the '1.0' version of this feature). I get the impression that ordering is implicit for this call - the batch's drawing operations occur in exact order. It might be worthwhile to have a way to indicate to the implementation that you don't care about order, so that it is free to rearrange the draw operations by image and reduce state changes. Doing that in userspace js is made difficult since you can't easily do efficient table lookup for images. if rgba multiplication were to make it into canvas2d sometime in the next decade, that would nicely replace globalAlpha as a per-draw value. This is an analogue to per-vertex colors in 3d graphics and is used in virtually every hardware-accelerated 2d game out there, whether to tint characters when drawing text, fade things in and out, or flash the screen various colors. That would be another reason to make feature detection easier. Would it be possible to sneak rgba multiplication in under the guise of this feature? ;) Without it, I'm forced to use WebGL and reduce compatibility just for something relatively trivial on the implementer's side. (I should note that from what I've heard, Direct2D actually makes this hard to implement. On the bright side there's a workaround for RGBA multiplication based on generating per-channel bitmaps from the source bitmap (k, r/g/b), then blending them source-over/add/add/add. drawImageBatch would improve perf for the r/g/b part of it, so it's still an improvement. On Mon, Aug 4, 2014 at 3:39 PM, Robert O'Callahan wrote: > It looks reasonable to me. > > How do these calls interact with globalAlpha etc? You talk about > decomposing them to individual drawImage calls; does that mean each image > draw is treated as a separate composite operation? > > Currently you have to choose between using a single image or passing an > array with one element per image-draw. It seems to me it would be more > flexible to always pass an array but allow the parameters array to refer to > an image by index. Did you consider that approach? > > Rob > -- > oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo > owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo > osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo > owohooo > osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o > oioso > oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo > owohooo > osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro > ooofo > otohoeo ofoioroeo ooofo ohoeololo.
Re: [whatwg] [2D Canvas] Proposal: batch variants of drawImage
It looks reasonable to me. How do these calls interact with globalAlpha etc? You talk about decomposing them to individual drawImage calls; does that mean each image draw is treated as a separate composite operation? Currently you have to choose between using a single image or passing an array with one element per image-draw. It seems to me it would be more flexible to always pass an array but allow the parameters array to refer to an image by index. Did you consider that approach? Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo.
