Re: [whatwg] createImageData should take unsigned long
On Fri, 04 Sep 2009 02:44:25 +0200, Oliver Hunt oli...@apple.com wrote: On Sep 3, 2009, at 4:50 PM, Robert O'Callahan wrote: On Fri, Sep 4, 2009 at 4:48 AM, Oliver Hunt oli...@apple.com wrote: On Sep 3, 2009, at 4:54 AM, Ian Hickson wrote: Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( WebKit's implementation has always worked with high dpi backing stores and follows the spec accordingly. Under what circumstances do you use more than one device pixel per CSS pixel? Does it require the user to turn on UI scaling on Mac? Regardless, I bet that most people using Webkit to write scripts using getImageData still get it wrong, because they have normal screens. Implementing high-res backing store in more browsers won't solve this problem, not until the average developer has a high-dpi screen. But I repeat myself. Indeed -- i was merely commenting that there was actually a correct implementation -- i am still of the opinion that exposing image data was a bad thing and that a filtering API would have been superior. Oh well, the past is the past and we must now live with it. :-/ How can one use this implementation? In my tests getImageData(0,0,w,h) simply returns a wxh ImageData object. It would be interesting to enable this non-1:1 backing store to see if sites break or not. -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] createImageData should take unsigned long
On Mon, 31 Aug 2009, Philip J�genstedt wrote: On Mon, 31 Aug 2009 08:08:05 +0200, Ian Hickson i...@hixie.ch wrote: On Mon, 24 Aug 2009, Philip J�genstedt wrote: As far as I can see there's no good reason why createImageData should take a float as input rather than unsigned long. Having it as float creates the odd situation where (0.1, 0.1) gives a 1x1 ImageData while (10.1, 10.1) gives a 10x10 or 11x11 depening on if you ceil or round the input (not defined). Unless there's a compelling reason to allow something like (0.1, 0.1) I suggest changing the type and leaving the float-unsigned conversion to WebIDL. Twenty years from now, when we're using 960dpi screens, 1 CSS pixel might well map to ten device pixels reliably, such that people will want sub-CSS-pixel-level accuracy in their calls to createImageData(). I get the impression this has all been discussed before. Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( I expect we'll introduce a new API that actually works once there is a browser vendor actually interested in supporting higher-DPI canvases. In any event, judging by existing implementations, the behavior of createImageData(w, h) isn't as clear as it needs to be: http://software.hixie.ch/utilities/js/live-dom-viewer/saved/223 Firefox: log: ctx.createImageData(-1.1,1) = [Exception... log: ctx.createImageData(-1,1) = [Exception... log: ctx.createImageData(-0.1,1) = [Exception... log: ctx.createImageData(0,1) = [Exception... log: ctx.createImageData(0.1,1) = [Exception... log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 Safari/Chrome: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = 1x1 log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 2x1 My interpretation of the spec: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = INDEX_SIZE_ERR log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 The spec doesn't say what size the ImageData objects should be in these cases. Your interpretation is correct insofar as the exception is concerned, though. If the spec doesn't say to round rather than ceil, we're bound to have subtle compatibility bugs on this. The spec says it doesn't matter so long as you're consistent. On Mon, 31 Aug 2009, Robert O'Callahan wrote: Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). I agree, but Ian doesn't. It's not so much that I disagree so much as that there is no point fixing this now, since whatever new API we introduce today will just end up broken in the exact some way as the existing API. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] createImageData should take unsigned long
On Thu, 03 Sep 2009 13:54:03 +0200, Ian Hickson i...@hixie.ch wrote: On Mon, 31 Aug 2009, Philip J�genstedt wrote: On Mon, 31 Aug 2009 08:08:05 +0200, Ian Hickson i...@hixie.ch wrote: On Mon, 24 Aug 2009, Philip J�genstedt wrote: As far as I can see there's no good reason why createImageData should take a float as input rather than unsigned long. Having it as float creates the odd situation where (0.1, 0.1) gives a 1x1 ImageData while (10.1, 10.1) gives a 10x10 or 11x11 depening on if you ceil or round the input (not defined). Unless there's a compelling reason to allow something like (0.1, 0.1) I suggest changing the type and leaving the float-unsigned conversion to WebIDL. Twenty years from now, when we're using 960dpi screens, 1 CSS pixel might well map to ten device pixels reliably, such that people will want sub-CSS-pixel-level accuracy in their calls to createImageData(). I get the impression this has all been discussed before. Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( I expect we'll introduce a new API that actually works once there is a browser vendor actually interested in supporting higher-DPI canvases. I wasn't involved then, but I can only presume that there was no perceived benefit of high-DPI ImageData since you can get high-quality rendering just as well with techniques that don't rely on the canvas being higher resolution than the display device. In any event, judging by existing implementations, the behavior of createImageData(w, h) isn't as clear as it needs to be: http://software.hixie.ch/utilities/js/live-dom-viewer/saved/223 Firefox: log: ctx.createImageData(-1.1,1) = [Exception... log: ctx.createImageData(-1,1) = [Exception... log: ctx.createImageData(-0.1,1) = [Exception... log: ctx.createImageData(0,1) = [Exception... log: ctx.createImageData(0.1,1) = [Exception... log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 Safari/Chrome: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = 1x1 log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 2x1 My interpretation of the spec: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = INDEX_SIZE_ERR log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 The spec doesn't say what size the ImageData objects should be in these cases. Your interpretation is correct insofar as the exception is concerned, though. If the spec doesn't say to round rather than ceil, we're bound to have subtle compatibility bugs on this. The spec says it doesn't matter so long as you're consistent. On Mon, 31 Aug 2009, Robert O'Callahan wrote: Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). I agree, but Ian doesn't. It's not so much that I disagree so much as that there is no point fixing this now, since whatever new API we introduce today will just end up broken in the exact some way as the existing API. The fix is not to introduce a new API that can handle high-DPI ImageData, but rather to make the spec reflect the reality that high-DPI ImageData implementations won't be possible with this version of the API. That would include, among other things, specifying that getImageData(0, 0, w, h) returns a wxh ImageData object, removing createImageData(ImageData) and making createImageData(w, h) take unsigned long and return a wxh ImageData object. In other words, aligning with what implementations already do (and will continue to do for compatibility reasons). -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] createImageData should take unsigned long
Philip Jägenstedt wrote: I wasn't involved then, but I can only presume that there was no perceived benefit of high-DPI ImageData since you can get high-quality rendering just as well with techniques that don't rely on the canvas being higher resolution than the display device. To be clear, in this context high-DPI means using the DPI of the display device, instead of the about 96dpi you get with CSS pixels. So if this were done and I have a 300-dpi screen, there would be, say, 3 imagedata pixels per CSS pixel. If you have a 200-dpi screen, then on our computer there would be 2 imagedata pixels per CSS pixel. Script would have to be able to deal with this, which is where the APIs like createImageData(imageData) come in. -Boris
Re: [whatwg] createImageData should take unsigned long
On Sep 3, 2009, at 4:54 AM, Ian Hickson wrote: Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( WebKit's implementation has always worked with high dpi backing stores and follows the spec accordingly. --Oliver
Re: [whatwg] createImageData should take unsigned long
On Fri, Sep 4, 2009 at 4:48 AM, Oliver Hunt oli...@apple.com wrote: On Sep 3, 2009, at 4:54 AM, Ian Hickson wrote: Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( WebKit's implementation has always worked with high dpi backing stores and follows the spec accordingly. Under what circumstances do you use more than one device pixel per CSS pixel? Does it require the user to turn on UI scaling on Mac? Regardless, I bet that most people using Webkit to write scripts using getImageData still get it wrong, because they have normal screens. Implementing high-res backing store in more browsers won't solve this problem, not until the average developer has a high-dpi screen. But I repeat myself. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] createImageData should take unsigned long
On Sep 3, 2009, at 4:50 PM, Robert O'Callahan wrote: On Fri, Sep 4, 2009 at 4:48 AM, Oliver Hunt oli...@apple.com wrote: On Sep 3, 2009, at 4:54 AM, Ian Hickson wrote: Yeah, that seems likely, since none of you implemented the higher-DPI ImageData in your first versions. :-( WebKit's implementation has always worked with high dpi backing stores and follows the spec accordingly. Under what circumstances do you use more than one device pixel per CSS pixel? Does it require the user to turn on UI scaling on Mac? Regardless, I bet that most people using Webkit to write scripts using getImageData still get it wrong, because they have normal screens. Implementing high-res backing store in more browsers won't solve this problem, not until the average developer has a high-dpi screen. But I repeat myself. Indeed -- i was merely commenting that there was actually a correct implementation -- i am still of the opinion that exposing image data was a bad thing and that a filtering API would have been superior. Oh well, the past is the past and we must now live with it. :-/ --Oliver
Re: [whatwg] createImageData should take unsigned long
On Mon, 24 Aug 2009, Philip Jägenstedt wrote: As far as I can see there's no good reason why createImageData should take a float as input rather than unsigned long. Having it as float creates the odd situation where (0.1, 0.1) gives a 1x1 ImageData while (10.1, 10.1) gives a 10x10 or 11x11 depening on if you ceil or round the input (not defined). Unless there's a compelling reason to allow something like (0.1, 0.1) I suggest changing the type and leaving the float-unsigned conversion to WebIDL. Twenty years from now, when we're using 960dpi screens, 1 CSS pixel might well map to ten device pixels reliably, such that people will want sub-CSS-pixel-level accuracy in their calls to createImageData(). -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] createImageData should take unsigned long
On Mon, 31 Aug 2009 08:08:05 +0200, Ian Hickson i...@hixie.ch wrote: On Mon, 24 Aug 2009, Philip Jägenstedt wrote: As far as I can see there's no good reason why createImageData should take a float as input rather than unsigned long. Having it as float creates the odd situation where (0.1, 0.1) gives a 1x1 ImageData while (10.1, 10.1) gives a 10x10 or 11x11 depening on if you ceil or round the input (not defined). Unless there's a compelling reason to allow something like (0.1, 0.1) I suggest changing the type and leaving the float-unsigned conversion to WebIDL. Twenty years from now, when we're using 960dpi screens, 1 CSS pixel might well map to ten device pixels reliably, such that people will want sub-CSS-pixel-level accuracy in their calls to createImageData(). I get the impression this has all been discussed before. Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). I don't doubt that high DPI screens will happen, but when it does browsers are more likely to provide an extra flag like getImageData(..., useDevicePixels) or another opt-in method in order to stay compatible with existing content. Another option for the page author is simply creating a 1000x1000 canvas and setting its CSS width/height to 100x100 (assuming the CSS pixel:device pixel ratio can be found via script). In any event, judging by existing implementations, the behavior of createImageData(w, h) isn't as clear as it needs to be: http://software.hixie.ch/utilities/js/live-dom-viewer/saved/223 Firefox: log: ctx.createImageData(-1.1,1) = [Exception... log: ctx.createImageData(-1,1) = [Exception... log: ctx.createImageData(-0.1,1) = [Exception... log: ctx.createImageData(0,1) = [Exception... log: ctx.createImageData(0.1,1) = [Exception... log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 Safari/Chrome: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = 1x1 log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 2x1 My interpretation of the spec: log: ctx.createImageData(-1.1,1) = 1x1 log: ctx.createImageData(-1,1) = 1x1 log: ctx.createImageData(-0.1,1) = 1x1 log: ctx.createImageData(0,1) = INDEX_SIZE_ERR log: ctx.createImageData(0.1,1) = 1x1 log: ctx.createImageData(1,1) = 1x1 log: ctx.createImageData(1.1,1) = 1x1 If the spec doesn't say to round rather than ceil, we're bound to have subtle compatibility bugs on this. -- Philip Jägenstedt Core Developer Opera Software
Re: [whatwg] createImageData should take unsigned long
On Mon, Aug 31, 2009 at 1:55 AM, Philip Jägenstedt phil...@opera.comwrote: I get the impression this has all been discussed before. It has. Still, it seems unlikely that any browser will ever be able to switch to anything but a 1:1 CSS pixel:device pixel ratio, as that would break all existing pages assuming that getImageData(0, 0, 100, 100) returns a 100x100 bitmap (because assuming that is much easier, unless you read the spec carefully you're unlikely to know it could ever be any different). I agree, but Ian doesn't. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] createImageData should take unsigned long
Philip Jägenstedt wrote: In any event, judging by existing implementations, the behavior of createImageData(w, h) isn't as clear as it needs to be: Firefox: Just to be clear, the Firefox code for this predates the spec text. I would assume so does Webkit's. Once we're actually trying to implement the spec, we'll probably comment if we think it makes no sense or isn't clear... ;) -Boris
Re: [whatwg] createImageData should take unsigned long
On Mon, Aug 31, 2009 at 11:06 PM, Anne van Kesteren ann...@opera.comwrote: Once we get huge screens and lots of processing power people can just blow up the canvas grid and then scale it down with CSS. Works just as well and makes the data more portable. I think we can do better than that. It's fine to use high-dpi backing store automatically for canvas in general. ImageData is the only situation where it matters if there's more than one device pixel per CSS pixel, and authors should be able to opt into taking advantage of that fact, but not be exposed to it by default, IMHO. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
[whatwg] createImageData should take unsigned long
As far as I can see there's no good reason why createImageData should take a float as input rather than unsigned long. Having it as float creates the odd situation where (0.1, 0.1) gives a 1x1 ImageData while (10.1, 10.1) gives a 10x10 or 11x11 depening on if you ceil or round the input (not defined). Unless there's a compelling reason to allow something like (0.1, 0.1) I suggest changing the type and leaving the float-unsigned conversion to WebIDL. -- Philip Jägenstedt Opera Software
Re: [whatwg] createImageData
On Jun 9, 2008, at 8:25 PM, Jonas Sicking wrote: Vladimir Vukicevic wrote: Sorry it took me a bit to respond here... so, ok, based on the discussion, I'd suggest: - user-created ImageData-like objects should be supported, e.g. with language such as: Do note that dealing with user-created objects isn't trivial. You have to be prepared for dealing with the user-created object changing the whole world under you during a callback, this includes things like doing any modification to the canvas object itself, but also things like script navigating away and then causing a GC to tear down the world around you. This is usually not very hard to deal with, as long as you are not deep inside a long callstack when calling out to content. This is because you have to ensure that the whole callstack is ok with the world changing around it. As an additional note: inn implementations that support getters and setters, any property access to a user-created object may call back into arbitrary JS code. Even in those that do not, a toNumber conversion of an arbitrary JS value may call back into arbitrary JS code. Regards, Maciej
Re: [whatwg] createImageData
Vladimir Vukicevic wrote: Sorry it took me a bit to respond here... so, ok, based on the discussion, I'd suggest: - user-created ImageData-like objects should be supported, e.g. with language such as: Do note that dealing with user-created objects isn't trivial. You have to be prepared for dealing with the user-created object changing the whole world under you during a callback, this includes things like doing any modification to the canvas object itself, but also things like script navigating away and then causing a GC to tear down the world around you. This is usually not very hard to deal with, as long as you are not deep inside a long callstack when calling out to content. This is because you have to ensure that the whole callstack is ok with the world changing around it. / Jonas
Re: [whatwg] createImageData
Sorry it took me a bit to respond here... so, ok, based on the discussion, I'd suggest: - user-created ImageData-like objects should be supported, e.g. with language such as: The first argument to the method must be an ImageData object returned by createImageData(), getImageData(), or an object constructed with the necessary properties by the user. If the object was constructed by the user, its width and height dimensions are specified in device pixels (which may not map directly to CSS pixels). If null or any other object is given that does not present the ImageData interface, then the putImageData() method must raise a TYPE_MISMATCH_ERR exception. - ImageData objects returned by createImageData or getImageData should behave as currently specified; that is, they should explicitly clamp on pixel assignment. That gives users a choice over which approach they want to take, and whether they want clamping or not. How's that sound? - Vlad
Re: [whatwg] createImageData
On May 10, 2008, at 4:53 PM, Vladimir Vukicevic wrote: I would amend the spec to state that if an object is passed to putImageData with the necessary properties, but without having been created by create/getImageData beforehand, that its dimensions are aways in device pixels. Some suggested language in section 3.12.11.1.11(!): Instead of: If the first argment to the method is null or not an ImageData object that was returned by createImageData() or getImageData() then the putImageData() method must raise a TYPE_MISMATCH_ERR exception. I would suggest: The first argument to the method must be an ImageData object returned by createImageData(), getImageData(), or an object constructed with the necessary properties by the user. If the object was constructed by the user, its width and height dimensions are specified in device pixels (which may not map directly to CSS pixels). If null or any other object is given that does not present the ImageData interface, then the putImageData() method must raise a TYPE_MISMATCH_ERR exception. - Vlad
Re: [whatwg] createImageData
On May 13, 2008, at 1:53 PM, Vladimir Vukicevic wrote: The first argument to the method must be an ImageData object returned by createImageData(), getImageData(), or an object constructed with the necessary properties by the user. If the object was constructed by the user, its width and height dimensions are specified in device pixels (which may not map directly to CSS pixels). If null or any other object is given that does not present the ImageData interface, then the putImageData() method must raise a TYPE_MISMATCH_ERR exception. If we were to add that we should include a note to indicate that using a custom object is not recommended -- Any code that uses a custom created object will never benefit from improvements in ImageData performance made by the UA. That said I still don't believe custom objects should be allowed, aside from the resolution (which may or may not be relevant) and performance issues, a custom object with a generic JS array, rather than an ImageData object will have different behaviour -- a proper ImageData will clamp on assignment, and throw in cases that a custom object won't. - Vlad --Oliver
Re: [whatwg] createImageData
On May 13, 2008, at 2:58 PM, Oliver Hunt wrote: On May 13, 2008, at 1:53 PM, Vladimir Vukicevic wrote: The first argument to the method must be an ImageData object returned by createImageData(), getImageData(), or an object constructed with the necessary properties by the user. If the object was constructed by the user, its width and height dimensions are specified in device pixels (which may not map directly to CSS pixels). If null or any other object is given that does not present the ImageData interface, then the putImageData() method must raise a TYPE_MISMATCH_ERR exception. If we were to add that we should include a note to indicate that using a custom object is not recommended -- Any code that uses a custom created object will never benefit from improvements in ImageData performance made by the UA. I'm fine with adding that language (the first part, anyway); something like Using a custom object is not recommended as the UA may be able to optimize operations using ImageData if they were created via createImageData() or getImageData(). That said I still don't believe custom objects should be allowed, aside from the resolution (which may or may not be relevant) and performance issues, a custom object with a generic JS array, rather than an ImageData object will have different behaviour -- a proper ImageData will clamp on assignment, and throw in cases that a custom object won't. That verification seems odd; doing those checks (clamping, conversion to number) on every single pixel assignment is going the wrong direction for performance -- you really want to validate everything at once. - Vlad
Re: [whatwg] createImageData
That said I still don't believe custom objects should be allowed, aside from the resolution (which may or may not be relevant) and performance issues, a custom object with a generic JS array, rather than an ImageData object will have different behaviour -- a proper ImageData will clamp on assignment, and throw in cases that a custom object won't. That verification seems odd; doing those checks (clamping, conversion to number) on every single pixel assignment is going the wrong direction for performance -- you really want to validate everything at once. But by delaying clamping, etc you are requiring that the backing store be an array of boxed values, leading to increased memory usage, increased indirection, and increasing the cost of the final blit. My experience implementing this in WebKit showed a pure byte array backing store was significantly faster than using boxed values. - Vlad --Oliver
Re: [whatwg] createImageData
On May 13, 2008, at 3:37 PM, Oliver Hunt wrote: That said I still don't believe custom objects should be allowed, aside from the resolution (which may or may not be relevant) and performance issues, a custom object with a generic JS array, rather than an ImageData object will have different behaviour -- a proper ImageData will clamp on assignment, and throw in cases that a custom object won't. That verification seems odd; doing those checks (clamping, conversion to number) on every single pixel assignment is going the wrong direction for performance -- you really want to validate everything at once. But by delaying clamping, etc you are requiring that the backing store be an array of boxed values, leading to increased memory usage, increased indirection, and increasing the cost of the final blit. That's an implementation detail, I guess.. My experience implementing this in WebKit showed a pure byte array backing store was significantly faster than using boxed values. Faster for which operation, though? The put, or the actual manipulation? It's a tradeoff, really; if you're doing limited pixel manipulation, but lots of putImageData, you can optimize that directly by just calling putImageData once to an offscreen canvas and then blitting that with drawImage. If you're doing lots of pixel manipulation but only one putImageData, I guess you can use a JS array for your intermediate ops to avoid the checking overhead, and set the image data pixels all at once (though again paying the checking penalty per pixel), but having cheap putImageData. Throwing the error at putImageData time lets the implementation optimize in whatever way is most convenient/performant (either at pixel operation time by setting an error bit in the ImageData object which is checked by putImageData, or at putImageData time), and is (IMO) more flexible.. given that errors are an exceptional case, I don't think the spec should force the checking per pixel. - Vlad
Re: [whatwg] createImageData
My experience implementing this in WebKit showed a pure byte array backing store was significantly faster than using boxed values. Faster for which operation, though? The put, or the actual manipulation? It's a tradeoff, really; if you're doing limited pixel manipulation, but lots of putImageData, you can optimize that directly by just calling putImageData once to an offscreen canvas and then blitting that with drawImage. If you're doing lots of pixel manipulation but only one putImageData, I guess you can use a JS array for your intermediate ops to avoid the checking overhead, and set the image data pixels all at once (though again paying the checking penalty per pixel), but having cheap putImageData. Throwing the error at putImageData time lets the implementation optimize in whatever way is most convenient/performant (either at pixel operation time by setting an error bit in the ImageData object which is checked by putImageData, or at putImageData time), and is (IMO) more flexible.. given that errors are an exceptional case, I don't think the spec should force the checking per pixel. I found it faster in general across quite a few tests. I would argue that if you are using ImageData in a way that leads to you writing to the same pixel multiple times you should improve your algorithms (this is just the generic over painting issue). A very reall issue to consider though is the case where I've been very careful to only update those pixels that need to be updated. If the ImageData is not clamped, etc on put then *every* blit must do a complete revalidation of the entire ImageData data buffer. I think we need a list of use cases for ImageData, off the top of my head i can think of: * filters -- in general a single write per pixel, potentially multiple reads * Generated images -- still arguably single write per pixel * I'm not sure what to call this -- but things like http://jsmsxdemo.googlepages.com/jsmsx.html I honestly can't think of something that would (sanely) expect to be writing multiple times to the same pixel between blits, but i should note i haven't actively spent any significant time trying to come up with these. That said in all of the above cases the cost of immediate clamping is technically the same as delaying the clamp, although it also has the benefit of allowing reduced memory usage. - Vlad --Oliver
Re: [whatwg] createImageData
On May 13, 2008, at 4:10 PM, Oliver Hunt wrote: My experience implementing this in WebKit showed a pure byte array backing store was significantly faster than using boxed values. Faster for which operation, though? The put, or the actual manipulation? It's a tradeoff, really; if you're doing limited pixel manipulation, but lots of putImageData, you can optimize that directly by just calling putImageData once to an offscreen canvas and then blitting that with drawImage. If you're doing lots of pixel manipulation but only one putImageData, I guess you can use a JS array for your intermediate ops to avoid the checking overhead, and set the image data pixels all at once (though again paying the checking penalty per pixel), but having cheap putImageData. Throwing the error at putImageData time lets the implementation optimize in whatever way is most convenient/performant (either at pixel operation time by setting an error bit in the ImageData object which is checked by putImageData, or at putImageData time), and is (IMO) more flexible.. given that errors are an exceptional case, I don't think the spec should force the checking per pixel. I found it faster in general across quite a few tests. I would argue that if you are using ImageData in a way that leads to you writing to the same pixel multiple times you should improve your algorithms (this is just the generic over painting issue). I dunno, some kind of iterative algorithm that you want to visualize at random timesteps. You could keep the output in a separate array and copy over when you want to render it. A very reall issue to consider though is the case where I've been very careful to only update those pixels that need to be updated. If the ImageData is not clamped, etc on put then *every* blit must do a complete revalidation of the entire ImageData data buffer. Yep, that's true. I think we need a list of use cases for ImageData, off the top of my head i can think of: * filters -- in general a single write per pixel, potentially multiple reads * Generated images -- still arguably single write per pixel * I'm not sure what to call this -- but things like http://jsmsxdemo.googlepages.com/jsmsx.html I honestly can't think of something that would (sanely) expect to be writing multiple times to the same pixel between blits, but i should note i haven't actively spent any significant time trying to come up with these. That said in all of the above cases the cost of immediate clamping is technically the same as delaying the clamp, although it also has the benefit of allowing reduced memory usage. Yeah, those are all good use cases -- it just seems like requiring immediate clamping is basically specifying for a specific implementation, when the overall goal is checking for invalid data. Specifying that the error should come from putImageData would give implementations more flexibility, without limiting error checking. (You could argue that it's easier to get a precise error location by checking on pixel assignment, but I don't think that the potential cost and loss of flexibility is worth it. Once authors know that they have an error in their data, they can take other action to track it down.) - Vlad
Re: [whatwg] createImageData
On May 13, 2008, at 4:28 PM, Vladimir Vukicevic wrote: On May 13, 2008, at 4:10 PM, Oliver Hunt wrote: My experience implementing this in WebKit showed a pure byte array backing store was significantly faster than using boxed values. Faster for which operation, though? The put, or the actual manipulation? It's a tradeoff, really; if you're doing limited pixel manipulation, but lots of putImageData, you can optimize that directly by just calling putImageData once to an offscreen canvas and then blitting that with drawImage. If you're doing lots of pixel manipulation but only one putImageData, I guess you can use a JS array for your intermediate ops to avoid the checking overhead, and set the image data pixels all at once (though again paying the checking penalty per pixel), but having cheap putImageData. Throwing the error at putImageData time lets the implementation optimize in whatever way is most convenient/performant (either at pixel operation time by setting an error bit in the ImageData object which is checked by putImageData, or at putImageData time), and is (IMO) more flexible.. given that errors are an exceptional case, I don't think the spec should force the checking per pixel. I found it faster in general across quite a few tests. I would argue that if you are using ImageData in a way that leads to you writing to the same pixel multiple times you should improve your algorithms (this is just the generic over painting issue). I dunno, some kind of iterative algorithm that you want to visualize at random timesteps. You could keep the output in a separate array and copy over when you want to render it. I'm not sure what you mean. By my interpretation that would require more work when you validate on blit, because while you would not have to validate when you copy the invalidated region of your buffer into the ImageData buffer it would not need to be validated, but when you then go to blit the ImageData, putImageData *must* revalidate the entirety of the ImageData buffer (hmmm, i suppose the UA could try tracking dirty regions in the ImageData buffer to minimise revalidation? i suspect this would not be significantly better than just validating on every pixel put) If you actually meant you were using the ImageData buffer as your working buffer then you would possibly be doing excessive revalidation, but in my experience such a case would be atypical (and we're more interested in the performance of the normal case vs. edge cases) and the cost of clamping, etc is dwarfed by the dispatch cost just to do the assignment (at least in WebKit) so i'm not sure there would a significant loss in performance anyway. A very reall issue to consider though is the case where I've been very careful to only update those pixels that need to be updated. If the ImageData is not clamped, etc on put then *every* blit must do a complete revalidation of the entire ImageData data buffer. Yep, that's true. I think we need a list of use cases for ImageData, off the top of my head i can think of: * filters -- in general a single write per pixel, potentially multiple reads * Generated images -- still arguably single write per pixel * I'm not sure what to call this -- but things like http://jsmsxdemo.googlepages.com/jsmsx.html I honestly can't think of something that would (sanely) expect to be writing multiple times to the same pixel between blits, but i should note i haven't actively spent any significant time trying to come up with these. That said in all of the above cases the cost of immediate clamping is technically the same as delaying the clamp, although it also has the benefit of allowing reduced memory usage. Yeah, those are all good use cases -- it just seems like requiring immediate clamping is basically specifying for a specific implementation, when the overall goal is checking for invalid data. Specifying that the error should come from putImageData would give implementations more flexibility, without limiting error checking. (You could argue that it's easier to get a precise error location by checking on pixel assignment, but I don't think that the potential cost and loss of flexibility is worth it. Once authors know that they have an error in their data, they can take other action to track it down.) There is no implementor freedom here -- you either clamp immediately, or you do not clamp immediately, if one UA does not clamp then it will have different behaviour from the other UAs leading to behavioural incompatibility (eg. one site may expect the values to be clamped immediately, and therefore not work in UAs that don't clamp, another may believe that the clamping happens later leading it to break in those UAs that clamp immediately). This is not to say there's no room for implementation variation -- an implementation *could* maintain the data as
Re: [whatwg] createImageData
On May 13, 2008, at 4:28 PM, Vladimir Vukicevic wrote: On May 13, 2008, at 4:10 PM, Oliver Hunt wrote: I think we need a list of use cases for ImageData, off the top of my head i can think of: * filters -- in general a single write per pixel, potentially multiple reads * Generated images -- still arguably single write per pixel * I'm not sure what to call this -- but things like http://jsmsxdemo.googlepages.com/jsmsx.html I honestly can't think of something that would (sanely) expect to be writing multiple times to the same pixel between blits, but i should note i haven't actively spent any significant time trying to come up with these. That said in all of the above cases the cost of immediate clamping is technically the same as delaying the clamp, although it also has the benefit of allowing reduced memory usage. Yeah, those are all good use cases -- it just seems like requiring immediate clamping is basically specifying for a specific implementation, when the overall goal is checking for invalid data. Specifying that the error should come from putImageData would give implementations more flexibility, without limiting error checking. Wouldn't delaying clamping and error checking make it hard to use a native pixel data array as the ImageData backing store? In fact it would make it pretty much impossible unless you at least clamp/round/ throw when reading the data from JS. And even then, such an ImageData implementation would have to taint itself to record whether an error had occurred. Regards, Maciej
Re: [whatwg] createImageData
On May 9, 2008, at 5:53 PM, Ian Hickson wrote:
On Fri, 9 May 2008, Vladimir Vukicevic wrote:
I don't think the restriction that putImageData must only work with
objects returned by create/get is a good one
This restriction was made because it allows for dramatic (many
orders of
magnitude) optimisations. With createImageData(), the use cases for
custom
imageData objects should be catered for -- what are the cases where
you
would need another solution? (Note that hand-rolling imageData
objects is
dangerous since you don't know what resolution the backing store is
using,
necessarily, which is something else that createImageData() solves.)
Well, I don't agree that it's dangerous; canvas resolution
independence has always been hard to pin down, and I still maintain
that it shouldn't be treated any differently than an image is
treated. Canvas isn't supposed to replace SVG. However, regardless
of that, I don't think there's a reason to disallow custom-created
data objects, perhaps with a caveat that there may be issues.. get/
putImageData in Firefox 2, so adding that restriction may
unnecessarily break existing code that uses putImageData with a hand-
constructed ImageData object. I would amend the spec to state that if
an object is passed to putImageData with the necessary properties, but
without having been created by create/getImageData beforehand, that
its dimensions are aways in device pixels.
One problem with the desired goal of resolution independence is that
it only really makes sense if the target resolution is an integer
multiple of a CSS pixel. For example, with a 144dpi output device,
that's exactly 1.5 times CSS resolution. If I call createImageData
with dimensions 10,10, I would get an ImageData object with width 15
and height 15. What do I get if I call it with 11,11 though? That's
16.5 device pixels, and you're going to lose data either way you go,
because at putImageData time you're going to get seams no matter what
direction you round. This can maybe be solved with language in the
spec that specifies that a canvas should use a different ratio of CSS
pixels to device pixels only if one is an integer multiple of the
other, but that seems like an odd limitation (and it still requires
the implementation to decide what to do if a clean ratio isn't
possible).
Another approach would be to not try to solve this in canvas at all,
and instead specify that by default, all canvas elements are 96dpi,
and provide authors a way to explicitly override this -- then using a
combination of CSS Media Queries and other CSS, the exact dpi desired
could be specified. (You can sort of do this today, given that the
canvas width/height attributes are in CSS pixels, and that if CSS
dimensions are present a canvas is scaled like an image... so canvas
{ width: 100px; height: 100px; } ... canvas width=200 height=200/
would give a 192dpi canvas today, no?)
but it would be good to have some way to mark sections of the spec as
stable/unstable --
I've gone through and added annotations for each of the canvas
sections to
distinguish the stable parts from the unstable parts. Does that work?
otherwise someone's liable to take a snapshot and implement it, and
then
have it change under them if a portion is still in flux.
In general, the spec is unlikely to change significantly _before_ the
first implemenation. We get more feedback from the first
implementation of
anything than from people just looking at the spec. I agree that the
first
implementation should know what it's getting itself into, though. :-)
Well, it depends what you mean by spec -- I think that what gets put
down as the initial spec is likely to change significantly from when
the feature is first proposed to where it's added to the spec; I agree
that there would be more feedback after a first implementation, but I
don't think that means that the first proposal-spec discussion/
feedback period should be skipped.
The annotations do help make it clear what's in what state though,
thanks!
- Vlad
Re: [whatwg] createImageData
On May 10, 2008, at 4:53 PM, Vladimir Vukicevic wrote:
Another approach would be to not try to solve this in canvas at all,
and instead specify that by default, all canvas elements are 96dpi,
and provide authors a way to explicitly override this -- then using
a combination of CSS Media Queries and other CSS, the exact dpi
desired could be specified. (You can sort of do this today, given
that the canvas width/height attributes are in CSS pixels, and that
if CSS dimensions are present a canvas is scaled like an image... so
canvas { width: 100px; height: 100px; } ... canvas width=200
height=200/ would give a 192dpi canvas today, no?)
Canvas was designed with the intent of allowing resolution
independent, removing that intent in the name of a feature that is not
used in the general case seems to be a fairly substantial step back
from that goal. Unfortunately the solution of using a larger canvas
scaled to fit a smaller region isn't a real solution. For lower
resolution displays it results in higher memory usage and greater
computational cost than is otherwise necessary, and for high dpi
displays it results either the same issues as the low dpi case (if the
canvas resolution is still too high) or it results in a lower
resolution display than the display is capable of.
Moreover using a larger canvas than necessary will result in much
higher memory usage for engines like WebKit which do support high DPI
displays, and so will end up with a scaled high resolution canvas.
--Oliver
Re: [whatwg] createImageData
On May 10, 2008, at 5:44 PM, Oliver Hunt wrote:
On May 10, 2008, at 4:53 PM, Vladimir Vukicevic wrote:
Another approach would be to not try to solve this in canvas at
all, and instead specify that by default, all canvas elements are
96dpi, and provide authors a way to explicitly override this --
then using a combination of CSS Media Queries and other CSS, the
exact dpi desired could be specified. (You can sort of do this
today, given that the canvas width/height attributes are in CSS
pixels, and that if CSS dimensions are present a canvas is scaled
like an image... so canvas { width: 100px; height: 100px; } ...
canvas width=200 height=200/ would give a 192dpi canvas
today, no?)
Canvas was designed with the intent of allowing resolution
independent, removing that intent in the name of a feature that is
not used in the general case seems to be a fairly substantial step
back from that goal. Unfortunately the solution of using a larger
canvas scaled to fit a smaller region isn't a real solution. For
lower resolution displays it results in higher memory usage and
greater computational cost than is otherwise necessary, and for high
dpi displays it results either the same issues as the low dpi case
(if the canvas resolution is still too high) or it results in a
lower resolution display than the display is capable of.
Eh? The resolution used should be whatever the appropriate resolution
should be; I'm certainly not suggesting that everyone unilaterally
create canvases with 2x pixel resolution, I'm saying that the features
exist to allow authors to (dynamically) create a canvas at whatever
the appropriate resolution is relative to CSS resolution. Canvas was
designed to allow for programmatic 2D rendering for web content;
resolution independence would certainly be nice, but it was never a
goal of the canvas spec. In fact, the spec explicitly states that the
canvas element represents a resolution-dependent bitmap canvas.
- Vlad
Re: [whatwg] createImageData - new ImageData() ?
On Sun, 10 Feb 2008 23:25:51 -, Ian Hickson [EMAIL PROTECTED] wrote: That would mean that passing ImageData around between two canvas elements doesn't always work as expected. I think that's highly undesirable. Is there any implementation where we know this will the case? Not today, but why preclude it? Implementations could get higher quality renderings on canvases that get resized dynamically, by using a bigger backing store. What's wrong with what we have now? It's very easy to write code which assumes that size of imageData is the same as size given in canvas width height (I know, because I did it :) Since the ratio is system/browser-dependent and 200dpi screens aren't popular yet, such bug may be easily overlooked/ignored and get widely deployed. The difference in sizes isn't intuitive. For example if browser doubled number of pixels in ImageData only when user used zoom, I think authors would rather think that browser's zoom is buggy. I think that by default getImageData should return data with same size as specified in width/height attributes of canvas. There might be another method (getImageScreenData?) or method argument that returns all pixels of canvas. ImageData can be made portable between canvases by adding aspect ratio field or additional width/height fields given in CSS pixels. -- regards, Kornel Lesiński
Re: [whatwg] createImageData - new ImageData() ?
On Sun, 3 Feb 2008, Anne van Kesteren wrote: Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) The relationship between the height and width arguments and the height and width of the ImageData structure returned can vary on a per-canvas basis. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] createImageData - new ImageData() ?
On Sun, 10 Feb 2008 10:21:34 +0100, Ian Hickson [EMAIL PROTECTED] wrote: On Sun, 3 Feb 2008, Anne van Kesteren wrote: Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) The relationship between the height and width arguments and the height and width of the ImageData structure returned can vary on a per-canvas basis. That would mean that passing ImageData around between two canvas elements doesn't always work as expected. I think that's highly undesirable. Is there any implementation where we know this will the case? -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
Re: [whatwg] createImageData - new ImageData() ?
On Feb 10, 2008 11:14 PM, Anne van Kesteren [EMAIL PROTECTED] wrote: That would mean that passing ImageData around between two canvas elements doesn't always work as expected. I think that's highly undesirable. Is there any implementation where we know this will the case? Not yet, but in Gecko when we get around to implementing variable-resolution canvas backing stores, I expect it would happen because different documents --- even different documents that can reference each other --- can be zoomed different amounts, which affects the device-pixels-per-CSS-pixel ratio we use internally. It's actually worse than that, because we respond to dynamic DPI changes (due to zooming or changes in preferences; I'm not sure if we actually support detection of hardware DPI changes, e.g. due to screen mode switching, but we easily could). So we might want to dynamically rescale the canvas backing store. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [whatwg] createImageData - new ImageData() ?
On Sun, 10 Feb 2008, Anne van Kesteren wrote: On Sun, 10 Feb 2008 10:21:34 +0100, Ian Hickson [EMAIL PROTECTED] wrote: On Sun, 3 Feb 2008, Anne van Kesteren wrote: Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) The relationship between the height and width arguments and the height and width of the ImageData structure returned can vary on a per-canvas basis. That would mean that passing ImageData around between two canvas elements doesn't always work as expected. I think that's highly undesirable. Is there any implementation where we know this will the case? Not today, but why preclude it? Implementations could get higher quality renderings on canvases that get resized dynamically, by using a bigger backing store. What's wrong with what we have now? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
[whatwg] createImageData - new ImageData() ?
Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
Re: [whatwg] createImageData - new ImageData() ?
On Feb 3, 2008, at 2:38 AM, Anne van Kesteren wrote: Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) The biggest problem is that you would have to define the behaviour in a fairly counterintuitive way new ImageData(400, 400) would not necessarily result in an ImageData object with width=400, height=400 as a system with a non-1:1 css to device pixel ratio. The moment people start using getImageData() as storage mechanism (which they will if browsers do not fix their data: image origin stuff) you'll have a problem with this if the application is accessed through different devices that have different ratios. I suspect though that given that Firefox has a 1:1 ratio sites will simply rely on that... Part of my commentary was on a misunderstanding of the spec sorry :( That said, what do you mean by using getImageData as a storage mechanism? --Oliver -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
Re: [whatwg] createImageData - new ImageData() ?
On Sun, 03 Feb 2008 12:20:54 +0100, Oliver Hunt [EMAIL PROTECTED] wrote: On Feb 3, 2008, at 2:38 AM, Anne van Kesteren wrote: Why not give the object a constructor? I think that's cleaner. Also, Opera already supports that :-) The biggest problem is that you would have to define the behaviour in a fairly counterintuitive way new ImageData(400, 400) would not necessarily result in an ImageData object with width=400, height=400 as a system with a non-1:1 css to device pixel ratio. How is this a non-issue for createImageData()? The moment people start using getImageData() as storage mechanism (which they will if browsers do not fix their data: image origin stuff) you'll have a problem with this if the application is accessed through different devices that have different ratios. I suspect though that given that Firefox has a 1:1 ratio sites will simply rely on that... Part of my commentary was on a misunderstanding of the spec sorry :( That said, what do you mean by using getImageData as a storage mechanism? If you have some kind of painting application and to store the result of the painting so that the user can edit it in another session every now and then you do getImageData() on the whole canvas and export the data to some database through XMLHttpRequest so when the visitor comes back he doesn't have to start all over again. -- Anne van Kesteren http://annevankesteren.nl/ http://www.opera.com/
