Re: [whatwg] WebGL and ImageBitmaps
On Sun, May 18, 2014 at 8:10 PM, K. Gadd wrote: > The point I was trying to make there is that for many format > conversions or encoding conversions (RGB<->YUV, RGB<->HSL), not all > input values are degraded equally. The amount of error introduced > depends on the inputs. There are going to be some values for which the > conversion is more or less accurate - for example, in most cases I > would expect black and white to convert without any error. As a > result, you can't just pick a few random colors and fill a canvas with > them and decide based on that whether or not error is being > introduced. At a minimum, you should use a couple test pattern bitmaps > and do a comparison of the result. Keep in mind that all the > discussions of profile conversion so far have been about bitmaps, not > synthesized solid colors. > > I am, of course, not an expert - but I have observed this with > repeated RGB<->HSL conversions in the past (testing poor > implementations that introduced accumulated error against relatively > good implementations that did not accumulate very much error over > time.) > > http://en.wikipedia.org/wiki/SRGB#Specification_of_the_transformation > > Note that as described there, clipping and rounding may occur and > linear <-> gamma-corrected conversions may also occur. But that was not was Justin said. He said they didn't know what profile was used so sometimes things are not color managed correctly. It was also possible to infer what machine you were running based on the detected profile. Color managment is not equal to the simple color transformations you descirbe. > We also can't > know what color profile configuration your machine happens to be using > when you run these tests, and what browser you're using. Both of those > are important when saying that you can/can't reproduce the issue. I think you have things backwards. YOU raised an issue with color management and I'm trying to reproduce it but I'm failing because you didn't give us enough information. I tried: - many different colors - different profiles - repeated put/getImageData calls on the same canvas and can't reproduce. When and on what platform/browser does this problem occur? > On Sun, May 18, 2014 at 8:22 AM, Rik Cabanier wrote: > > > > > > > > On Sun, May 18, 2014 at 2:15 AM, K. Gadd wrote: > >> > >> I'd expect that the error might not accumulate for most color values. > >> Rounding would potentially kick in once you get the first loss of > >> precision. > > > > > > That doesn't make sense. If this is a shift because of color management, > it > > should happen for pretty much all values. > > I changed my profile to generate wild color shifts and tried random color > > values but don't see any changes in any browser. > > > > Could this just be happening with images that have profiles? > > > >> > >> I've only historically seen color shifts upon repeated > >> rendering in scenarios where you're losing lots of precision, or > >> losing energy (bad RGB <-> HSV conversions, for example) - you don't > >> actually need a lot of precision to fix that as long as your > >> coefficients are right. > >> On Fri, May 16, 2014 at 8:41 PM, Rik Cabanier > wrote: > >> > > >> > > >> > > >> > On Fri, May 16, 2014 at 3:06 PM, Justin Novosad > >> > wrote: > >> >> > >> >> On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier > >> >> wrote: > >> >>> > >> >>> > >> >>> Is the Web page not composited in sRGB? If so, it seems the backing > >> >>> store > >> >>> should be sRGB too. > >> >>> > >> >> > >> >> > >> >> The web page is not composited in sRGB. It is composited in the > output > >> >> device's color space, which is often sRGB or close to sRGB, but not > >> >> always. > >> >> A notable significant exception is pre Snow Leopard Macs that use a > >> >> gamma > >> >> 1.8 transfer curve. > >> >> By the way, sniffing the display color profile through getImageData > is > >> >> a > >> >> known fingerprinting technique. This factor alone can be sufficient > to > >> >> fingerprint a user who has a calibrated monitor. > >> > > >> > > >> > I'm unable to reproduce what you're describing. So, if I fill with a > >> > color > >> > and repeatedly do a getImageData/putImageData, should I see color > >> > shifts? > >> > > > > > >
Re: [whatwg] WebGL and ImageBitmaps
The point I was trying to make there is that for many format conversions or encoding conversions (RGB<->YUV, RGB<->HSL), not all input values are degraded equally. The amount of error introduced depends on the inputs. There are going to be some values for which the conversion is more or less accurate - for example, in most cases I would expect black and white to convert without any error. As a result, you can't just pick a few random colors and fill a canvas with them and decide based on that whether or not error is being introduced. At a minimum, you should use a couple test pattern bitmaps and do a comparison of the result. Keep in mind that all the discussions of profile conversion so far have been about bitmaps, not synthesized solid colors. I am, of course, not an expert - but I have observed this with repeated RGB<->HSL conversions in the past (testing poor implementations that introduced accumulated error against relatively good implementations that did not accumulate very much error over time.) http://en.wikipedia.org/wiki/SRGB#Specification_of_the_transformation Note that as described there, clipping and rounding may occur and linear <-> gamma-corrected conversions may also occur. We also can't know what color profile configuration your machine happens to be using when you run these tests, and what browser you're using. Both of those are important when saying that you can/can't reproduce the issue. On Sun, May 18, 2014 at 8:22 AM, Rik Cabanier wrote: > > > > On Sun, May 18, 2014 at 2:15 AM, K. Gadd wrote: >> >> I'd expect that the error might not accumulate for most color values. >> Rounding would potentially kick in once you get the first loss of >> precision. > > > That doesn't make sense. If this is a shift because of color management, it > should happen for pretty much all values. > I changed my profile to generate wild color shifts and tried random color > values but don't see any changes in any browser. > > Could this just be happening with images that have profiles? > >> >> I've only historically seen color shifts upon repeated >> rendering in scenarios where you're losing lots of precision, or >> losing energy (bad RGB <-> HSV conversions, for example) - you don't >> actually need a lot of precision to fix that as long as your >> coefficients are right. >> On Fri, May 16, 2014 at 8:41 PM, Rik Cabanier wrote: >> > >> > >> > >> > On Fri, May 16, 2014 at 3:06 PM, Justin Novosad >> > wrote: >> >> >> >> On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier >> >> wrote: >> >>> >> >>> >> >>> Is the Web page not composited in sRGB? If so, it seems the backing >> >>> store >> >>> should be sRGB too. >> >>> >> >> >> >> >> >> The web page is not composited in sRGB. It is composited in the output >> >> device's color space, which is often sRGB or close to sRGB, but not >> >> always. >> >> A notable significant exception is pre Snow Leopard Macs that use a >> >> gamma >> >> 1.8 transfer curve. >> >> By the way, sniffing the display color profile through getImageData is >> >> a >> >> known fingerprinting technique. This factor alone can be sufficient to >> >> fingerprint a user who has a calibrated monitor. >> > >> > >> > I'm unable to reproduce what you're describing. So, if I fill with a >> > color >> > and repeatedly do a getImageData/putImageData, should I see color >> > shifts? >> > > >
Re: [whatwg] WebGL and ImageBitmaps
On Sun, May 18, 2014 at 2:15 AM, K. Gadd wrote: > I'd expect that the error might not accumulate for most color values. > Rounding would potentially kick in once you get the first loss of > precision. That doesn't make sense. If this is a shift because of color management, it should happen for pretty much all values. I changed my profile to generate wild color shifts and tried random color values but don't see any changes in any browser. Could this just be happening with images that have profiles? > I've only historically seen color shifts upon repeated > rendering in scenarios where you're losing lots of precision, or > losing energy (bad RGB <-> HSV conversions, for example) - you don't > actually need a lot of precision to fix that as long as your > coefficients are right. > On Fri, May 16, 2014 at 8:41 PM, Rik Cabanier wrote: > > > > > > > > On Fri, May 16, 2014 at 3:06 PM, Justin Novosad > wrote: > >> > >> On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier > wrote: > >>> > >>> > >>> Is the Web page not composited in sRGB? If so, it seems the backing > store > >>> should be sRGB too. > >>> > >> > >> > >> The web page is not composited in sRGB. It is composited in the output > >> device's color space, which is often sRGB or close to sRGB, but not > always. > >> A notable significant exception is pre Snow Leopard Macs that use a > gamma > >> 1.8 transfer curve. > >> By the way, sniffing the display color profile through getImageData is a > >> known fingerprinting technique. This factor alone can be sufficient to > >> fingerprint a user who has a calibrated monitor. > > > > > > I'm unable to reproduce what you're describing. So, if I fill with a > color > > and repeatedly do a getImageData/putImageData, should I see color shifts? > > >
Re: [whatwg] WebGL and ImageBitmaps
I'd expect that the error might not accumulate for most color values. Rounding would potentially kick in once you get the first loss of precision. I've only historically seen color shifts upon repeated rendering in scenarios where you're losing lots of precision, or losing energy (bad RGB <-> HSV conversions, for example) - you don't actually need a lot of precision to fix that as long as your coefficients are right. On Fri, May 16, 2014 at 8:41 PM, Rik Cabanier wrote: > > > > On Fri, May 16, 2014 at 3:06 PM, Justin Novosad wrote: >> >> On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier wrote: >>> >>> >>> Is the Web page not composited in sRGB? If so, it seems the backing store >>> should be sRGB too. >>> >> >> >> The web page is not composited in sRGB. It is composited in the output >> device's color space, which is often sRGB or close to sRGB, but not always. >> A notable significant exception is pre Snow Leopard Macs that use a gamma >> 1.8 transfer curve. >> By the way, sniffing the display color profile through getImageData is a >> known fingerprinting technique. This factor alone can be sufficient to >> fingerprint a user who has a calibrated monitor. > > > I'm unable to reproduce what you're describing. So, if I fill with a color > and repeatedly do a getImageData/putImageData, should I see color shifts? >
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 16, 2014 at 3:06 PM, Justin Novosad wrote: > On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier wrote: > >> >> Is the Web page not composited in sRGB? If so, it seems the backing store >> should be sRGB too. >> >> > > The web page is not composited in sRGB. It is composited in the output > device's color space, which is often sRGB or close to sRGB, but not always. > A notable significant exception is pre Snow Leopard Macs that use a gamma > 1.8 transfer curve. > By the way, sniffing the display color profile through getImageData is a > known fingerprinting technique. This factor alone can be sufficient to > fingerprint a user who has a calibrated monitor. > I'm unable to reproduce what you're describing. So, if I fill with a color and repeatedly do a getImageData/putImageData, should I see color shifts?
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 16, 2014 at 5:42 PM, Rik Cabanier wrote: > > Is the Web page not composited in sRGB? If so, it seems the backing store > should be sRGB too. > > The web page is not composited in sRGB. It is composited in the output device's color space, which is often sRGB or close to sRGB, but not always. A notable significant exception is pre Snow Leopard Macs that use a gamma 1.8 transfer curve. By the way, sniffing the display color profile through getImageData is a known fingerprinting technique. This factor alone can be sufficient to fingerprint a user who has a calibrated monitor.
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 16, 2014 at 12:16 PM, Justin Novosad wrote: > > > > On Fri, May 16, 2014 at 12:27 PM, Ian Hickson wrote: > >> On Fri, 16 May 2014, Justin Novosad wrote >> > Blink/WebKit uses output-referred color space, which is bad for some >> > inter-op cases, but good for performance. Calling drawImage will produce >> > inter-operable behavior as far as visual output is concerned, but >> > getImageData will yield values that have the display color profile baked >> > in. >> >> I'm not quite sure what you mean here. If you mean that you can set >> 'fillStyle' to a colour, fillRect the canvas, and the get the data for >> that pixel and find that it's a different colour, then that's >> non-conforming. If you mean that you can take a bitmap image without >> colour-profile information and draw it on a canvas and then getImageData() >> will return different results, then again, that's non-conforming. >> >> If you mean that drawing an image with a color profile will result in >> getImageData() returning different colour pixels on different systems, >> then that's allowed, because the colour space of the canvas (and the rest >> of the Web platform, which must be the same colour space) is not defined. >> >> Yes the later is what I mean. It is allowed, and it is causing headaches > for many web developers. One possible solution would be to impose that > ImageData be in sRGB color space. Unfortunately, that would imply loss of > precision due to color space conversion rounding errors in a > getImageData/putImageData round trip. > Can you explain why that is? Presumably, the image data is converted to sRGB before you use it to composite its pixels. > But that is probably a lesser evil. I wonder if making this change would > break anything on the web... > >> >> > Some web developers have worked around this by reverse-engineering the >> > client-specific canvas to sRGB colorspace transform by running a test >> > pattern through drawImage+getImageData. It is horrible that we are >> > making devs resort to this. >> >> I'm not really sure what this work around achieves. Can you elaborate? >> > > For example, if a web app wants to apply an image processing algorithm, it > would use getImageData to retrieve the original pixel values, process the > data, and display the results using putImageData. The color space of the > image data is undefined and it affects the behavior of the image processing > algorithm. In order to standardize the behavior of the image processing > algorithm, the image data must be converted to a known color space. The > required color space transformation can not be queried but it can be > determined experimentally by taking an that is in a known color space > and contains known color values. You draw that image to a canvas using > drawImage, and read it back using getImageData. The color values returned > by getImageData and the known corresponding color values of the original > image provide a set of color-space correspondences that can be used to feed > a curve fitting algorithm in order to reverse-engineer the parameters of > the color space conversion that maps the unknown ImageData color space to > the known color space of the test image. > I agree. That is horrible! > If you just want to do everything in sRGB, then putting all your images in >> sRGB but without giving color space information (or setting the option to >> 'strip', if we add these createImageBitmap() options) would result in what >> you want, no? >> > > Only if the canvas backing store is forced to be in sRGB. > Is the Web page not composited in sRGB? If so, it seems the backing store should be sRGB too. > You'd have to manually (or on the server) convert images that were in >> other colour spaces, though. >> >> >> > Adding a colorspace option to createImageBitmap is not enough IMHO. I >> > think we need a more global color-management approach for canvas. >> >> If we need colour management, we need it for the Web as a whole, not just >> for canvas. So far, though, implementations have not been implementing the >> features that have been proposed, so...: >> >>http://www.w3.org/TR/css3-color/#dropped >> >> > I think think CSS and HTML can survive well without color management > features, as long as the color behavior is well defined, which seems to be > the case except for > ImageData is problematic because it stores data that in an undefined color > space. > > > >> -- >> Ian Hickson U+1047E)\._.,--,'``.fL >> http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. >> Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.' >> > >
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 16, 2014 at 12:27 PM, Ian Hickson wrote: > On Fri, 16 May 2014, Justin Novosad wrote > > Blink/WebKit uses output-referred color space, which is bad for some > > inter-op cases, but good for performance. Calling drawImage will produce > > inter-operable behavior as far as visual output is concerned, but > > getImageData will yield values that have the display color profile baked > > in. > > I'm not quite sure what you mean here. If you mean that you can set > 'fillStyle' to a colour, fillRect the canvas, and the get the data for > that pixel and find that it's a different colour, then that's > non-conforming. If you mean that you can take a bitmap image without > colour-profile information and draw it on a canvas and then getImageData() > will return different results, then again, that's non-conforming. > > If you mean that drawing an image with a color profile will result in > getImageData() returning different colour pixels on different systems, > then that's allowed, because the colour space of the canvas (and the rest > of the Web platform, which must be the same colour space) is not defined. > > Yes the later is what I mean. It is allowed, and it is causing headaches for many web developers. One possible solution would be to impose that ImageData be in sRGB color space. Unfortunately, that would imply loss of precision due to color space conversion rounding errors in a getImageData/putImageData round trip. But that is probably a lesser evil. I wonder if making this change would break anything on the web... > > > Some web developers have worked around this by reverse-engineering the > > client-specific canvas to sRGB colorspace transform by running a test > > pattern through drawImage+getImageData. It is horrible that we are > > making devs resort to this. > > I'm not really sure what this work around achieves. Can you elaborate? > For example, if a web app wants to apply an image processing algorithm, it would use getImageData to retrieve the original pixel values, process the data, and display the results using putImageData. The color space of the image data is undefined and it affects the behavior of the image processing algorithm. In order to standardize the behavior of the image processing algorithm, the image data must be converted to a known color space. The required color space transformation can not be queried but it can be determined experimentally by taking an that is in a known color space and contains known color values. You draw that image to a canvas using drawImage, and read it back using getImageData. The color values returned by getImageData and the known corresponding color values of the original image provide a set of color-space correspondences that can be used to feed a curve fitting algorithm in order to reverse-engineer the parameters of the color space conversion that maps the unknown ImageData color space to the known color space of the test image. > > If you just want to do everything in sRGB, then putting all your images in > sRGB but without giving color space information (or setting the option to > 'strip', if we add these createImageBitmap() options) would result in what > you want, no? > Only if the canvas backing store is forced to be in sRGB. > You'd have to manually (or on the server) convert images that were in > other colour spaces, though. > > > > Adding a colorspace option to createImageBitmap is not enough IMHO. I > > think we need a more global color-management approach for canvas. > > If we need colour management, we need it for the Web as a whole, not just > for canvas. So far, though, implementations have not been implementing the > features that have been proposed, so...: > >http://www.w3.org/TR/css3-color/#dropped > > I think think CSS and HTML can survive well without color management features, as long as the color behavior is well defined, which seems to be the case except for ImageData is problematic because it stores data that in an undefined color space. > -- > Ian Hickson U+1047E)\._.,--,'``.fL > http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. > Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.' >
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 16, 2014 at 9:27 AM, Ian Hickson wrote: > On Fri, 16 May 2014, Justin Novosad wrote: >> >> This is a longstanding issue with 2D canvas that several developers have >> complained about over the years. The color space of the canvas backing >> store is currently unspecified. > > It's defined that the colour space must be the same as used for CSS and > , but yeah the final colour space is left up to the UA: > >http://whatwg.org/html#color-spaces-and-color-correction > > >> Blink/WebKit uses output-referred color space, which is bad for some >> inter-op cases, but good for performance. Calling drawImage will produce >> inter-operable behavior as far as visual output is concerned, but >> getImageData will yield values that have the display color profile baked >> in. > > I'm not quite sure what you mean here. If you mean that you can set > 'fillStyle' to a colour, fillRect the canvas, and the get the data for > that pixel and find that it's a different colour, then that's > non-conforming. If you mean that you can take a bitmap image without > colour-profile information and draw it on a canvas and then getImageData() > will return different results, then again, that's non-conforming. > > If you mean that drawing an image with a color profile will result in > getImageData() returning different colour pixels on different systems, > then that's allowed, because the colour space of the canvas (and the rest > of the Web platform, which must be the same colour space) is not defined. > > >> Some web developers have worked around this by reverse-engineering the >> client-specific canvas to sRGB colorspace transform by running a test >> pattern through drawImage+getImageData. It is horrible that we are >> making devs resort to this. > > I'm not really sure what this work around achieves. Can you elaborate? > > If you just want to do everything in sRGB, then putting all your images in > sRGB but without giving color space information (or setting the option to > 'strip', if we add these createImageBitmap() options) would result in what > you want, no? > > You'd have to manually (or on the server) convert images that were in > other colour spaces, though. > > >> Adding a colorspace option to createImageBitmap is not enough IMHO. I >> think we need a more global color-management approach for canvas. > > If we need colour management, we need it for the Web as a whole, not just > for canvas. So far, though, implementations have not been implementing the > features that have been proposed, so...: > >http://www.w3.org/TR/css3-color/#dropped Recently there has been activity in some implementations to finally solve the color management problem. I can't speak for Safari, but have heard that as of Mac OS X 10.9, that implementation is rendering web pages into the sRGB color space per the CSS spec, and converting to the display's color profile on the way to the screen. I was told that there was a concern there would be a compatibility impact with web apps implicitly expecting to work in the display's color space, but surprisingly, everything seemed to work during this transition. In Chrome there is also work ongoing to handle multiple monitors with different display color profiles. Noel Gordon (CC'd) is driving this and the list of active issues can be seen at https://code.google.com/p/chromium/issues/list?q=owner%3Anoel%40chromium.org . It's likely the implementation will be done in stages: first rendering images correctly when dragging windows from monitor to monitor, and then rendering all web page content correctly. In sum, while it has taken a while for implementations to finally start tackling the color management problem, it's happening now. -Ken
Re: [whatwg] WebGL and ImageBitmaps
On Fri, 16 May 2014, Justin Novosad wrote: > > This is a longstanding issue with 2D canvas that several developers have > complained about over the years. The color space of the canvas backing > store is currently unspecified. It's defined that the colour space must be the same as used for CSS and , but yeah the final colour space is left up to the UA: http://whatwg.org/html#color-spaces-and-color-correction > Blink/WebKit uses output-referred color space, which is bad for some > inter-op cases, but good for performance. Calling drawImage will produce > inter-operable behavior as far as visual output is concerned, but > getImageData will yield values that have the display color profile baked > in. I'm not quite sure what you mean here. If you mean that you can set 'fillStyle' to a colour, fillRect the canvas, and the get the data for that pixel and find that it's a different colour, then that's non-conforming. If you mean that you can take a bitmap image without colour-profile information and draw it on a canvas and then getImageData() will return different results, then again, that's non-conforming. If you mean that drawing an image with a color profile will result in getImageData() returning different colour pixels on different systems, then that's allowed, because the colour space of the canvas (and the rest of the Web platform, which must be the same colour space) is not defined. > Some web developers have worked around this by reverse-engineering the > client-specific canvas to sRGB colorspace transform by running a test > pattern through drawImage+getImageData. It is horrible that we are > making devs resort to this. I'm not really sure what this work around achieves. Can you elaborate? If you just want to do everything in sRGB, then putting all your images in sRGB but without giving color space information (or setting the option to 'strip', if we add these createImageBitmap() options) would result in what you want, no? You'd have to manually (or on the server) convert images that were in other colour spaces, though. > Adding a colorspace option to createImageBitmap is not enough IMHO. I > think we need a more global color-management approach for canvas. If we need colour management, we need it for the Web as a whole, not just for canvas. So far, though, implementations have not been implementing the features that have been proposed, so...: http://www.w3.org/TR/css3-color/#dropped -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] WebGL and ImageBitmaps
On Thu, May 15, 2014 at 12:27 AM, Rik Cabanier wrote: > On Wed, May 14, 2014 at 7:45 PM, Glenn Maynard wrote: > > > Yes, color correction from sRGB to your monitor should not affect drawing > on canvas. (What if you had multiple monitors :-)) > This is a longstanding issue with 2D canvas that several developers have complained about over the years. The color space of the canvas backing store is currently unspecified. Blink/WebKit uses output-referred color space, which is bad for some inter-op cases, but good for performance. Calling drawImage will produce inter-operable behavior as far as visual output is concerned, but getImageData will yield values that have the display color profile baked in. Some web developers have worked around this by reverse-engineering the client-specific canvas to sRGB colorspace transform by running a test pattern through drawImage+getImageData. It is horrible that we are making devs resort to this. Adding a colorspace option to createImageBitmap is not enough IMHO. I think we need a more global color-management approach for canvas. I commented on https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643
Re: [whatwg] WebGL and ImageBitmaps
On Thu, May 15, 2014 at 10:59 AM, Ian Hickson wrote: > On Wed, 14 May 2014, Kenneth Russell wrote: >> > >> > On the assumption that they are, I filed some bugs to cover this: >> > >> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642 >> >createImageBitmap() options: image data orientation >> > >> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643 >> >createImageBitmap() options: color space handling >> > >> >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 >> >createImageBitmap() options: alpha channel handling >> > >> > As usual with such bugs, if there is interest amongst browser vendors >> > in implementing any of these, please comment on the bug. >> >> For the record, I've commented on the above bugs. I'd personally like to >> see them all specified and implemented so that image loading for the >> purpose of uploading to WebGL textures can be more efficient. > > I see that your comment was that the WebGL WG would like them specified. > > Does your saying that mean that Chrome wants to implement these also? Just > having a working group want them specified doesn't help get us to multiple > implementations... To clarify the Chrome team's desire to implement these I added another comment to each bug, and pointed to the relevant portion of the WebGL spec in response to the latest comment on https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 . -Ken
Re: [whatwg] WebGL and ImageBitmaps
On Wed, 14 May 2014, Kenneth Russell wrote: > > > > On the assumption that they are, I filed some bugs to cover this: > > > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642 > >createImageBitmap() options: image data orientation > > > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643 > >createImageBitmap() options: color space handling > > > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 > >createImageBitmap() options: alpha channel handling > > > > As usual with such bugs, if there is interest amongst browser vendors > > in implementing any of these, please comment on the bug. > > For the record, I've commented on the above bugs. I'd personally like to > see them all specified and implemented so that image loading for the > purpose of uploading to WebGL textures can be more efficient. I see that your comment was that the WebGL WG would like them specified. Does your saying that mean that Chrome wants to implement these also? Just having a working group want them specified doesn't help get us to multiple implementations... -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 9, 2014 at 2:58 PM, Ian Hickson wrote: > On Fri, 9 May 2014, Ian Hickson wrote: >> On Thu, 18 Jul 2013, Justin Novosad wrote: >> > >> > To help us iterate further, I've attempted to capture the essence of >> > this thread on the whatwg wiki, using the problem solving template. I >> > tried to capture the main ideas that we seem to agree on so far and I >> > started to think about how to handle special cases. >> > >> > http://wiki.whatwg.org/wiki/ImageBitmap_Options >> >> Are the "strongly desired options" in the above wiki page still the >> options we should be adding? > > On the assumption that they are, I filed some bugs to cover this: > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642 >createImageBitmap() options: image data orientation > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643 >createImageBitmap() options: color space handling > >https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 >createImageBitmap() options: alpha channel handling > > As usual with such bugs, if there is interest amongst browser vendors in > implementing any of these, please comment on the bug. For the record, I've commented on the above bugs. I'd personally like to see them all specified and implemented so that image loading for the purpose of uploading to WebGL textures can be more efficient. -Ken
Re: [whatwg] WebGL and ImageBitmaps
On Wed, May 14, 2014 at 7:45 PM, Glenn Maynard wrote: > On Wed, May 14, 2014 at 6:27 PM, Glenn Maynard wrote: > > > That's only an issue when sampling without premultiplication, right? > > > > I had to refresh my memory on this: > > > > https://zewt.org/~glenn/test-premultiplied-scaling/ > > > > The first image is using WebGL to blit unpremultiplied. The second is > > WebGL blitting premultiplied. The last is 2d canvas. (We're talking > about > > canvas here, of course, but WebGL makes it easier to test the different > > behavior.) This blits a red rectangle surrounded by transparent space on > > top of a red canvas. The black square is there so I can tell that it's > > actually drawing something. > > > > The first one gives a seam around the transparent area, as the white > > pixels (which are completely transparent in the image) are sampled into > the > > visible part. I think this is the problem we're talking about. The > second > > gives no seam, and the Canvas one gives no seam, indicating that it's a > > premultiplied blit. I don't know if that's specified, but the behavior > is > > the same in Chrome and FF. > > > > It looks right on red, but if the background is green you can still see the > post-premultiplied black being pulled in. It's really just GL_REPEAT that > you want, repeating the outer edge. > > > On Wed, May 14, 2014 at 9:21 PM, K. Gadd wrote: > > > The reason one pixel isn't sufficient is that if the minification > > ratio is below 50% (say, 33%), sampling algorithms other than > > non-mipmapped-bilinear will begin sampling more than 4 pixels (or one > > quad, in gpu shading terminology), so you now need enough transparent > > pixels around all your textures to ensure that sampling never crosses > > the boundaries into another image. > > > > I'm well aware of the issues of sampling sprite sheets; I've dealt with the > issue at length in the past. That's unrelated to my last mail, however, > which was about premultiplication (which is something I've not used as > much). > > > > I agree with this, but I'm not going to assume it's actually possible > > for a canvas implementation to work this way. I assume that color > > profile conversions are non-trivial (in fact, I'm nearly certain they > > are non-trivial), so doing the conversion every time you render a > > canvas to the compositor is probably expensive, especially if your GPU > > isn't powerful enough to do it in a shader (mobile devices, perhaps) - > > so I expect that most implementations do the conversion once at load > > time, to prepare an image for rendering. Until it became possible to > > retrieve image pixels with getImageData, this was a good, safe > > optimization. > > > > What I meant is that I think color correction simply shouldn't apply to > canvas at all. That may not be ideal, but I'm not sure of anything else > that won't cause severe interop issues. > Maybe the color correction described here is happening: https://hsivonen.fi/png-gamma/ If so, the image that's drawn on the canvas should match what the browser is showing on screen. Without an example, it's just speculation of course. > To be clear, colorspace conversion--converting from sRGB to RGB--isn't a > problem, other than probably needing to be specified more clearly and being > put behind an option somewhere, so you can avoid a lossy colorspace > conversion. The problem is color correction that takes the user's monitor > configuration into account, since the user's monitor settings shouldn't be > visible to script. I don't know enough about color correction to know if > this can be done efficiently in an interoperable way, so the data scripts > see isn't affected by the user's configuration. Yes, color correction from sRGB to your monitor should not affect drawing on canvas. (What if you had multiple monitors :-))
Re: [whatwg] WebGL and ImageBitmaps
On Tue, May 13, 2014 at 6:59 PM, K. Gadd wrote: > On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > > Can you give an explicit example where browsers are having different > > behavior when using drawImage? > > I thought I was pretty clear about this... colorspace conversion and > alpha conversion happen here depending on the user's display > configuration, the color profile of the source image, and what browser > you're using. I've observed differences between Firefox and Chrome > here, along with different behavior on OS X (presumably due to their > different implementation of color profiles). > > In this case 'different' means 'loading & drawing an image to a canvas > gives different results via getImageData'. > > On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > > Would this be solved with Greg's proposal for flags on ImageBitmap: > > > http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-June/251541.html > > I believe so. I think I was on record when he first posted that I > consider the alpha and colorspace flags he described as adequate. > FlipY is considerably less important to me, but I can see how people > might want it (honestly, reversing the order of scanlines is a very > cheap operation; you can do it in the sampling stage of your shader, > and actually *have* to in OpenGL because of their coordinate system > when you're doing render-to-texture.) > > On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > >> Very specifically here, by 'known color space' i just mean that the > >> color space of the image is exposed to the end user. I don't think we > >> can possibly pick a standard color space to always use; the options > >> are 'this machine's current color space' and 'the color space of the > >> input bitmap'. In many cases the color space of the input bitmap is > >> effectively 'no color space', and game developers feed the raw rgba to > >> the GPU. It's important to support that use case without degrading the > >> image data. > > > > > > Is that not the case today? > > It is very explicitly not the case, which is why we are discussing it. > It is not currently possible to do lossless manipulation of PNG images > in a web browser using canvas. The issues I described where you get > different results from getImageData are a part of that. > > On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > > Safari never created a temporary image and I recently updated Firefox so > it > > matches Safari. > > Both Safari, IE and Firefox will now sample outside of the drawImage > region. > > Chrome does not but they will fix that at some point. > > This is incorrect. A quick google search for 'webkit drawimage source > rectangle temporary' reveals such, in a post to this list. > > http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-December/080583.html > My statement to this effect was based on my (imperfect) memory of that > post. 'CGImage' (to me) says Safari since it's an Apple API, and the > post mentions Safari. I made a codepen that showed the issue: http://codepen.io/adobe/pen/jIzbv Firefox was not matching the behavior on mac because it created a intermediate image. I fixed that in https://bugzilla.mozilla.org/show_bug.cgi?id=987292 I agree that the code you linked to exists in WebKit but they add padding so it samples outside the source again.
Re: [whatwg] WebGL and ImageBitmaps
On Wed, May 14, 2014 at 6:27 PM, Glenn Maynard wrote: > That's only an issue when sampling without premultiplication, right? > > I had to refresh my memory on this: > > https://zewt.org/~glenn/test-premultiplied-scaling/ > > The first image is using WebGL to blit unpremultiplied. The second is > WebGL blitting premultiplied. The last is 2d canvas. (We're talking about > canvas here, of course, but WebGL makes it easier to test the different > behavior.) This blits a red rectangle surrounded by transparent space on > top of a red canvas. The black square is there so I can tell that it's > actually drawing something. > > The first one gives a seam around the transparent area, as the white > pixels (which are completely transparent in the image) are sampled into the > visible part. I think this is the problem we're talking about. The second > gives no seam, and the Canvas one gives no seam, indicating that it's a > premultiplied blit. I don't know if that's specified, but the behavior is > the same in Chrome and FF. > It looks right on red, but if the background is green you can still see the post-premultiplied black being pulled in. It's really just GL_REPEAT that you want, repeating the outer edge. On Wed, May 14, 2014 at 9:21 PM, K. Gadd wrote: > The reason one pixel isn't sufficient is that if the minification > ratio is below 50% (say, 33%), sampling algorithms other than > non-mipmapped-bilinear will begin sampling more than 4 pixels (or one > quad, in gpu shading terminology), so you now need enough transparent > pixels around all your textures to ensure that sampling never crosses > the boundaries into another image. > I'm well aware of the issues of sampling sprite sheets; I've dealt with the issue at length in the past. That's unrelated to my last mail, however, which was about premultiplication (which is something I've not used as much). > I agree with this, but I'm not going to assume it's actually possible > for a canvas implementation to work this way. I assume that color > profile conversions are non-trivial (in fact, I'm nearly certain they > are non-trivial), so doing the conversion every time you render a > canvas to the compositor is probably expensive, especially if your GPU > isn't powerful enough to do it in a shader (mobile devices, perhaps) - > so I expect that most implementations do the conversion once at load > time, to prepare an image for rendering. Until it became possible to > retrieve image pixels with getImageData, this was a good, safe > optimization. > What I meant is that I think color correction simply shouldn't apply to canvas at all. That may not be ideal, but I'm not sure of anything else that won't cause severe interop issues. To be clear, colorspace conversion--converting from sRGB to RGB--isn't a problem, other than probably needing to be specified more clearly and being put behind an option somewhere, so you can avoid a lossy colorspace conversion. The problem is color correction that takes the user's monitor configuration into account, since the user's monitor settings shouldn't be visible to script. I don't know enough about color correction to know if this can be done efficiently in an interoperable way, so the data scripts see isn't affected by the user's configuration. -- Glenn Maynard
Re: [whatwg] WebGL and ImageBitmaps
Replies inline On Wed, May 14, 2014 at 4:27 PM, Glenn Maynard wrote: > On Mon, May 12, 2014 at 3:19 AM, K. Gadd wrote: >> >> This is the traditional solution for scenarios where you are sampling >> from a filtered texture in 3d. However, it only works if you never >> scale images, which is actually not the case in many game scenarios. > > > That's only an issue when sampling without premultiplication, right? > > I had to refresh my memory on this: > > https://zewt.org/~glenn/test-premultiplied-scaling/ > > The first image is using WebGL to blit unpremultiplied. The second is WebGL > blitting premultiplied. The last is 2d canvas. (We're talking about canvas > here, of course, but WebGL makes it easier to test the different behavior.) > This blits a red rectangle surrounded by transparent space on top of a red > canvas. The black square is there so I can tell that it's actually drawing > something. > > The first one gives a seam around the transparent area, as the white pixels > (which are completely transparent in the image) are sampled into the visible > part. I think this is the problem we're talking about. The second gives no > seam, and the Canvas one gives no seam, indicating that it's a premultiplied > blit. I don't know if that's specified, but the behavior is the same in > Chrome and FF. The reason one pixel isn't sufficient is that if the minification ratio is below 50% (say, 33%), sampling algorithms other than non-mipmapped-bilinear will begin sampling more than 4 pixels (or one quad, in gpu shading terminology), so you now need enough transparent pixels around all your textures to ensure that sampling never crosses the boundaries into another image. http://fgiesen.wordpress.com/2011/07/10/a-trip-through-the-graphics-pipeline-2011-part-8/ explains the concept of quads, along with relevant issues like centroid interpolation. Anyone talking about correctness or performance in modern accelerated rendering might benefit from reading this whole series. You do make the good point that whether or not the canvas implementation is using premultiplied textures has an effect on the result of scaling and filtering (since doing scaling/filtering on nonpremultiplied rgba produces color bleeding from transparent pixels). Is that currently specified? I don't think I've seen bleeding artifacts recently, but I'm not certain whether the spec requires this explicitly. This issue is however not color bleeding - color bleeding is a math 'error' that results from not using premultiplication - but that the filtering algorithm samples pixels outside the actual 'rectangle' intended to be drawn. (This is an implicit problem with sampling based on texture coordinates and derivatives instead of pixel offsets) If you search for 'padding texture atlases' you can see some examples that show why this is a tricky problem and a single pixel of padding is not sufficient: http://wiki.polycount.com/EdgePadding There are some related problems here for image compression as well, due to the block-oriented nature of codecs like JPEG and DXTC. Luckily they aren't something the user agent has to deal with in their canvas implementation, but that's another example where a single pixel of padding isn't enough. > On Tue, May 13, 2014 at 8:59 PM, K. Gadd wrote: >> I thought I was pretty clear about this... colorspace conversion and >> alpha conversion happen here depending on the user's display >> configuration, the color profile of the source image, and what browser >> you're using. I've observed differences between Firefox and Chrome >> here, along with different behavior on OS X (presumably due to their >> different implementation of color profiles). >> >> In this case 'different' means 'loading & drawing an image to a canvas >> gives different results via getImageData'. > > > That's a description, not an explicit example. An example would be a URL > demonstrating the issue. http://joedev.net/JSIL/Numbers/ was the first game to report an issue from this, because his levels are authored as images. He ended up solving the problem by following my advice to manually strip color profile information from all his images (though this is not a panacea; a browser could decide that profile-information-less images are now officially sRGB, and then profile-convert them to the display profile) It's been long enough that I don't know if his uploaded build works anymore or whether it will demonstrate the issue. It's possible he removed his dependency on images by now. Here is what I told the developer in an email thread when he first reported the issue (and by 'reported' I mean 'sent me a very confused email saying that his game didn't work in Firefox and he had no idea why'): > The reason it's not working in Firefox right now is due to a firefox bug, > because your PNG files contain what's called a 'sRGB chunk': > https://bugzilla.mozilla.org/show_bug.cgi?id=867594 > I don't know if this bug can be fixed on Firefox's side because
Re: [whatwg] WebGL and ImageBitmaps
On Mon, May 12, 2014 at 3:19 AM, K. Gadd wrote: > On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > > I'm assuming you're referring to the case where if you try to draw a > > subpart of an image and for some reason it has to be sampled (e.g. you're > > drawing it larger than the source), the anti-aliasing is optimised for > > tiling and so you get "leakage" from the next sprite over. > > > > If so, the solution is just to separate the sprites by a pixel of > > transparent black, no? > > This is the traditional solution for scenarios where you are sampling > from a filtered texture in 3d. However, it only works if you never > scale images, which is actually not the case in many game scenarios. > That's only an issue when sampling without premultiplication, right? I had to refresh my memory on this: https://zewt.org/~glenn/test-premultiplied-scaling/ The first image is using WebGL to blit unpremultiplied. The second is WebGL blitting premultiplied. The last is 2d canvas. (We're talking about canvas here, of course, but WebGL makes it easier to test the different behavior.) This blits a red rectangle surrounded by transparent space on top of a red canvas. The black square is there so I can tell that it's actually drawing something. The first one gives a seam around the transparent area, as the white pixels (which are completely transparent in the image) are sampled into the visible part. I think this is the problem we're talking about. The second gives no seam, and the Canvas one gives no seam, indicating that it's a premultiplied blit. I don't know if that's specified, but the behavior is the same in Chrome and FF. On Tue, May 13, 2014 at 8:59 PM, K. Gadd wrote: > On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > > Can you give an explicit example where browsers are having different > > behavior when using drawImage? > > I thought I was pretty clear about this... colorspace conversion and > alpha conversion happen here depending on the user's display > configuration, the color profile of the source image, and what browser > you're using. I've observed differences between Firefox and Chrome > here, along with different behavior on OS X (presumably due to their > different implementation of color profiles). > > In this case 'different' means 'loading & drawing an image to a canvas > gives different results via getImageData'. > That's a description, not an explicit example. An example would be a URL demonstrating the issue. The effects of color profiles should never be visible to script--they should be applied when the canvas is drawn to the screen, not when the image is decoded or the canvas is manipulated. That seems hard to implement, though, if you're blitting images to a canvas that all have different color profiles. It's probably better to ignore color profiles for canvas entirely than to expose the user's monitor configuration like this... -- Glenn Maynard
Re: [whatwg] WebGL and ImageBitmaps
On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > Can you give an explicit example where browsers are having different > behavior when using drawImage? I thought I was pretty clear about this... colorspace conversion and alpha conversion happen here depending on the user's display configuration, the color profile of the source image, and what browser you're using. I've observed differences between Firefox and Chrome here, along with different behavior on OS X (presumably due to their different implementation of color profiles). In this case 'different' means 'loading & drawing an image to a canvas gives different results via getImageData'. On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > Would this be solved with Greg's proposal for flags on ImageBitmap: > http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-June/251541.html I believe so. I think I was on record when he first posted that I consider the alpha and colorspace flags he described as adequate. FlipY is considerably less important to me, but I can see how people might want it (honestly, reversing the order of scanlines is a very cheap operation; you can do it in the sampling stage of your shader, and actually *have* to in OpenGL because of their coordinate system when you're doing render-to-texture.) On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: >> Very specifically here, by 'known color space' i just mean that the >> color space of the image is exposed to the end user. I don't think we >> can possibly pick a standard color space to always use; the options >> are 'this machine's current color space' and 'the color space of the >> input bitmap'. In many cases the color space of the input bitmap is >> effectively 'no color space', and game developers feed the raw rgba to >> the GPU. It's important to support that use case without degrading the >> image data. > > > Is that not the case today? It is very explicitly not the case, which is why we are discussing it. It is not currently possible to do lossless manipulation of PNG images in a web browser using canvas. The issues I described where you get different results from getImageData are a part of that. On Mon, May 12, 2014 at 4:44 PM, Rik Cabanier wrote: > Safari never created a temporary image and I recently updated Firefox so it > matches Safari. > Both Safari, IE and Firefox will now sample outside of the drawImage region. > Chrome does not but they will fix that at some point. This is incorrect. A quick google search for 'webkit drawimage source rectangle temporary' reveals such, in a post to this list. http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-December/080583.html My statement to this effect was based on my (imperfect) memory of that post. 'CGImage' (to me) says Safari since it's an Apple API, and the post mentions Safari. -kg
Re: [whatwg] WebGL and ImageBitmaps
On Mon, May 12, 2014 at 1:19 AM, K. Gadd wrote: > Gosh, this thread is old. I'm going to try and compose a coherent > response but at this point I've forgotten a lot of the context... > > On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > > On Thu, 18 Jul 2013, K. Gadd wrote: > >> > >> Ultimately the core here is that without control over colorspace > >> conversion, any sort of deterministic image processing in HTML5 is off > >> the table, and you have to write your own image decoders, encoders, and > >> manipulation routines in JavaScript using raw typed arrays. Maybe that's > >> how it has to be, but it would be cool to at least support basic > >> variations of these use cases in Canvas since getImageData/putImageData > >> already exist and are fairly well-specified (other than this problem, > >> and some nits around source rectangles and alpha transparency). > > > > Given that the user's device could be a very low-power device, or one > with > > a very small screen, but the user might still want to be manipulating > very > > large images, it might be best to do the "master" manipulation on the > > server anyway. > > This request is not about efficient image manipulation (as you point > out, this is best done on a high power machine) - without control over > colorspace conversion any image processing is nondeterministic. There > are games and apps out there that rely on getting the exact same > pixels out of a given Image on all machines, and that's impossible > right now due to differing behaviors. You see demoscene projects > packing data into bitmaps (yuck), or games using images as the > canonical representation of user-generated content. The latter, I > think, is entirely defensible - maybe even desirable, since it lets > end users interact with the game using photoshop or mspaint. > Supporting these use cases in a cross-browser manner is impossible > right now, yet they work in the desktop versions of these games. > > On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > > On Thu, 18 Jul 2013, K. Gadd wrote: > >> Out of the features suggested previously in the thread, I would > >> immediately be able to make use of control over colorspace conversion > >> and an ability to opt into premultiplied alpha. Not getting > >> premultiplied alpha, as is the case in virtually every canvas > >> implementation I've tried, has visible negative consequences for image > >> quality and also reduces the performance of some use cases where bitmap > >> manipulation needs to happen, due to the fact that premultiplied alpha > >> is the 'preferred' form for certain types of rendering and the math > >> works out better. I think the upsides to getting premultiplication are > >> the same here as they are in WebGL: faster uploads/downloads, better > >> results, etc. > > > > Can you elaborate on exactly what this would look like in terms of the > API > > implications? What changes to the spec did you have in mind? > > I don't remember what my exact intent here was, but I'll try to > resynthesize it: > The key here is to have a clear understanding of what data you get out > of an ImageBitmap. It is *not* necessary for the end user to be able > to specify it, as long as the spec dictates that all browsers provide > the exact same format to end users. > If we pick one format and lock to it, we want a format that discards > as little source image data as possible (preferably *no* data is > discarded) - which would mean the raw source image data, without any > colorspace or alpha channel conversion applied. > Can you give an explicit example where browsers are having different behavior when using drawImage? > This allows all the procedural image manipulation cases described > above, and makes it a very fast and straightforward path for loading > images you plan to pass off to the GPU as a WebGL texture. There's a > bit more on this below... Would this be solved with Greg's proposal for flags on ImageBitmap: http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-June/251541.html > On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > > On Thu, 18 Jul 2013, K. Gadd wrote: > >> To clearly state what would make ImageBitmap useful for the use cases I > >> encounter and my end-users encounter: > >> ImageBitmap should be a canonical representation of a 2D bitmap, with a > >> known color space, known pixel format, known alpha representation > >> (premultiplied/not premultiplied), and ready for immediate rendering or > >> pixel data access. It's okay if it's immutable, and it's okay if > >> constructing one from an or a Blob takes time, as long as once I > have > >> an ImageBitmap I can use it to render and use it to extract pixel data > >> without user configuration/hardware producing unpredictable results. > > > > This seems reasonable, but it's not really detailed enough for me to turn > > it into spec. What colour space? What exactly should we be doing to the > > alpha channel? > > Very specifically here, by 'known c
Re: [whatwg] WebGL and ImageBitmaps
Gosh, this thread is old. I'm going to try and compose a coherent response but at this point I've forgotten a lot of the context... On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > On Thu, 18 Jul 2013, K. Gadd wrote: >> >> Ultimately the core here is that without control over colorspace >> conversion, any sort of deterministic image processing in HTML5 is off >> the table, and you have to write your own image decoders, encoders, and >> manipulation routines in JavaScript using raw typed arrays. Maybe that's >> how it has to be, but it would be cool to at least support basic >> variations of these use cases in Canvas since getImageData/putImageData >> already exist and are fairly well-specified (other than this problem, >> and some nits around source rectangles and alpha transparency). > > Given that the user's device could be a very low-power device, or one with > a very small screen, but the user might still want to be manipulating very > large images, it might be best to do the "master" manipulation on the > server anyway. This request is not about efficient image manipulation (as you point out, this is best done on a high power machine) - without control over colorspace conversion any image processing is nondeterministic. There are games and apps out there that rely on getting the exact same pixels out of a given Image on all machines, and that's impossible right now due to differing behaviors. You see demoscene projects packing data into bitmaps (yuck), or games using images as the canonical representation of user-generated content. The latter, I think, is entirely defensible - maybe even desirable, since it lets end users interact with the game using photoshop or mspaint. Supporting these use cases in a cross-browser manner is impossible right now, yet they work in the desktop versions of these games. On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > On Thu, 18 Jul 2013, K. Gadd wrote: >> Out of the features suggested previously in the thread, I would >> immediately be able to make use of control over colorspace conversion >> and an ability to opt into premultiplied alpha. Not getting >> premultiplied alpha, as is the case in virtually every canvas >> implementation I've tried, has visible negative consequences for image >> quality and also reduces the performance of some use cases where bitmap >> manipulation needs to happen, due to the fact that premultiplied alpha >> is the 'preferred' form for certain types of rendering and the math >> works out better. I think the upsides to getting premultiplication are >> the same here as they are in WebGL: faster uploads/downloads, better >> results, etc. > > Can you elaborate on exactly what this would look like in terms of the API > implications? What changes to the spec did you have in mind? I don't remember what my exact intent here was, but I'll try to resynthesize it: The key here is to have a clear understanding of what data you get out of an ImageBitmap. It is *not* necessary for the end user to be able to specify it, as long as the spec dictates that all browsers provide the exact same format to end users. If we pick one format and lock to it, we want a format that discards as little source image data as possible (preferably *no* data is discarded) - which would mean the raw source image data, without any colorspace or alpha channel conversion applied. This allows all the procedural image manipulation cases described above, and makes it a very fast and straightforward path for loading images you plan to pass off to the GPU as a WebGL texture. There's a bit more on this below... On Fri, May 9, 2014 at 12:02 PM, Ian Hickson wrote: > On Thu, 18 Jul 2013, K. Gadd wrote: >> To clearly state what would make ImageBitmap useful for the use cases I >> encounter and my end-users encounter: >> ImageBitmap should be a canonical representation of a 2D bitmap, with a >> known color space, known pixel format, known alpha representation >> (premultiplied/not premultiplied), and ready for immediate rendering or >> pixel data access. It's okay if it's immutable, and it's okay if >> constructing one from an or a Blob takes time, as long as once I have >> an ImageBitmap I can use it to render and use it to extract pixel data >> without user configuration/hardware producing unpredictable results. > > This seems reasonable, but it's not really detailed enough for me to turn > it into spec. What colour space? What exactly should we be doing to the > alpha channel? Very specifically here, by 'known color space' i just mean that the color space of the image is exposed to the end user. I don't think we can possibly pick a standard color space to always use; the options are 'this machine's current color space' and 'the color space of the input bitmap'. In many cases the color space of the input bitmap is effectively 'no color space', and game developers feed the raw rgba to the GPU. It's important to support that use case without degrading the image data. A
Re: [whatwg] WebGL and ImageBitmaps
On Fri, 9 May 2014, Ian Hickson wrote: > On Thu, 18 Jul 2013, Justin Novosad wrote: > > > > To help us iterate further, I've attempted to capture the essence of > > this thread on the whatwg wiki, using the problem solving template. I > > tried to capture the main ideas that we seem to agree on so far and I > > started to think about how to handle special cases. > > > > http://wiki.whatwg.org/wiki/ImageBitmap_Options > > Are the "strongly desired options" in the above wiki page still the > options we should be adding? On the assumption that they are, I filed some bugs to cover this: https://www.w3.org/Bugs/Public/show_bug.cgi?id=25642 createImageBitmap() options: image data orientation https://www.w3.org/Bugs/Public/show_bug.cgi?id=25643 createImageBitmap() options: color space handling https://www.w3.org/Bugs/Public/show_bug.cgi?id=25644 createImageBitmap() options: alpha channel handling As usual with such bugs, if there is interest amongst browser vendors in implementing any of these, please comment on the bug. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] WebGL and ImageBitmaps
On Fri, May 9, 2014 at 2:02 PM, Ian Hickson wrote: > Given that the user's device could be a very low-power device, or one with > a very small screen, but the user might still want to be manipulating very > large images, it might be best to do the "master" manipulation on the > server anyway. > If I have a photo library with thousands of images, I don't want to upload each image--possibly megabytes each--to the server in order to manipulate it. Also, doing the work on the user's system scales to lots of users more sensibly than doing manipulations of large images on a server. I'm assuming you're referring to the case where if you try to draw a > subpart of an image and for some reason it has to be sampled (e.g. you're > drawing it larger than the source), the anti-aliasing is optimised for > tiling and so you get "leakage" from the next sprite over. > > If so, the solution is just to separate the sprites by a pixel of > transparent black, no? > If you're downscaling by more than 2:1, you need to put more than one pixel between the images, which means you have to author sprite sheets differently depending on how far down you need to zoom. A drawing flag makes a lot more sense. -- Glenn Maynard
[whatwg] WebGL and ImageBitmaps
I had been waiting until the situation around canvas in workers cleared up a bit, but having spoken to some browser vendors it seems that canvas in workers, while desired by everyone, is not a high priority issue and so might not get cleared up for some time. Therefore, I'm now looking again at some other canvas issues that we might be able to resolve first. On Thu, 18 Jul 2013, Justin Novosad wrote: > > To help us iterate further, I've attempted to capture the essence of > this thread on the whatwg wiki, using the problem solving template. I > tried to capture the main ideas that we seem to agree on so far and I > started to think about how to handle special cases. > > http://wiki.whatwg.org/wiki/ImageBitmap_Options Are the "strongly desired options" in the above wiki page still the options we should be adding? (For my own edification, when, how, and why would you use the premultiplyAlpha option? What would using it incorrectly look like?) On Thu, 11 Jul 2013, Kenneth Russell wrote: > > The step of preparing the image for use, either with WebGL or 2D canvas, > is expensive. Today, this step is necessarily done synchronously when an > HTMLImageElement is uploaded to WebGL. The current ImageBitmap proposal > would still require this synchronous step, so for WebGL at least, it > provides no improvement over the current HTML5 APIs. A major goal of > ImageBitmap was to allow Web Workers to load them, and even this ability > currently provides no advantage over HTMLImageElement. > > It is never possible to figure out automatically how the image needs to > be treated when preparing it for use with WebGL. I'm not sure where that > idea came from. On the contrary, there are eight possibilities (2^3), > and different applications require different combinations. The wiki page proposal has 16 possibilities; are some of these unnecessary? On Fri, 12 Jul 2013, Justin Novosad wrote: > > The main concern I have with all this is the potential for OOM crashes. > I'm happy as long as the spec remains vague about what "undue latency" > means, so we still have the possibility of gracefully degrading > performance in low memory conditions by evicting prepared/decoded > buffers when necessary, to only hang on to compressed copies in the > in-memory resource cache. As far as the decode cache is concerned, the > idea I have is to give priority to pixel buffers that are held by > ImageBitmap objects, and only evict them as a last resort. Certainly the spec will always allow such leniency. At the end of the day, performance requirements are meaningless since we can't control the hardware or machine load anyway. On Tue, 16 Jul 2013, Kenneth Russell wrote: > > Additionally, the WebGL spec can be updated to state that the parameters > UNPACK_FLIP_Y_WEBGL, etc. don't apply to ImageBitmap, so the only way to > affect the decoding is with the dictionary of options. Is there some list from WebGL that we can reuse for createImageBitmap()? On Thu, 18 Jul 2013, K. Gadd wrote: > > Ultimately the core here is that without control over colorspace > conversion, any sort of deterministic image processing in HTML5 is off > the table, and you have to write your own image decoders, encoders, and > manipulation routines in JavaScript using raw typed arrays. Maybe that's > how it has to be, but it would be cool to at least support basic > variations of these use cases in Canvas since getImageData/putImageData > already exist and are fairly well-specified (other than this problem, > and some nits around source rectangles and alpha transparency). Given that the user's device could be a very low-power device, or one with a very small screen, but the user might still want to be manipulating very large images, it might be best to do the "master" manipulation on the server anyway. > Out of the features suggested previously in the thread, I would > immediately be able to make use of control over colorspace conversion > and an ability to opt into premultiplied alpha. Not getting > premultiplied alpha, as is the case in virtually every canvas > implementation I've tried, has visible negative consequences for image > quality and also reduces the performance of some use cases where bitmap > manipulation needs to happen, due to the fact that premultiplied alpha > is the 'preferred' form for certain types of rendering and the math > works out better. I think the upsides to getting premultiplication are > the same here as they are in WebGL: faster uploads/downloads, better > results, etc. Can you elaborate on exactly what this would look like in terms of the API implications? What changes to the spec did you have in mind? > To clearly state what would make ImageBitmap useful for the use cases I > encounter and my end-users encounter: > ImageBitmap should be a canonical representation of a 2D bitmap, with a > known color space, known pixel format, known alpha representation > (premultipli
