Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Ashley Gullen]

It's an interesting option to create a non-premultiplied ImageBitmap and
convert on to an ImageData. However I was under the impression the point of
ImageBitmap was to be drawn "without undue latency", which browsers could
interpret as a pre-allocated GPU texture. Doesn't this make it difficult to
use ImageBitmap as a conversion intermediary? If GPU readback is involved,
that could be slow, and if it's in RAM, isn't it storing the image twice?
This point was one of the motivating factors in writing a spec that could
do things like a direct asynchronous Blob -> ImageData.




On 10 February 2016 at 20:27, Justin Novosad  wrote:

> On Wed, Feb 10, 2016 at 2:38 PM, Ashley Gullen  wrote:
>
>> ImageBitmap is not useful for getting the data from: it still requires
>> synchronous use of a canvas to turn in to an ImageData. I would encourage
>> browser vendors to look at my spec proposal to avoid this:
>> http://wicg.github.io/img-conversion/
>
>
> Yes, that is on our radar. We need APIs to move image data without making
> so many intermediate copies in RAM. AFAIK, this appears to be a real
> problem for image manipulation apps, which tend to quickly run out of RAM.
> ImageBitmap partially addresses this, but it does little for apps that
> need to do direct image data manipulation.
>
> ImageData.toBlob is low hanging fruit IMHO.
>
> For the create method, I'm not sure. I think a smaller API surface that
> would provide the same functionality is a transfer method that transfers an
> ImageBitmap into an ImageData. This would neuter the image bitmap, it would
> throw an exception if the ImageBitmap is tainted. To get lossless
> unpremultiplied image data you would leverage the imagebitmap options.
>
>
>>
>> On 10 February 2016 at 18:29, Gregg Tavares  wrote:
>>
>> > Is there a reason in the proposal many of the options default to
>> > "implementation specific behavior"?
>> >
>> > If the point of ImageBitmap is to get the data (use Image if you don't
>> > care), then it seems like having any "implementation defined" options,
>> > especially as the default, is just asking for lurking bugs in websites
>> >
>>
>
>

[whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Xida Chen]

The Whatwg thread of requesting the ImageBitmap options feature can be
found here:
https://lists.w3.org/Archives/Public/public-whatwg-archive/2013Jun/0114.html
https://lists.w3.org/Archives/Public/public-whatwg-archive/2013Jul/0075.html

The Whatwg proposal page can be found here:
https://wiki.whatwg.org/wiki/ImageBitmap_Options

We intend to push this feature forward in Blink, particularly we intend to
spec and implement the "Strongly desired options" listed on the Whatwg
proposal page. We would appreciate comments and suggestions on the proposal.

Any thoughts or objections before we start drafting a change to the spec?

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Gregg Tavares]

Is there a reason in the proposal many of the options default to
"implementation specific behavior"?

If the point of ImageBitmap is to get the data (use Image if you don't
care), then it seems like having any "implementation defined" options,
especially as the default, is just asking for lurking bugs in websites

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Justin Novosad]

On Wed, Feb 10, 2016 at 1:29 PM, Gregg Tavares  wrote:

> Is there a reason in the proposal many of the options default to
> "implementation specific behavior"?
>

Yes. It is because different browsers have implemented their graphics
pipelines in different ways. For example, decoded images may or may not be
cached in a form that has the display device's color profile baked-in.  The
defaults are designed to avoid coercing implementation into a specific
graphics pipeline design. This follows what was done in WebGL with pixel
storage parameters. See UNPACK_COLORSPACE_CONVERSION_WEBGL :
https://www.khronos.org/registry/webgl/specs/latest/1.0/#PIXEL_STORAGE_PARAMETERS

Similarly, implementation may or may not be designed in a way where storing
images in alpha pre-multiplied form is optimal.


>
> If the point of ImageBitmap is to get the data (use Image if you don't
> care), then it seems like having any "implementation defined" options,
> especially as the default, is just asking for lurking bugs in websites.
>

Debatable. In blink, for example, images get the display color profile
applied at image decode time and the WebGL render buffer goes straight to
display with out any color space conversion. Other implementations may use
an sRGB or other intermediate color space and apply a color correction
downstream when bringing the render buffer to the display. Blink might do
this in the future too, who knows. There are pros and cons to both
approaches which we do not need to debate here, but the fact is that there
are a lot of differences between implementations, which we have to live
with. And it is not all bad that the spec allows for this flexibility,
which allows implementors to chose certain compromises. So using the
browser default color space conversion should be the right thing when
passing color values straight through WebGL (or 2D canvas). And forcing no
color correction, should be the right thing for image data that is not
color values (like bump maps).  Right now we are missing a solution for
device-independent (or implementation-independent) shading, but that is a
WebGL problem.

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Boris Zbarsky]

On 2/10/16 1:25 PM, Domenic Denicola wrote:

In new JavaScript-only APIs we've made the decision to move away from the 
potentially-confusing HTML style crossOrigin enums in favor of the RequestCredentials 
enum used by Fetch: https://fetch.spec.whatwg.org/#requestcredentials. You can see this 
in e.g. https://github.com/whatwg/html/pull/608 where I chose the same initial 
crossOrigin design and Anne convinced me to move to credentials. I imagine we'll continue 
to use crossorigin="" and corresponding reflected crossOrigin IDL attributes 
for any HTML elements, but for JS-only APIs RequestCredentials is the way to go.


That's not _quite_ the same thing.  The HTML setup basically lets you 
specify one of:


1)  No CORS (attr not set).
2)  CORS, RequestCredentials == "include" (crossrigin="use-credentials")
3)  CORS, RequestCredentials == "same-origin" (any other attr value)

Note that in the pull request you reference your default was not 
actually any of those situations, as far as I can tell, so I agree that 
using "crossOrigin" there was not a good fit.  But for the ImageBitmap 
case, we do want to support case 1 above, at least assuming tainted 
ImageBitmap is a thing.  If it's not, then I agree that just a 
RequestCredentials value is probably sufficient and all the loads 
involved should use CORS.


That said, the actual phrasing around "crossOrigin" in
https://wiki.whatwg.org/wiki/ImageBitmap_Options doesn't make much sense 
(e.g. it in fact is not a "CORS settings attribute" because it's not a 
markup attribute at all).  But we can wordsmith it better once we agree 
on what we actually want it to do.


-Boris

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Kenneth Russell]

On Wed, Feb 10, 2016 at 10:25 AM, Domenic Denicola  wrote:

> From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Xida
> Chen
>
> > We intend to push this feature forward in Blink, particularly we intend
> to
> > spec and implement the "Strongly desired options" listed on the Whatwg
> > proposal page. We would appreciate comments and suggestions on the
> > proposal.
> >
> > Any thoughts or objections before we start drafting a change to the spec?
>
> Exciting stuff!
>

Agreed -- thanks Xida for pushing this feature forward. It'll significantly
improve texture handling performance in WebGL applications.



>
> > 'none': Do not change orientation.
>
> Does this mean "disregard image orientation metadata" or does it mean "do
> not change orientation from what the metadata specifies"?
>
> > enum colorspaceConversion, default = 'default'
>
> Is there any precedent on the web platform (or in Open GL APIs, perhaps?)
> for whether "colorspace" is one word or two words? If two words, this
> should be colorSpaceConversion. Happy to defer to your expertise here.
>

The WebGL spec has been using a single word for this, but two words are
probably more correct.



>
> > 'default': Implementation-specific behavior, possibly optimized for the
> implementation's graphics framework.
>
> This is a bit unfortunate as an interop hazard. I suppose it's no worse
> than today though. Has this been discussed in the past? Apologies, I was
> only able to skim the previous email thread.
>

This provides the functionality of the existing
UNPACK_COLORSPACE_CONVERSION_WEBGL flag in the WebGL spec. By default
images uploaded to WebGL are decoded "however the browser does it" -- which
isn't tightly specified -- but most WebGL apps that draw images just do
this and get correct results.

However, in special cases and for sophisticated apps, it's absolutely
necessary to get the raw data that's in the image, which is why there has
to be a switch to turn it off.

-Ken




>
> ---
>
> This comment is on something outside the "strongly desired features"
> section so probably not relevant to your immediate work. Just wanted to
> mention it.
>
> > DOMString? crossOrigin
>
> In new JavaScript-only APIs we've made the decision to move away from the
> potentially-confusing HTML style crossOrigin enums in favor of the
> RequestCredentials enum used by Fetch:
> https://fetch.spec.whatwg.org/#requestcredentials. You can see this in
> e.g. https://github.com/whatwg/html/pull/608 where I chose the same
> initial crossOrigin design and Anne convinced me to move to credentials. I
> imagine we'll continue to use crossorigin="" and corresponding reflected
> crossOrigin IDL attributes for any HTML elements, but for JS-only APIs
> RequestCredentials is the way to go.
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Domenic Denicola]

From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of Xida Chen

> We intend to push this feature forward in Blink, particularly we intend to
> spec and implement the "Strongly desired options" listed on the Whatwg
> proposal page. We would appreciate comments and suggestions on the
> proposal.
> 
> Any thoughts or objections before we start drafting a change to the spec?

Exciting stuff!

> 'none': Do not change orientation.

Does this mean "disregard image orientation metadata" or does it mean "do not 
change orientation from what the metadata specifies"?

> enum colorspaceConversion, default = 'default'

Is there any precedent on the web platform (or in Open GL APIs, perhaps?) for 
whether "colorspace" is one word or two words? If two words, this should be 
colorSpaceConversion. Happy to defer to your expertise here.

> 'default': Implementation-specific behavior, possibly optimized for the 
> implementation's graphics framework.

This is a bit unfortunate as an interop hazard. I suppose it's no worse than 
today though. Has this been discussed in the past? Apologies, I was only able 
to skim the previous email thread.

---

This comment is on something outside the "strongly desired features" section so 
probably not relevant to your immediate work. Just wanted to mention it.

> DOMString? crossOrigin

In new JavaScript-only APIs we've made the decision to move away from the 
potentially-confusing HTML style crossOrigin enums in favor of the 
RequestCredentials enum used by Fetch: 
https://fetch.spec.whatwg.org/#requestcredentials. You can see this in e.g. 
https://github.com/whatwg/html/pull/608 where I chose the same initial 
crossOrigin design and Anne convinced me to move to credentials. I imagine 
we'll continue to use crossorigin="" and corresponding reflected crossOrigin 
IDL attributes for any HTML elements, but for JS-only APIs RequestCredentials 
is the way to go.

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Ashley Gullen]

ImageBitmap is not useful for getting the data from: it still requires
synchronous use of a canvas to turn in to an ImageData. I would encourage
browser vendors to look at my spec proposal to avoid this:
http://wicg.github.io/img-conversion/

On 10 February 2016 at 18:29, Gregg Tavares  wrote:

> Is there a reason in the proposal many of the options default to
> "implementation specific behavior"?
>
> If the point of ImageBitmap is to get the data (use Image if you don't
> care), then it seems like having any "implementation defined" options,
> especially as the default, is just asking for lurking bugs in websites
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Justin Novosad]

On Wed, Feb 10, 2016 at 2:38 PM, Ashley Gullen  wrote:

> ImageBitmap is not useful for getting the data from: it still requires
> synchronous use of a canvas to turn in to an ImageData. I would encourage
> browser vendors to look at my spec proposal to avoid this:
> http://wicg.github.io/img-conversion/


Yes, that is on our radar. We need APIs to move image data without making
so many intermediate copies in RAM. AFAIK, this appears to be a real
problem for image manipulation apps, which tend to quickly run out of RAM.
ImageBitmap partially addresses this, but it does little for apps that need
to do direct image data manipulation.

ImageData.toBlob is low hanging fruit IMHO.

For the create method, I'm not sure. I think a smaller API surface that
would provide the same functionality is a transfer method that transfers an
ImageBitmap into an ImageData. This would neuter the image bitmap, it would
throw an exception if the ImageBitmap is tainted. To get lossless
unpremultiplied image data you would leverage the imagebitmap options.


>
> On 10 February 2016 at 18:29, Gregg Tavares  wrote:
>
> > Is there a reason in the proposal many of the options default to
> > "implementation specific behavior"?
> >
> > If the point of ImageBitmap is to get the data (use Image if you don't
> > care), then it seems like having any "implementation defined" options,
> > especially as the default, is just asking for lurking bugs in websites
> >
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Justin Novosad]

On Wed, Feb 10, 2016 at 2:29 PM, Boris Zbarsky  wrote:

> On 2/10/16 1:25 PM, Domenic Denicola wrote:
>
>> In new JavaScript-only APIs we've made the decision to move away from the
>> potentially-confusing HTML style crossOrigin enums in favor of the
>> RequestCredentials enum used by Fetch:
>> https://fetch.spec.whatwg.org/#requestcredentials. You can see this in
>> e.g. https://github.com/whatwg/html/pull/608 where I chose the same
>> initial crossOrigin design and Anne convinced me to move to credentials. I
>> imagine we'll continue to use crossorigin="" and corresponding reflected
>> crossOrigin IDL attributes for any HTML elements, but for JS-only APIs
>> RequestCredentials is the way to go.
>>
>
> That's not _quite_ the same thing.  The HTML setup basically lets you
> specify one of:
>
> 1)  No CORS (attr not set).
> 2)  CORS, RequestCredentials == "include" (crossrigin="use-credentials")
> 3)  CORS, RequestCredentials == "same-origin" (any other attr value)
>
> Note that in the pull request you reference your default was not actually
> any of those situations, as far as I can tell, so I agree that using
> "crossOrigin" there was not a good fit.  But for the ImageBitmap case, we
> do want to support case 1 above, at least assuming tainted ImageBitmap is a
> thing.  If it's not, then I agree that just a RequestCredentials value is
> probably sufficient and all the loads involved should use CORS.
>

Tainted ImageBitmaps is a thing: https://github.com/whatwg/html/pull/385
That said, this is not one of the options we intend to move forward with in
the near term (It is not in the highly desired list). It is not clear that
the feature is even needed. I am no CORS expert, but I think you can get an
untainted ImageBitmap from a cross-origin image by using: XHR (with
credentials) -> blob -> createImageBitmap


> That said, the actual phrasing around "crossOrigin" in
> https://wiki.whatwg.org/wiki/ImageBitmap_Options doesn't make much sense
> (e.g. it in fact is not a "CORS settings attribute" because it's not a
> markup attribute at all).  But we can wordsmith it better once we agree on
> what we actually want it to do.
>
> -Boris
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Kenneth Russell]

On Wed, Feb 10, 2016 at 12:06 PM, Justin Novosad  wrote:

> On Wed, Feb 10, 2016 at 2:29 PM, Boris Zbarsky  wrote:
>
> > On 2/10/16 1:25 PM, Domenic Denicola wrote:
> >
> >> In new JavaScript-only APIs we've made the decision to move away from
> the
> >> potentially-confusing HTML style crossOrigin enums in favor of the
> >> RequestCredentials enum used by Fetch:
> >> https://fetch.spec.whatwg.org/#requestcredentials. You can see this in
> >> e.g. https://github.com/whatwg/html/pull/608 where I chose the same
> >> initial crossOrigin design and Anne convinced me to move to
> credentials. I
> >> imagine we'll continue to use crossorigin="" and corresponding reflected
> >> crossOrigin IDL attributes for any HTML elements, but for JS-only APIs
> >> RequestCredentials is the way to go.
> >>
> >
> > That's not _quite_ the same thing.  The HTML setup basically lets you
> > specify one of:
> >
> > 1)  No CORS (attr not set).
> > 2)  CORS, RequestCredentials == "include" (crossrigin="use-credentials")
> > 3)  CORS, RequestCredentials == "same-origin" (any other attr value)
> >
> > Note that in the pull request you reference your default was not actually
> > any of those situations, as far as I can tell, so I agree that using
> > "crossOrigin" there was not a good fit.  But for the ImageBitmap case, we
> > do want to support case 1 above, at least assuming tainted ImageBitmap
> is a
> > thing.  If it's not, then I agree that just a RequestCredentials value is
> > probably sufficient and all the loads involved should use CORS.
> >
>
> Tainted ImageBitmaps is a thing: https://github.com/whatwg/html/pull/385
> That said, this is not one of the options we intend to move forward with in
> the near term (It is not in the highly desired list). It is not clear that
> the feature is even needed. I am no CORS expert, but I think you can get an
> untainted ImageBitmap from a cross-origin image by using: XHR (with
> credentials) -> blob -> createImageBitmap
>

Agreed -- I don't think the ImageBitmap constructor from URL, or the
crossOrigin ImageBitmap constructor argument, are needed any more. It would
be good to fully implement ImageBitmap (including asynchronous decoding),
plus some of the WebGL-related constructor arguments, and verify that the
texture uploads are much faster. At that point let's delete the crossOrigin
ImageBitmap constructor argument, and the proposal for creating
ImageBitmaps from URLs.

-Ken




>
>
> > That said, the actual phrasing around "crossOrigin" in
> > https://wiki.whatwg.org/wiki/ImageBitmap_Options doesn't make much sense
> > (e.g. it in fact is not a "CORS settings attribute" because it's not a
> > markup attribute at all).  But we can wordsmith it better once we agree
> on
> > what we actually want it to do.
> >
> > -Boris
> >
>

Re: [whatwg] Reviving ImageBitmap options: Intend to spec and implement

[Robert O'Callahan]

On Thu, Feb 11, 2016 at 9:27 AM, Justin Novosad  wrote:

> On Wed, Feb 10, 2016 at 2:38 PM, Ashley Gullen  wrote:
>
> > ImageBitmap is not useful for getting the data from: it still requires
> > synchronous use of a canvas to turn in to an ImageData. I would encourage
> > browser vendors to look at my spec proposal to avoid this:
> > http://wicg.github.io/img-conversion/
>
>
> Yes, that is on our radar. We need APIs to move image data without making
> so many intermediate copies in RAM. AFAIK, this appears to be a real
> problem for image manipulation apps, which tend to quickly run out of RAM.
> ImageBitmap partially addresses this, but it does little for apps that need
> to do direct image data manipulation.
>

http://kakukogou.github.io/spec-imagebitmap-extension/

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn