Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Thu, Jan 7, 2016 at 10:32 AM, Robert O'Callahanwrote: > Some of the things that would need to be handled: > -- controls need to not expose sensitive data about file > paths No great ideas here other than simply not rendering them. > -- For SVG images we disable native themes to avoid those being inspectable > by the Web site > -- Non-origin-clean canvas images, frames and MediaStream frames > would have to be suppressed > -- Non-same origin content (, , etc) would have to be blocked. > This isn't as simple as a change to Fetch, since a site could create an > element and load its contents in an unrestricted browsing context and move > it into a different document with different rules. Well, if we restrict and popups, how would you create an unrestricted browsing context? > -- :visited (I do wonder though if with this many restrictions folks would still want this feature, since it cripples quite a few things.) -- https://annevankesteren.nl/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Fri, Jan 8, 2016 at 11:08 AM, Jeff Gilbertwrote: > On Wed, Jan 6, 2016 at 8:07 PM, Luke Wagner wrote: > > FWIW, I was wondering if we could go a step further and allow > > (optional) user interaction with the rendered DOM elements. That way > > you could, say, select text on a 3d surface with a mouse or use an > > tag. It seems like this would be possible if the vertex/pixel > > shaders were constrained to, say, affine transformations which starts > > to sound rather similar to the whitelisting approach mentioned > > elsewhere in this thread for mitigating timing attacks. > > I think this should be handled outside of the WebGL API. This seems to > be a bunch of extra work and complexity to handle a relatively narrow > use-case. I believe it's much easier to just provide functionality in > the platform to allow a library to implement such specific uses. > Some features are always going to be really hard to provide with fully custom UI. For example , for security reasons. Also full custom UI for text entry is practically impossible to implement on par with browser built-in UI, due to the need to integrate with system IMEs, assistive tech, spellchecking dictionaries, etc. That's why I think finding ways to integrate live Web content into WebGL scenes would be really valuable. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Thu, Jan 7, 2016 at 8:46 PM, Anne van Kesterenwrote: > At least enforcing CORS-same-origin would be somewhat trivial from a > specification perspective since all fetches go through Fetch. Limiting > plugins and other affected features would be some added conditionals > here and there. I don't see how content changes would have an impact > since you can only change the policy through navigation at which point > you'd have a new global and such anyway. > Some of the things that would need to be handled: -- controls need to not expose sensitive data about file paths -- For SVG images we disable native themes to avoid those being inspectable by the Web site -- Non-origin-clean canvas images, frames and MediaStream frames would have to be suppressed -- Non-same origin content (, , etc) would have to be blocked. This isn't as simple as a change to Fetch, since a site could create an element and load its contents in an unrestricted browsing context and move it into a different document with different rules. -- :visited Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Fri, Jan 8, 2016 at 2:36 PM, Jeff Gilbertwrote: > I think rather we should aim to provide a way for the application to > implement custom remapping of events. A simple version of this is > allowing the app to synthesize enough of the inputs to virtually use > our existing UI. I think this is much simpler than dredging around in > the WebGL API. > That might work. It'd get complicated for touch events. Also there are security considerations since right now a page can't generate events that go through into cross-origin s (though in practice it can use clickjacking to make them happen, in many cases). There's still the output side, of course. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
I think rather we should aim to provide a way for the application to implement custom remapping of events. A simple version of this is allowing the app to synthesize enough of the inputs to virtually use our existing UI. I think this is much simpler than dredging around in the WebGL API. On Thu, Jan 7, 2016 at 4:12 PM, Robert O'Callahanwrote: > On Fri, Jan 8, 2016 at 11:08 AM, Jeff Gilbert wrote: >> >> On Wed, Jan 6, 2016 at 8:07 PM, Luke Wagner wrote: >> > FWIW, I was wondering if we could go a step further and allow >> > (optional) user interaction with the rendered DOM elements. That way >> > you could, say, select text on a 3d surface with a mouse or use an >> > tag. It seems like this would be possible if the vertex/pixel >> > shaders were constrained to, say, affine transformations which starts >> > to sound rather similar to the whitelisting approach mentioned >> > elsewhere in this thread for mitigating timing attacks. >> >> I think this should be handled outside of the WebGL API. This seems to >> be a bunch of extra work and complexity to handle a relatively narrow >> use-case. I believe it's much easier to just provide functionality in >> the platform to allow a library to implement such specific uses. > > > Some features are always going to be really hard to provide with fully > custom UI. For example , for security reasons. Also full > custom UI for text entry is practically impossible to implement on par with > browser built-in UI, due to the need to integrate with system IMEs, > assistive tech, spellchecking dictionaries, etc. > > That's why I think finding ways to integrate live Web content into WebGL > scenes would be really valuable. > > Rob > -- > lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf > toD > selthor stor edna siewaoeodm or v sstvr esBa kbvted,t > rdsme,aoreseoouoto > o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr > .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr > esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Fri, Jan 8, 2016 at 9:58 AM,wrote: > There are two use cases for this functionality needed by the WebVR team. > > The one needed earliest is to implement HUD interfaces and dialogues. In > this case, we will only be using this API from chrome privileged code. > jgilbert has started on a mechanism we can use for now for this case as a > chrome-only API (Bug 1237489). > > For the second case, which could benefit everyone, we would like to use > DOM elements in any WebGL scene. The intent would be to deprecate the API > added in Bug 1237489 once the Khronos specification for > WEBGL_dynamic_texture is more mature and can be used in unprivileged > content. jgilbert mentions that the WEBGL_dynamic_texture specification is > due for a pruning refactor that should simplify things a bit. > > Would it help to reduce the scope and brittle-ness of the security model > by white-listing elements that are allowed in non-privileged content? The > original WEBGL_dynamic_texture proposal names specifically > HTMLVideoElement, HTMLCanvasElement and HTMLImageElement. Perhaps these > would be a good start, with an intent to expand later? > I don't think this really helps since the element types are only part of the problem. If we wish to style content in a way that it gets sanitized of security > sensitive content, perhaps we could roll it up in a convenient CSS > attribute applied to the element we are capturing as a texture while also > ensuring that the element generates a layer and is taken out of flow from > the 2d document. > > How would you feel about this CSS: > > position: embed > > It would work like "position: absolute" in that it takes the element out > of flow; however, the element would not be positioned or rendered in the 2d > layout. It would also ensure that the element generates a layer, similar > to "will-change: transform". When called from non-privileged content, it > would enforce the whitelist, CORS, and sanitize any security sensitive > output such as those that Roc has identified earlier. > I don't think this really helps either since it still leaves us with the core problem: a security model that's hard to spec, hard to implement, and hard for authors to understand. Different ways to opt into it don't address that. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Wed, Jan 6, 2016 at 8:07 PM, Luke Wagnerwrote: > FWIW, I was wondering if we could go a step further and allow > (optional) user interaction with the rendered DOM elements. That way > you could, say, select text on a 3d surface with a mouse or use an > tag. It seems like this would be possible if the vertex/pixel > shaders were constrained to, say, affine transformations which starts > to sound rather similar to the whitelisting approach mentioned > elsewhere in this thread for mitigating timing attacks. I think this should be handled outside of the WebGL API. This seems to be a bunch of extra work and complexity to handle a relatively narrow use-case. I believe it's much easier to just provide functionality in the platform to allow a library to implement such specific uses. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Thursday, January 7, 2016 at 1:32:59 AM UTC-8, Robert O'Callahan wrote: > On Thu, Jan 7, 2016 at 8:46 PM, Anne van Kesterenwrote: > > > At least enforcing CORS-same-origin would be somewhat trivial from a > > specification perspective since all fetches go through Fetch. Limiting > > plugins and other affected features would be some added conditionals > > here and there. I don't see how content changes would have an impact > > since you can only change the policy through navigation at which point > > you'd have a new global and such anyway. > > > > Some of the things that would need to be handled: > -- controls need to not expose sensitive data about > file paths > -- For SVG images we disable native themes to avoid those being inspectable > by the Web site > -- Non-origin-clean canvas images, frames and MediaStream frames > would have to be suppressed > -- Non-same origin content (, , etc) would have to be blocked. > This isn't as simple as a change to Fetch, since a site could create an > element and load its contents in an unrestricted browsing context and move > it into a different document with different rules. > -- :visited > > Rob > -- > lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf > toD > selthor stor edna siewaoeodm or v sstvr esBa kbvted,t > rdsme,aoreseoouoto > o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea > lurpr > .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr > esn There are two use cases for this functionality needed by the WebVR team. The one needed earliest is to implement HUD interfaces and dialogues. In this case, we will only be using this API from chrome privileged code. jgilbert has started on a mechanism we can use for now for this case as a chrome-only API (Bug 1237489). For the second case, which could benefit everyone, we would like to use DOM elements in any WebGL scene. The intent would be to deprecate the API added in Bug 1237489 once the Khronos specification for WEBGL_dynamic_texture is more mature and can be used in unprivileged content. jgilbert mentions that the WEBGL_dynamic_texture specification is due for a pruning refactor that should simplify things a bit. Would it help to reduce the scope and brittle-ness of the security model by white-listing elements that are allowed in non-privileged content? The original WEBGL_dynamic_texture proposal names specifically HTMLVideoElement, HTMLCanvasElement and HTMLImageElement. Perhaps these would be a good start, with an intent to expand later? If we wish to style content in a way that it gets sanitized of security sensitive content, perhaps we could roll it up in a convenient CSS attribute applied to the element we are capturing as a texture while also ensuring that the element generates a layer and is taken out of flow from the 2d document. How would you feel about this CSS: position: embed It would work like "position: absolute" in that it takes the element out of flow; however, the element would not be positioned or rendered in the 2d layout. It would also ensure that the element generates a layer, similar to "will-change: transform". When called from non-privileged content, it would enforce the whitelist, CORS, and sanitize any security sensitive output such as those that Roc has identified earlier. A chrome-only version with a different name could be used from privileged code to implement backwards compatibility of 2d web pages in a VR based browser: position: embed-unsanitized (I'd love to hear other attribute naming ideas) For the case of 2d web pages in a VR based browser, we would need to include things such as iframes, cross-origin content, and :visited link styles. These could be restricted to be used by privileged code only with "position: embed-unsanitized" Thanks for exploring this with me and the great feedback! ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Wed, Jan 6, 2016 at 9:50 PM, Robert O'Callahanwrote: > Where would you put that flag? Simplest would be an HTTP header I suppose. > I think this has basically the same problems: very difficult to specify and > police, and fragile when the content changes. At least enforcing CORS-same-origin would be somewhat trivial from a specification perspective since all fetches go through Fetch. Limiting plugins and other affected features would be some added conditionals here and there. I don't see how content changes would have an impact since you can only change the policy through navigation at which point you'd have a new global and such anyway. -- https://annevankesteren.nl/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Wed, Jan 6, 2016 at 9:52 PM, Anne van Kesterenwrote: > On Tue, Jan 5, 2016 at 10:18 PM, Robert O'Callahan > wrote: > > That's an option, but it's a very large problem that's very difficult to > > get right. And there are so many things that need to be plugged, it would > > make the feature very fragile: change the Web content in some subtle way > > and suddenly it looks wrong or disappears altogether for no apparent > reason. > > We could also go the other way. And have some kind of opt-in flag that > requires everything to be CORS-same-origin and doesn't allow :visited > and plugins or some such. And if you have that enabled you get to use > some new features. Might be easier than tracking on a per layout box > basis whether it violates some constraints. > Where would you put that flag? I think this has basically the same problems: very difficult to specify and police, and fragile when the content changes. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On 2016-01-04 3:54 PM, Xidorn Quan wrote: > On Tue, Jan 5, 2016 at 8:46 AM, Kearwood "Kip" Gilbert >wrote: >> Hello All, >> >> In WebVR, we often present UI as a Head's Up Display (HUD) that floats >> in front of the user. Additionally, we often wish to show 2d graphics, >> video, and CSS animations as a texture in 3d scenes. Creating these >> textures is something that CSS and HTML are great at. >> >> Unfortunately, I am not aware of an easy and efficient way to capture an >> animated of an interactive HTML Element and bring it into the WebGL >> context. A "moz-element" -like API would be useful here. > Is it possible to access pixels' color inside the texture? If yes, > please mind the privacy issue around :visited selector on links, as > users' history would be leaked via either painting link directly, or > painting an element using -moz-element(#a-link-element) as background. Thanks Xidorn! The proposed WEBGL_security_sensitive_resources extension describes how we could ensure that content does not read pixels from buffers that were rendered with security sensitive textures. If we don't implement read-only buffers with WEBGL_security_sensitive_resources, would it be sufficient to render content styled without the :visited selector and not allow cross-origin content? Are there some other security concerns I may have missed? Cheers, - Kip > > More information can be found in [1] and also [2]. > > [1] > https://developer.mozilla.org/en-US/docs/Web/CSS/Privacy_and_the_:visited_selector > [2] > https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Drawing_DOM_objects_into_a_canvas#Security > > - Xidorn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Wed, Jan 6, 2016 at 9:45 AM, Kearwood "Kip" Gilbertwrote: > If it is not feasible to prevent variation in timing or to otherwise > prevent content from determining the content of the texture, perhaps > another approach would be to require all elements to be sanitized before > capturing their textures. For example, :visited styles should not be > evaluated and cross-origin content should not be allowed. > That's an option, but it's a very large problem that's very difficult to get right. And there are so many things that need to be plugged, it would make the feature very fragile: change the Web content in some subtle way and suddenly it looks wrong or disappears altogether for no apparent reason. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On 2016-01-04 4:46 PM, Robert O'Callahan wrote: > On Tue, Jan 5, 2016 at 10:46 AM, Kearwood "Kip" Gilbert < > kgilb...@mozilla.com> wrote: > >> In WebVR, we often present UI as a Head's Up Display (HUD) that floats >> in front of the user. Additionally, we often wish to show 2d graphics, >> video, and CSS animations as a texture in 3d scenes. Creating these >> textures is something that CSS and HTML are great at. >> >> Unfortunately, I am not aware of an easy and efficient way to capture an >> animated of an interactive HTML Element and bring it into the WebGL >> context. A "moz-element" -like API would be useful here. >> >> Perhaps we could solve this by implementing and extending the proposed >> WEBGL_dynamic_texture extension: >> >> >> https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ > > This proposal seems unnecessarily complex. Is there a way for me to send > feedback? There is a mailing list, public_we...@khronos.org: https://www.khronos.org/webgl/public-mailing-list/ There is also a github repository with issues and pull requests: https://github.com/KhronosGroup/WebGL/tree/master/extensions/proposals > > Essentially, we would extend the same API but allow the WDTStream >> interface to apply to more HTML elements, not just HTMLCanvasElement, >> HTMLImageElement, or HTMLVideoElement. >> >> We would also need to implement WEBGL_security_sensitive_resources to >> enforce the security model: >> >> >> https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > > I wish I'd known about this proposal earlier! This looks pretty good, > though I'd always thought this would be too complicated to spec and > implement to be practical. Glad to be wrong! Although I think we should get > as much feedback as possible on this in case of hidden gotchas. > > Does this sound like a good idea? I feel that this is something that >> all WebGL developers would want, as it would make building front-ends >> > for games much easier. > Yes, I think together these would be very useful. > > If others feel the same, I would also like to follow up with a proposal >> to make the captured HTML elements interactive through use of an >> explicit "pick buffer" added to canvases. >> > How would that work? Being able to synthesize mouse (touch?) events in HTML > elements would add another set of issues. The intent is to enable content to work transparently with multiple cursor models implemented at the platform level. We wish the cursor to remain responsive even if the page is not. As vertex shaders may be applied to elements and elements may be occluded by other objects in a 3d scene, content needs to describe the relationship between the original coordinate system of the elements and the rendered output. The proposed method of doing this is with a pick-buffer. A pick buffer can be viewed as a simple GPU accelerated raycasting engine to the underlying scene. A pick buffer is composed of an additional render target, which can be either rendered in a separate pass or using WEBGL_draw_buffers in a single pass. Rather than interpreting the 32-bpp data as RGBA images for display, the fragment shader will encode an object id and interpolated uv coordinates. When the user passes their mouse over the canvas (or adjusts their gaze with a stereoscopic VR cursor), the pick-buffer would be sampled and interpreted by platform-level code. The information within the sample would be used to identify the underlying element (though an pick buffer id to element id lookup table) as well as the coordinate within the un-transformed space of the element to synthesize events on. Security is a concern, as content may use this feature to obscure the appearance of an element with discontinuity between the displayed buffer and the pick buffer. I would suggest that we enforce a CORS-like policy that disables pick buffers unless all source textures used to render the buffer came from sanitized inputs and are not cross-origin. In the case of implementing a VR web browser with 2d backwards compatibility, perhaps some restrictions can be relaxed in the case of chrome-only access to the API. To avoid performance overhead, the pick buffer could be rendered by content at lower intervals or with a lower resolution than the rendered output. > > I assume the idea of mixing CSS 3D-transformed elements into a WebGL scene > has been rejected for some reason? > > Rob Mixing CSS 3D-transformed elements into WebGL is still on the table; however, this proposal is more future-proof enabling more expression in content. In the VR HUD case it is more ergonomic to present panels as curved surfaces rather than flat planes. When placing elements within the 3d scene itself, it is desirable to light the 2d DOM elements with the same lighting model as the rest of the scene, cast shadows, receive shadows, and receive reflections. (Think of interactive 2D dom content embedded within a glossy reflective
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On 2016-01-04 4:54 PM, Robert O'Callahan wrote: > On Tue, Jan 5, 2016 at 1:18 PM, Jonas Sickingwrote: > >> A big problem is sticking HTML/CSS content into WebGL is that WebGL >> effectively enables reading pixel data through custom shaders and >> timing attacks. >> > If you read > https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > carefully I think it's designed to prevent timing attacks by forbidding > shader control flow from depending on security-sensitive texture data. > > It's hard for me to judge how implementable it is, but in principle it > should be doable. It requires analysis of shader code. > > Rob Another vector to read back content is the depth buffer, glQueries, and use of the discard instruction. WEBGL_security_sensitive_resources prevents writing to the depth buffer when binding a security sensitive texture. If it is not feasible to prevent variation in timing or to otherwise prevent content from determining the content of the texture, perhaps another approach would be to require all elements to be sanitized before capturing their textures. For example, :visited styles should not be evaluated and cross-origin content should not be allowed. - Kip ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
WEBGL_dynamic_texture is due for a pruning refactor (I think I'm on the hook for this), so don't base anything on it as-is. IIRC, we don't believe WEBGL_security_sensitive_resources is viably implementable in a safe way (timing attacks!), so I suggest ignoring it. Extending texture uploads to allow dom::Element uploads is easily done from a technical standpoint, but doing it efficiently will take novel non-trivial work. (not every dom::Element has a Layer/Image) Adding picking to WebGL is a non-starter. >From an API standpoint, it could be interesting to try to use ImageBitmaps as handles to snapshots of dom::Elements. I think that it would be most efficient just to have a meeting about these topics, instead of iterating slower via email. -Jeff On Mon, Jan 4, 2016 at 1:46 PM, Kearwood "Kip" Gilbertwrote: > Hello All, > > In WebVR, we often present UI as a Head's Up Display (HUD) that floats > in front of the user. Additionally, we often wish to show 2d graphics, > video, and CSS animations as a texture in 3d scenes. Creating these > textures is something that CSS and HTML are great at. > > Unfortunately, I am not aware of an easy and efficient way to capture an > animated of an interactive HTML Element and bring it into the WebGL > context. A "moz-element" -like API would be useful here. > > Perhaps we could solve this by implementing and extending the proposed > WEBGL_dynamic_texture extension: > > https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ > > Essentially, we would extend the same API but allow the WDTStream > interface to apply to more HTML elements, not just HTMLCanvasElement, > HTMLImageElement, or HTMLVideoElement. > > We would also need to implement WEBGL_security_sensitive_resources to > enforce the security model: > > https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > > Does this sound like a good idea? I feel that this is something that > all WebGL developers would want, as it would make building front-ends > for games much easier. > > If others feel the same, I would also like to follow up with a proposal > to make the captured HTML elements interactive through use of an > explicit "pick buffer" added to canvases. > > I look forward to your feedback. > > Cheers, > - Kearwood "Kip" Gilbert > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 8:46 AM, Kearwood "Kip" Gilbertwrote: > Hello All, > > In WebVR, we often present UI as a Head's Up Display (HUD) that floats > in front of the user. Additionally, we often wish to show 2d graphics, > video, and CSS animations as a texture in 3d scenes. Creating these > textures is something that CSS and HTML are great at. > > Unfortunately, I am not aware of an easy and efficient way to capture an > animated of an interactive HTML Element and bring it into the WebGL > context. A "moz-element" -like API would be useful here. Is it possible to access pixels' color inside the texture? If yes, please mind the privacy issue around :visited selector on links, as users' history would be leaked via either painting link directly, or painting an element using -moz-element(#a-link-element) as background. More information can be found in [1] and also [2]. [1] https://developer.mozilla.org/en-US/docs/Web/CSS/Privacy_and_the_:visited_selector [2] https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Drawing_DOM_objects_into_a_canvas#Security - Xidorn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
Thanks for your feedback, Jeff! My comments below... On 2016-01-04 2:50 PM, Jeff Gilbert wrote: > WEBGL_dynamic_texture is due for a pruning refactor (I think I'm on > the hook for this), so don't base anything on it as-is. I'm glad I checked with you first. I would be interested in any alternatives that might be on the horizon. > > IIRC, we don't believe WEBGL_security_sensitive_resources is viably > implementable in a safe way (timing attacks!), so I suggest ignoring > it. Perhaps enforcing a CORS-like model would be simpler to secure, by limiting what content you can get handles for? > > Extending texture uploads to allow dom::Element uploads is easily done > from a technical standpoint, but doing it efficiently will take novel > non-trivial work. (not every dom::Element has a Layer/Image) Agree. Perhaps we could make the problem simpler by forcing an element to have a layer when it is captured? Perhaps similar to will-change. > > Adding picking to WebGL is a non-starter. I suspect that most of the picking / interactivity could be implemented in JS. Some extensions may make this more efficient by enabling content to perform picking asynchronously in a webworker with WEBGL_shared_resources or perhaps by using glQuery and EXT_disjoint_timer_query. > > From an API standpoint, it could be interesting to try to use > ImageBitmaps as handles to snapshots of dom::Elements. This is promising.. I'll explore this a bit. > > I think that it would be most efficient just to have a meeting about > these topics, instead of iterating slower via email. Sounds great, if you don't mind joining in. I'll ping you and get something set up. > > -Jeff Thanks again, Jeff! > > On Mon, Jan 4, 2016 at 1:46 PM, Kearwood "Kip" Gilbert >wrote: >> Hello All, >> >> In WebVR, we often present UI as a Head's Up Display (HUD) that floats >> in front of the user. Additionally, we often wish to show 2d graphics, >> video, and CSS animations as a texture in 3d scenes. Creating these >> textures is something that CSS and HTML are great at. >> >> Unfortunately, I am not aware of an easy and efficient way to capture an >> animated of an interactive HTML Element and bring it into the WebGL >> context. A "moz-element" -like API would be useful here. >> >> Perhaps we could solve this by implementing and extending the proposed >> WEBGL_dynamic_texture extension: >> >> https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ >> >> Essentially, we would extend the same API but allow the WDTStream >> interface to apply to more HTML elements, not just HTMLCanvasElement, >> HTMLImageElement, or HTMLVideoElement. >> >> We would also need to implement WEBGL_security_sensitive_resources to >> enforce the security model: >> >> https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ >> >> Does this sound like a good idea? I feel that this is something that >> all WebGL developers would want, as it would make building front-ends >> for games much easier. >> >> If others feel the same, I would also like to follow up with a proposal >> to make the captured HTML elements interactive through use of an >> explicit "pick buffer" added to canvases. >> >> I look forward to your feedback. >> >> Cheers, >> - Kearwood "Kip" Gilbert >> >> ___ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
"-Moz-Element" -like API by extending WEBGL_dynamic_texture
Hello All, In WebVR, we often present UI as a Head's Up Display (HUD) that floats in front of the user. Additionally, we often wish to show 2d graphics, video, and CSS animations as a texture in 3d scenes. Creating these textures is something that CSS and HTML are great at. Unfortunately, I am not aware of an easy and efficient way to capture an animated of an interactive HTML Element and bring it into the WebGL context. A "moz-element" -like API would be useful here. Perhaps we could solve this by implementing and extending the proposed WEBGL_dynamic_texture extension: https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ Essentially, we would extend the same API but allow the WDTStream interface to apply to more HTML elements, not just HTMLCanvasElement, HTMLImageElement, or HTMLVideoElement. We would also need to implement WEBGL_security_sensitive_resources to enforce the security model: https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ Does this sound like a good idea? I feel that this is something that all WebGL developers would want, as it would make building front-ends for games much easier. If others feel the same, I would also like to follow up with a proposal to make the captured HTML elements interactive through use of an explicit "pick buffer" added to canvases. I look forward to your feedback. Cheers, - Kearwood "Kip" Gilbert ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 11:50 AM, Jeff Gilbertwrote: > I think that it would be most efficient just to have a meeting about > these topics, instead of iterating slower via email. > FWIW I feel like it's more efficient to use email, especially if not all issues can be resolved in a single meeting. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Mon, Jan 4, 2016 at 4:46 PM, Robert O'Callahanwrote: > On Tue, Jan 5, 2016 at 10:46 AM, Kearwood "Kip" Gilbert < > kgilb...@mozilla.com> wrote: > >> In WebVR, we often present UI as a Head's Up Display (HUD) that floats >> in front of the user. Additionally, we often wish to show 2d graphics, >> video, and CSS animations as a texture in 3d scenes. Creating these >> textures is something that CSS and HTML are great at. >> >> Unfortunately, I am not aware of an easy and efficient way to capture an >> animated of an interactive HTML Element and bring it into the WebGL >> context. A "moz-element" -like API would be useful here. >> >> Perhaps we could solve this by implementing and extending the proposed >> WEBGL_dynamic_texture extension: >> >> >> https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ > > > This proposal seems unnecessarily complex. Is there a way for me to send > feedback? Yeah, we have a public mailing list: public_we...@khronos.org As with anything WebGL related, you can also just talk to me about it. > > Essentially, we would extend the same API but allow the WDTStream >> interface to apply to more HTML elements, not just HTMLCanvasElement, >> HTMLImageElement, or HTMLVideoElement. >> >> We would also need to implement WEBGL_security_sensitive_resources to >> enforce the security model: >> >> >> https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > > > I wish I'd known about this proposal earlier! This looks pretty good, > though I'd always thought this would be too complicated to spec and > implement to be practical. Glad to be wrong! Although I think we should get > as much feedback as possible on this in case of hidden gotchas. Historically in our investigations of this, it seemed very very hard to guarantee a lack of timing vectors, even just for arithmetic. (Particularly since we're handing the sources to the driver, which will try to optimize away as much as it can) > > Does this sound like a good idea? I feel that this is something that >> all WebGL developers would want, as it would make building front-ends >> > for games much easier. >> > > Yes, I think together these would be very useful. It only helps some games. Most game development efforts will likely be unaffected, since it's generally easiest to use an existing engine, which will already have such things handled. Most games will either just use overlayed HTML, or have an in-engine solution for UI. The desire to embed web elements in a scene itself is relatively rare. (Certainly few existing games do this) > > If others feel the same, I would also like to follow up with a proposal >> to make the captured HTML elements interactive through use of an >> explicit "pick buffer" added to canvases. >> > > How would that work? Being able to synthesize mouse (touch?) events in HTML > elements would add another set of issues. > > I assume the idea of mixing CSS 3D-transformed elements into a WebGL scene > has been rejected for some reason? This can't reasonably be done given the level of abstraction provided by GL APIs. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 1:18 PM, Jonas Sickingwrote: > A big problem is sticking HTML/CSS content into WebGL is that WebGL > effectively enables reading pixel data through custom shaders and > timing attacks. > If you read https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ carefully I think it's designed to prevent timing attacks by forbidding shader control flow from depending on security-sensitive texture data. It's hard for me to judge how implementable it is, but in principle it should be doable. It requires analysis of shader code. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Mon, Jan 4, 2016 at 5:46 PM, Robert O'Callahanwrote: > On Tue, Jan 5, 2016 at 2:38 PM, Jeff Gilbert wrote: > >> > Essentially, we would extend the same API but allow the WDTStream >> >> interface to apply to more HTML elements, not just HTMLCanvasElement, >> >> HTMLImageElement, or HTMLVideoElement. >> >> >> >> We would also need to implement WEBGL_security_sensitive_resources to >> >> enforce the security model: >> >> >> >> >> >> >> https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ >> > >> > >> > I wish I'd known about this proposal earlier! This looks pretty good, >> > though I'd always thought this would be too complicated to spec and >> > implement to be practical. Glad to be wrong! Although I think we should >> get >> > as much feedback as possible on this in case of hidden gotchas. >> >> Historically in our investigations of this, it seemed very very hard >> to guarantee a lack of timing vectors, even just for arithmetic. >> (Particularly since we're handing the sources to the driver, which >> will try to optimize away as much as it can) >> > > Feedback from GPU vendors would be key here, I think. I'd like to hear that > before declaring it dead. If they already did --- RIP :-). What if, rather than using a black-list approach indicating which operations are forbidden, we use a white-list approach indicating a small number of ways that security-sensitive texture data may be used? For example only allow it to be scaled, added to and clamped to a range. Or some such. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 2:38 PM, Jeff Gilbertwrote: > Yeah, we have a public mailing list: public_we...@khronos.org > As with anything WebGL related, you can also just talk to me about it. > I don't want to step on whatever you were thinking of changing, but I'm happy to go to the list if you prefer that. My initial thought is that it would be much simpler to leave all timing issues up to the browser. If you draw one of these dynamic textures in a requestAnimationFrame callback (either on the main thread or in a Worker), the browser can guess the frame presentation time and bind the correct source frame to the dynamic texture automatically. Latching would be automatically scoped to the duration of the rAF callback. My other comment is that the API currently doesn't support Workers but it should. Perhaps the best way to do that would be an OffscreenVideo proxy object similar to OffscreenCanvas (but without the constructor). Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Mon, Jan 4, 2016 at 4:54 PM, Robert O'Callahanwrote: > On Tue, Jan 5, 2016 at 1:18 PM, Jonas Sicking wrote: >> >> A big problem is sticking HTML/CSS content into WebGL is that WebGL >> effectively enables reading pixel data through custom shaders and >> timing attacks. > > > If you read > https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > carefully I think it's designed to prevent timing attacks by forbidding > shader control flow from depending on security-sensitive texture data. > > It's hard for me to judge how implementable it is, but in principle it > should be doable. It requires analysis of shader code. That's really cool if it's possible to do! We'd have to be careful to make sure that security-sensitive texture data can't affect paint performance in other ways. For example if such data can be used (directly or indirectly) as alpha channel of a surface, you could probably stick a slow-rendering texture behind it and that way check for certain values. Also, if any of the GLSL functions have performance which depends on the values passed to it, then we'd have to forbid passing security-sensitive texture data to such a function. So for example if abs(x) is faster for positive numbers than negative numbers, if sin(0) is faster than sin(x!=0), or if all([f, f, f, f]) is faster than all([t, t, t, t]) we have to forbid passing security-sensitive texture data to those functions. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 10:46 AM, Kearwood "Kip" Gilbert < kgilb...@mozilla.com> wrote: > In WebVR, we often present UI as a Head's Up Display (HUD) that floats > in front of the user. Additionally, we often wish to show 2d graphics, > video, and CSS animations as a texture in 3d scenes. Creating these > textures is something that CSS and HTML are great at. > > Unfortunately, I am not aware of an easy and efficient way to capture an > animated of an interactive HTML Element and bring it into the WebGL > context. A "moz-element" -like API would be useful here. > > Perhaps we could solve this by implementing and extending the proposed > WEBGL_dynamic_texture extension: > > > https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ This proposal seems unnecessarily complex. Is there a way for me to send feedback? Essentially, we would extend the same API but allow the WDTStream > interface to apply to more HTML elements, not just HTMLCanvasElement, > HTMLImageElement, or HTMLVideoElement. > > We would also need to implement WEBGL_security_sensitive_resources to > enforce the security model: > > > https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ I wish I'd known about this proposal earlier! This looks pretty good, though I'd always thought this would be too complicated to spec and implement to be practical. Glad to be wrong! Although I think we should get as much feedback as possible on this in case of hidden gotchas. Does this sound like a good idea? I feel that this is something that > all WebGL developers would want, as it would make building front-ends > for games much easier. > Yes, I think together these would be very useful. If others feel the same, I would also like to follow up with a proposal > to make the captured HTML elements interactive through use of an > explicit "pick buffer" added to canvases. > How would that work? Being able to synthesize mouse (touch?) events in HTML elements would add another set of issues. I assume the idea of mixing CSS 3D-transformed elements into a WebGL scene has been rejected for some reason? Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: "-Moz-Element" -like API by extending WEBGL_dynamic_texture
On Tue, Jan 5, 2016 at 2:38 PM, Jeff Gilbertwrote: > > Essentially, we would extend the same API but allow the WDTStream > >> interface to apply to more HTML elements, not just HTMLCanvasElement, > >> HTMLImageElement, or HTMLVideoElement. > >> > >> We would also need to implement WEBGL_security_sensitive_resources to > >> enforce the security model: > >> > >> > >> > https://www.khronos.org/registry/webgl/extensions/WEBGL_security_sensitive_resources/ > > > > > > I wish I'd known about this proposal earlier! This looks pretty good, > > though I'd always thought this would be too complicated to spec and > > implement to be practical. Glad to be wrong! Although I think we should > get > > as much feedback as possible on this in case of hidden gotchas. > > Historically in our investigations of this, it seemed very very hard > to guarantee a lack of timing vectors, even just for arithmetic. > (Particularly since we're handing the sources to the driver, which > will try to optimize away as much as it can) > Feedback from GPU vendors would be key here, I think. I'd like to hear that before declaring it dead. If they already did --- RIP :-). Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform