Re: [webkit-dev] enhancement - html hole element
Julien Isorce wrote: A typical scenario when running WebKit in an embedded environment like a TV or STB is to punch through graphics to the video plane. At Samsung Research UK we have developed a new hole element that acts much like a canvas to - Expose a rectangular hole in a web page. - Support a mechanism to retrieve the position and size of the hole from JavaScript whenever its dimensions or location change. Hi Julien! Your usecase definitely resonates with me, so here you have a couple of considerations from our (Collabora) experience. The problem we had to face for a customer was to show a 1440x900 @ 60fps captured video stream with a custom HTML UI on top (with animations and the usual stuff) in real time on a rather slow platform. At first we took the most web-friendly route we could think of, by simply modifying the out-of-tree Clutter port of WebKit to understand a special v4l2:// URL scheme for video elements and then piping the video stream to the video texture through GL (thus not using any special hardware overlay). Unfortunately this was not enough to achieve 60fps and we then deployed a solution which allowed us to make good use of the hardware overlays provided by the platform: we basically moved the video out of WebKit and into its own hardware overlay, put WebKit-Clutter in its own overlay on top of the video and then enabled its transparent-page support. This way we just needed to set background:transparent on the html element to see through the page to the video stream. We then injected a trivial JS API to give pages the chance to control the position and size of the video overlay: with the help of requestAnimationFrame() and some transparent placeholder div this gave us the ability to implement fancy zoom-in/out animations synchronized with visible page elements with surprisingly little code. It also had the benefit of decoupling the 60fps video stream from the web engine, which was then set to run at 30fps to better fit the graphic hardware capabilities, noticeably reducing janking during animations. In the future I see this usecase better served by having WebKit run as a Wayland nested compositor, which would allow the main compositor to transparently decide to use hardware overlays, GL composition or some other platform-specific mechanism on a frame-per-frame basis. In the context of the GTK+ port, by using the waylandsink GStreamer component it would be possible to push video frames directly to the hardware (potentially from hardware decoders) without any copy. I hope you'll find this useful! -- Emanuele Aina www.collabora.com ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] enhancement - html hole element
Hi, Thx for your replies! We also had the solution based on plugin. But we prefer to not use plugins if possible. *Some details about our hole element implementation: We have created a dedicated TextureMapperHoleBackingStore (which actually could be integrated into TextureMapperSurfaceBackingStore). So we call this GraphicsLayer::setContentsToPlatformLayer function from RenderLayerBacking.cpp. I.e, we have put a new case: “if render().isHole()” at the beginning of the existing black: “if render().isEmbeddedObject () else if render(). isVideo() else if render().isAcceleratedCanvas()” in “RenderLayerBacking::updateGraphicsLayerConfiguration()”. Note that the canvas case happens only if WEB_GL or ACCELERATED_2D_CANVAS is enabled so moving the hole to video element is probably a better option. We have validated it on upstream WebKitGtk and WebKitEfl, and the code we put in WebKit is the same for both platforms. * It makes sense to move this work to video. So instead of having a new tag, we suggest a new style value: video style=”background=-webkit-hole;” It will also simplify the diff a lot. Comments are welcome. On 19 September 2014 21:43, José Dapena Paz jdap...@igalia.com wrote: El jue, 18-09-2014 a las 12:07 -0700, Simon Fraser escribió: I don’t think it’s appropriate to add hole to WebKit. hole is really just adapting to a limitation of the platform’s compositing architecture. More powerful hardware like computers and smartphones are able to composite video above and below web content, which is achieved in WebCore via the accelerated compositing code path. I don’t think it’s appropriate to burden the platform with an element that only applies on power-limited hardware. The right solution for a WebKit implementation would be to solve this via the accelerated composting code. Just use a video or object in your markup, then implement the hole-punching via the GraphicsLayer subsystem. +1. Implementing punching hole with the integration of GraphicsLayer and MediaPlayer works well for the videos. Same for plugins. I see the punching hole is platform specific. No need for a new HTML tag. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] enhancement - html hole element
El jue, 18-09-2014 a las 12:07 -0700, Simon Fraser escribió: I don’t think it’s appropriate to add hole to WebKit. hole is really just adapting to a limitation of the platform’s compositing architecture. More powerful hardware like computers and smartphones are able to composite video above and below web content, which is achieved in WebCore via the accelerated compositing code path. I don’t think it’s appropriate to burden the platform with an element that only applies on power-limited hardware. The right solution for a WebKit implementation would be to solve this via the accelerated composting code. Just use a video or object in your markup, then implement the hole-punching via the GraphicsLayer subsystem. +1. Implementing punching hole with the integration of GraphicsLayer and MediaPlayer works well for the videos. Same for plugins. I see the punching hole is platform specific. No need for a new HTML tag. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] enhancement - html hole element
Hi, A typical scenario when running WebKit in an embedded environment like a TV or STB is to punch through graphics to the video plane. At Samsung Research UK we have developed a new hole element that acts much like a canvas to - Expose a rectangular hole in a web page. - Support a mechanism to retrieve the position and size of the hole from JavaScript whenever its dimensions or location change. We realise that there is currently no W3 specification for this element, but since our solution works we wanted to open the discussion to see if there is any wider interest, and if there is, to start the process of drafting a specification. Some details: - Principle: A hole is a rectangular area that behaves pretty much like a canvas, but everything in its area is transparent. Every layer behind the hole is transparent in this rectangle. In JavaScript you can register a handler which is notified when the hole position or size changes. - Use case: On TV this is useful to see the channel displayed behind the browser. Retrieving the hole's position and size from JavaScript allows to rescale the TV channel plane to match the hole area whenever the hole changes. Assume in JavaScript we have an object to control the TV channel plane. This use case applies in many interactive TV applications, so it could be useful for a lot of people. - Interfaces: We have defined a new tag hole which is similar to the canvas tag. We have defined a geometrychanged JS event which is sent when the hole dimensions change. This event contains the hole width, height and position in screen coordinates. - Hole in canvas: A hole tag has been defined to be less intrusive and having the feature in separate places in the code. But we now think it may be better to actually put the hole capabilities under the canvas. For example canvas style=background:hole; Note that setting it as transparent is not enough to have a hole as the layers behind the canvas won't be necessarily transparent, and not only in its area. The whole point of the hole capability is to make transparent every layers behind the hole and only in a specific rectangle. - Next steps: I would be very interested to hear if there are any other developers who need this functionality. If so, let's discuss the best way to share it, or possibly do it a better way. Comments are welcome. Julien ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] enhancement - html hole element
On Sep 18, 2014, at 9:22 AM, Julien Isorce j.iso...@samsung.com wrote: Hi, A typical scenario when running WebKit in an embedded environment like a TV or STB is to punch through graphics to the video plane. At Samsung Research UK we have developed a new hole element that acts much like a canvas to - Expose a rectangular hole in a web page. - Support a mechanism to retrieve the position and size of the hole from JavaScript whenever its dimensions or location change. We realise that there is currently no W3 specification for this element, but since our solution works we wanted to open the discussion to see if there is any wider interest, and if there is, to start the process of drafting a specification. Some details: - Principle: A hole is a rectangular area that behaves pretty much like a canvas, but everything in its area is transparent. Every layer behind the hole is transparent in this rectangle. In JavaScript you can register a handler which is notified when the hole position or size changes. - Use case: On TV this is useful to see the channel displayed behind the browser. Retrieving the hole's position and size from JavaScript allows to rescale the TV channel plane to match the hole area whenever the hole changes. Assume in JavaScript we have an object to control the TV channel plane. This use case applies in many interactive TV applications, so it could be useful for a lot of people. - Interfaces: We have defined a new tag hole which is similar to the canvas tag. We have defined a geometrychanged JS event which is sent when the hole dimensions change. This event contains the hole width, height and position in screen coordinates. - Hole in canvas: A hole tag has been defined to be less intrusive and having the feature in separate places in the code. But we now think it may be better to actually put the hole capabilities under the canvas. For example canvas style=background:hole; Note that setting it as transparent is not enough to have a hole as the layers behind the canvas won't be necessarily transparent, and not only in its area. The whole point of the hole capability is to make transparent every layers behind the hole and only in a specific rectangle. - Next steps: I would be very interested to hear if there are any other developers who need this functionality. If so, let's discuss the best way to share it, or possibly do it a better way. Comments are welcome. I don’t think it’s appropriate to add hole to WebKit. hole is really just adapting to a limitation of the platform’s compositing architecture. More powerful hardware like computers and smartphones are able to composite video above and below web content, which is achieved in WebCore via the accelerated compositing code path. I don’t think it’s appropriate to burden the platform with an element that only applies on power-limited hardware. The right solution for a WebKit implementation would be to solve this via the accelerated composting code. Just use a video or object in your markup, then implement the hole-punching via the GraphicsLayer subsystem. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
