Re: [whatwg] video and acceleration
On Wed, 29 Apr 2009, Simon Fraser wrote: Taking the video full-screen is an approach that makes a lot of sense for mobile devices. It's unfortunate that the spec shies away from the full-screen issue. The spec doesn't really shy away from it; it's just that the spec isn't the right place for it. There are two options as I see it: video-specific full page as you describe here, which is easily provided by a user-agent specific user interface (e.g. a button that appears when you hover on the video), and the general full-screen mechanism along with media-specific style sheets (e.g. F11 and media=projection). If the spec does say something about performance of video, I think it should be no more than a note that performance may differ across browsers, and can be affected in various ways that may be non-obvious to the page author, related to the layout and styling of the video and other elements on the page. I don't think such a note wold really be of much help to authors, so I haven't included it. I agree with the rest of your comments. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] video and acceleration
[Speaking with my Apple hat on.] We agree that upgrading the hardware is not an acceptable answer to this question. Many web-enabled devices have much less CPU power than the average laptop or desktop (think mobile devices), but many have quite capable GPUs. On desktop machines too, hardware acceleration of video playback is essential for optimum performance, and for acceptable CPU usage with large videos in non-trivial content (e.g. content with CSS effects on the video, or lots of expensive-to-render content under or over the video). Ignoring rotations, overlays etc. in order to get reasonable performance is an option, but should be considered a bug in the implementation. You could also imagine an attribute on the video element akin to the wmode attribute for plug-ins, which allow the page author to opt-in to faster rendering, with the expectation that the visual rendering will suffer. However, we feel strongly that the video spec should not be encumbered by an attribute of this kind. Taking the video full-screen is an approach that makes a lot of sense for mobile devices. It's unfortunate that the spec shies away from the full-screen issue. In an ideal world, hardware acceleration of video playback would just work, and the spec would have to say no more about it. We believe it's possible to do hardware acceleration of video (and other animating web content) and preserve rendering in many cases. WebKit actually has some experimental code for this. I'm not sure how much an HTML spec will be able to say about triggers that may cause the video rendering to fall off the hardware- accelerated path, since many of those triggers will be CSS-related, and implementation details will differ between browsers. From our experience in WebKit, those triggers may include: * clipping (via overflow on the video or an ancestor) * overlapping elements * blending (opacity) * being affected by transforms (via SVG, or CSS transforms on video or an ancestor) * CSS masks or reflections * CSS box decorations (border, background) If the spec does say something about performance of video, I think it should be no more than a note that performance may differ across browsers, and can be affected in various ways that may be non-obvious to the page author, related to the layout and styling of the video and other elements on the page. Simon On Apr 28, 2009, at 6:07 PM, Ian Fette wrote: Upgrade the hardware is not an acceptable answer. Video acceleration is meant to offload work from CPU (especially on constrained devices, e.g. mobile). You want to be able to do compositing on video card, so that you don't have to read the video out of the video card's memory, transfer it over the bus, to the CPU, do some transforms/overlays/..., and then send it back to the video card for display. Doing that absolutely kills framerate. As we (browsers) implement video I think a lot of us are starting with software rendering. Certainly we want to be able to do hardware acceleration at some point. Perhaps some things we will still be able to do in hardware, e.g. overlays of HTML or certain transforms (if the video device supports saying take this, translate it, and composite and the rendering engine only needs geometry data.) Other things we might not be able to do in hardware (e.g. if you have transparent flash video on top, and Flash wants to know what pixels are underneath it, then we would have to read that data off of the video card, send it to CPU, ...) I think what would be helpful is for browsers who are implementing video with hardware acceleration to publish information on what would make them fall back to software rendering. If it turns out that list is roughly similar across implementations, perhaps it could be added as a note in the spec that doing the following certain things may cause performance implications. We're probably not ready to do that yet given that we don't have enough implementation experience, but that would be my suggestion for how to move forward. -Ian On Tue, Apr 28, 2009 at 5:59 PM, Ian Hickson i...@hixie.ch wrote: On Sat, 28 Mar 2009, Benjamin M. Schwartz wrote: The video tag has great potential to be useful on low-powered computers and computing devices, where current internet video streaming solutions (such as Adobe's Flash) are too computationally expensive. My personal experience is with OLPC XO-1, on which Flash (and Gnash) are terribly slow for any purpose, but Theora+Vorbis playback is quite smooth at reasonable resolutions and bitrates. The video standard allows arbitrary manipulations of the video stream within the HTML renderer. To permit this, the initial implementations (such as the one in Firefox 3.5) will perform all video decoding operations on the CPU, including the tremendously expensive YUV-RGB conversion and scaling. This is viable only for moderate
Re: [whatwg] video and acceleration
On Sat, 28 Mar 2009, Benjamin M. Schwartz wrote: The video tag has great potential to be useful on low-powered computers and computing devices, where current internet video streaming solutions (such as Adobe's Flash) are too computationally expensive. My personal experience is with OLPC XO-1, on which Flash (and Gnash) are terribly slow for any purpose, but Theora+Vorbis playback is quite smooth at reasonable resolutions and bitrates. The video standard allows arbitrary manipulations of the video stream within the HTML renderer. To permit this, the initial implementations (such as the one in Firefox 3.5) will perform all video decoding operations on the CPU, including the tremendously expensive YUV-RGB conversion and scaling. This is viable only for moderate resolutions and extremely fast processors. Recognizing this, the Firefox developers expect that the decoding process will eventually be accelerated. However, an accelerated implementation of the video spec inevitably requires a 3D GPU, in order to permit transparent video, blended overlays, and arbitrary rotations. Pure software playback of video looks like a slideshow on the XO, or any device with similar CPU power, achieving 1 or 2 fps. However, these devices typically have a 2D graphics chip that provides video overlay acceleration: 1-bit alpha, YUV-RGB, and simple scaling, all in special-purpose hardware. Using the overlay (via XVideo on Linux) allows smooth, full-speed playback. THE QUESTION: What is the recommended way to handle the video tag on such hardware? Upgrade the hardware. There are two obvious solutions: 0. Implement the spec, and just let it be really slow. 1. Attempt to approximate the correct behavior, given the limitations of the hardware. Make the video appear where it's supposed to appear, and use the 1-bit alpha (dithered?) to blend static items over it. Ignore transparency of the video. Ignore rotations, etc. 2. Ignore the HTML context. Show the video in manners more suitable to the user (e.g. full-screen or in an independent resizable window). Which is preferable? Is it worth specifying a preferred behavior? From HTML's point of view, all are acceptable. From the user's point of view, 1 and 2 are preferable, probably at the user's option. I don't know what else to tell you. :-) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] video and acceleration
Upgrade the hardware is not an acceptable answer. Video acceleration is meant to offload work from CPU (especially on constrained devices, e.g. mobile). You want to be able to do compositing on video card, so that you don't have to read the video out of the video card's memory, transfer it over the bus, to the CPU, do some transforms/overlays/..., and then send it back to the video card for display. Doing that absolutely kills framerate. As we (browsers) implement video I think a lot of us are starting with software rendering. Certainly we want to be able to do hardware acceleration at some point. Perhaps some things we will still be able to do in hardware, e.g. overlays of HTML or certain transforms (if the video device supports saying take this, translate it, and composite and the rendering engine only needs geometry data.) Other things we might not be able to do in hardware (e.g. if you have transparent flash video on top, and Flash wants to know what pixels are underneath it, then we would have to read that data off of the video card, send it to CPU, ...) I think what would be helpful is for browsers who are implementing video with hardware acceleration to publish information on what would make them fall back to software rendering. If it turns out that list is roughly similar across implementations, perhaps it could be added as a note in the spec that doing the following certain things may cause performance implications. We're probably not ready to do that yet given that we don't have enough implementation experience, but that would be my suggestion for how to move forward. -Ian On Tue, Apr 28, 2009 at 5:59 PM, Ian Hickson i...@hixie.ch wrote: On Sat, 28 Mar 2009, Benjamin M. Schwartz wrote: The video tag has great potential to be useful on low-powered computers and computing devices, where current internet video streaming solutions (such as Adobe's Flash) are too computationally expensive. My personal experience is with OLPC XO-1, on which Flash (and Gnash) are terribly slow for any purpose, but Theora+Vorbis playback is quite smooth at reasonable resolutions and bitrates. The video standard allows arbitrary manipulations of the video stream within the HTML renderer. To permit this, the initial implementations (such as the one in Firefox 3.5) will perform all video decoding operations on the CPU, including the tremendously expensive YUV-RGB conversion and scaling. This is viable only for moderate resolutions and extremely fast processors. Recognizing this, the Firefox developers expect that the decoding process will eventually be accelerated. However, an accelerated implementation of the video spec inevitably requires a 3D GPU, in order to permit transparent video, blended overlays, and arbitrary rotations. Pure software playback of video looks like a slideshow on the XO, or any device with similar CPU power, achieving 1 or 2 fps. However, these devices typically have a 2D graphics chip that provides video overlay acceleration: 1-bit alpha, YUV-RGB, and simple scaling, all in special-purpose hardware. Using the overlay (via XVideo on Linux) allows smooth, full-speed playback. THE QUESTION: What is the recommended way to handle the video tag on such hardware? Upgrade the hardware. There are two obvious solutions: 0. Implement the spec, and just let it be really slow. 1. Attempt to approximate the correct behavior, given the limitations of the hardware. Make the video appear where it's supposed to appear, and use the 1-bit alpha (dithered?) to blend static items over it. Ignore transparency of the video. Ignore rotations, etc. 2. Ignore the HTML context. Show the video in manners more suitable to the user (e.g. full-screen or in an independent resizable window). Which is preferable? Is it worth specifying a preferred behavior? From HTML's point of view, all are acceptable. From the user's point of view, 1 and 2 are preferable, probably at the user's option. I don't know what else to tell you. :-) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] video and acceleration
On Tue, 28 Apr 2009, Ian Fette wrote: I think what would be helpful is for browsers who are implementing video with hardware acceleration to publish information on what would make them fall back to software rendering. If it turns out that list is roughly similar across implementations, perhaps it could be added as a note in the spec that doing the following certain things may cause performance implications. We're probably not ready to do that yet given that we don't have enough implementation experience, but that would be my suggestion for how to move forward. That seems reasonable. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] video and acceleration
On Sat, 28 Mar 2009 05:57:35 +0100, Benjamin M. Schwartz bmsch...@fas.harvard.edu wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dear What, Short: video won't work on slow devices. Help! Long: The video tag has great potential to be useful on low-powered computers and computing devices, where current internet video streaming solutions (such as Adobe's Flash) are too computationally expensive. My personal experience is with OLPC XO-1*, on which Flash (and Gnash) are terribly slow for any purpose, but Theora+Vorbis playback is quite smooth at reasonable resolutions and bitrates. The video standard allows arbitrary manipulations of the video stream within the HTML renderer. To permit this, the initial implementations (such as the one in Firefox 3.5) will perform all video decoding operations on the CPU, including the tremendously expensive YUV-RGB conversion and scaling. This is viable only for moderate resolutions and extremely fast processors. Recognizing this, the Firefox developers expect that the decoding process will eventually be accelerated. However, an accelerated implementation of the video spec inevitably requires a 3D GPU, in order to permit transparent video, blended overlays, and arbitrary rotations. Pure software playback of video looks like a slideshow on the XO, or any device with similar CPU power, achieving 1 or 2 fps. However, these devices typically have a 2D graphics chip that provides video overlay acceleration: 1-bit alpha, YUV-RGB, and simple scaling, all in special-purpose hardware.** Using the overlay (via XVideo on Linux) allows smooth, full-speed playback. THE QUESTION: What is the recommended way to handle the video tag on such hardware? There are two obvious solutions: 0. Implement the spec, and just let it be really slow. 1. Attempt to approximate the correct behavior, given the limitations of the hardware. Make the video appear where it's supposed to appear, and use the 1-bit alpha (dithered?) to blend static items over it. Ignore transparency of the video. Ignore rotations, etc. 2. Ignore the HTML context. Show the video in manners more suitable to the user (e.g. full-screen or in an independent resizable window). Which is preferable? Is it worth specifying a preferred behavior? In the typical case a simple hardware overlay correctly positioned could be used, but there will always be a need for a software fallback when rotation, filters, etc are used. Like Robert O'Callahan said, a user agent would need to detect when it is safe to use hardware acceleration and use it only then. If there is something that could be changed in the spec to make things a bit easier for user agents it might be an overlay attribute, just like SVG has: http://www.w3.org/TR/SVGTiny12/multimedia.html#compositingBehaviorAttribute I'm not convinced such an attribute would help, just pointing it out here... -- Philip Jägenstedt Opera Software
Re: [whatwg] video and acceleration
On Sat, Mar 28, 2009 at 5:57 PM, Benjamin M. Schwartz bmsch...@fas.harvard.edu wrote: What is the recommended way to handle the video tag on such hardware? There are two obvious solutions: 0. Implement the spec, and just let it be really slow. 1. Attempt to approximate the correct behavior, given the limitations of the hardware. Make the video appear where it's supposed to appear, and use the 1-bit alpha (dithered?) to blend static items over it. Ignore transparency of the video. Ignore rotations, etc. 2. Ignore the HTML context. Show the video in manners more suitable to the user (e.g. full-screen or in an independent resizable window). 3. Add an optimized path to detect when the YUV overlay can be used to give the correct rendering, and use it in that case. This is not a spec issue, it's entirely an implementation issue. You're welcome to file a bug on that at http://bugzilla.mozilla.org; I can explain there more details about how it could be done, but it's unlikely to happen anytime soon unless a volunteer contributes the code. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
