[dev-servo] CSS Houdini meeting report
Last week I was in Sydney for the first face to face meeting of “CSS Houdini”, a new joint task force of W3C’s CSS Working Group and Technical Architecture Group. The goal is to “explain the magic” of CSS, make it extensible by providing low-level primitives that can be built upon to add new higher-level features to the platform without baking them into browsers. Participants were from Microsoft, Google, Mozilla, HP, W3C, Disruptive Innovations, Apple, Vivliostyle, Adobe, and jQuery Foundation. I have heard concerns that Google or Apple would just expose whatever Blink/WebKit’s internals currently are, and the rest of us implementers would be stuck with that. Since this work is happening at W3C and participants include people working on competing engines, I am confident this won’t happen. I will also be watching for design decisions that could be incompatible with Servo’s goals (e.g. parallel layout, layout concurrent with script, etc.) About the task force: Mailing list: https://lists.w3.org/Archives/Public/public-houdini/ IRC: #houdini @ irc.w3.org Wiki: https://wiki.css-houdini.org/ Draft specifications: http://dev.w3.org/houdini/ Source repository: https://hg.css-houdini.org/drafts Read/write mirror repository (pull requests work): https://github.com/w3c/css-houdini-drafts Formatted minutes of these discussions should be published soon. In the meantime, the raw IRC logs are at: http://log.csswg.org/irc.w3.org/houdini/2015-02-06/ http://log.csswg.org/irc.w3.org/houdini/2015-02-07/ The group agreed on its own scope, and on starting new 8 specifications. I described them below based on my the minutes and my own recollection of the discussions. If you want to discussed the proposals themselves (as opposed to their implications for Servo), please do so on the public-houd...@w3.org mailing list. (Subscription info is at the URL above.) * Fragment Tree Expose read-only information to script about the fragment tree, which is roughly what implementations call the render tree, frame tree, or flow tree. This includes anonymous fragments, that are not generated directly by any DOM element. This was previously discussed as a Box Tree, but was renamed since boxes (as defined in CSS specs) don’t have geometry, only fragments (again per spec, after fragmentation across lines, columns, regions or pages) do. * CSS Parser API Expose various pieces of the CSS parser: selectors, rules, complete stylesheets, colors, gradients, unknown properties, etc. Web authors are often doing this ad-hoc with regular expressions, which is painful and hard to get right. The shape and API of returned objects (representing tokens or an AST or higher-level values) is yet to be determined. * CSS Properties and Values Extensions Currently, CSS implementations throw away (per spec) rules or declarations they don’t understand. The error recovery and fallback allow for some forward-compatibility, but this is somewhat limited. We now also have CSS Custom Properties, but they’re namespaced with the -- prefix. This proposes adding APIs enabling script to register new at-rules and properties with a details about their behavior and callbacks for parsing, as well as new values for existing types (e.g color, length, image, …), thus enabling polyfills. * Font Metrics Extensions Some desired features like drop caps a.k.a. initial letters require various measurements from fonts such as position of the baseline, height of ascenders and descenders, etc. This proposes APIs to determine which font is actually used for a given piece of text in the document (since the `font-family` property accepts a list of family names) and query such measurements. * Custom Paint This proposes adding callbacks into script during painting, to enable various graphical effects. One example given was an animated procedural background image of ripples, as if on a fluid surface, that propagate from a disturbance point and bounce against the borders of the element. * CSS Custom Layout This propose adding the ability to define new layout modes (like Flexbox is a layout mode) in script, with callbacks for the various phases of layout. These modes would probably be opted into through new values of the 'display' property. * CSS Scroll Extensions This proposes adding hooks to control scrolling (e.g. for snap points), and what scrolling should block on (e.g. slow scroll event handlers). * CSS Async Style As far as I understand, this proposes adding control on what phases of rendering of what parts of the document can be deferred or (de)prioritized. -- Simon Sapin ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
On Thu, Feb 19, 2015 at 10:25 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. You may be right, but it just seems pretty awful that you might accidentally stall the JS event loop because you touched the DOM and you didn't expect it to take so long (say, 30ms, dropping a few frames). The Servo team even said they want to look into making them async: https://news.ycombinator.com/item?id=9011767 However, I have no real evidence this will help anything. Eventually I'd love to see the Servo model in real-world browsers where things are running on separate threads. But that's, like, seriously long-term. On Fri, Feb 20, 2015 at 4:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Thanks, we'll take that into account. If the standards committees are taking a serious look into this, I would really recommend talking to one of the people at Facebook working on React Native. They spent years trying to get this kind of stuff working on the web and React Native is born out of problems they faced with getting 60 FPS. I think they would have tons of good feedback for you, for example like my paragraph above. I'm happy to help someone from that team get involved here if you want. (I'm sure you are well aware of the problems they would tell you, but they have some really good reasons for their solution) ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
[dev-servo] UI Workers
Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
Sorry, I may have missed the last part of your email where you say that you could provide touch positions as input too. This would be neat research and I'd like to see more details. Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Anyway, just trying to give feedback from a user. This is really important to me! On Thu, Feb 19, 2015 at 10:02 PM, James Long longs...@gmail.com wrote: I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
