Re: [dev-servo] UI Workers
On Thu, Feb 19, 2015 at 10:25 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. You may be right, but it just seems pretty awful that you might accidentally stall the JS event loop because you touched the DOM and you didn't expect it to take so long (say, 30ms, dropping a few frames). The Servo team even said they want to look into making them async: https://news.ycombinator.com/item?id=9011767 However, I have no real evidence this will help anything. Eventually I'd love to see the Servo model in real-world browsers where things are running on separate threads. But that's, like, seriously long-term. On Fri, Feb 20, 2015 at 4:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Thanks, we'll take that into account. If the standards committees are taking a serious look into this, I would really recommend talking to one of the people at Facebook working on React Native. They spent years trying to get this kind of stuff working on the web and React Native is born out of problems they faced with getting 60 FPS. I think they would have tons of good feedback for you, for example like my paragraph above. I'm happy to help someone from that team get involved here if you want. (I'm sure you are well aware of the problems they would tell you, but they have some really good reasons for their solution) ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] UI Workers
Sorry, I may have missed the last part of your email where you say that you could provide touch positions as input too. This would be neat research and I'd like to see more details. Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Anyway, just trying to give feedback from a user. This is really important to me! On Thu, Feb 19, 2015 at 10:02 PM, James Long longs...@gmail.com wrote: I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
