Re: [dev-servo] UI Workers
On Mon, Feb 23, 2015 at 4:26 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Feb 23, 2015 at 4:01 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Tue, Feb 24, 2015 at 12:09 PM, Jonas Sicking jo...@sicking.cc wrote: I think this would fall over more often than not. Most developers will not write their code to be resilient in the face of being suspended for extended periods of time. Upon reopening they would likely display error dialogs, or updated version of whatever was saved. In fact, I would argue that it's directly harmful to keep even some of the app logic working. It could cause things like overwriting server state with whatever state was serialized into the saved copy. You missed the part about the jail. Sure, the jail would prevent server overwrites. But there's still problems like code behaving strange due to being suspended so long, for example by removing content from the screen that is out of date. Additionally the jail is more likely to cause error dialogs to appear due to failed network requests. I would also argue that keeping a lot of the page working by running scripts, is going to be even more confusing when that page is then unable to actually save anything due to the jail. Rather than nuking all the scripts and ensuring that the only thing the user can do is scroll around on the page. Having thought about this overnight... Ultimately I don't feel like I'm convincing anyone on this thread. And I definitely think this is the layout team's call and not mine. I think we're making a different judgement call about how good your average websites will be to use features like ServiceWorkers to cache content for offline usage, and cache the content that users actually want to be available. (I don't for example think that native apps are particularly good at this). And/or we're making different judgement calls about how important it is for users to have content available offline. I would be interested to hear what features that we are getting in return though. I.e. what features we gain by enabling the UIWorker to access the world outside of the DOM. Maybe those features are clearly more valuable than browser-provided offline capabilities and printing. Either way though, I am sad that in our pursuit of competing with native, it feels like we're giving up more and more of the advantages that we do have over native. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
What does it mean to save your complex web app for later viewing? I don't think there's a lot of overlap between sites that would use the functionality roc is proposing, and sites that make sense to save for later viewing. Gavin On Mon, Feb 23, 2015 at 10:36 AM, Jonas Sicking jo...@sicking.cc wrote: On Sun, Feb 22, 2015 at 3:45 AM, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Feb 20, 2015 at 9:11 PM, Jonas Sicking jo...@sicking.cc wrote: On Thu, Feb 19, 2015 at 6:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. There's two use-cases that I think argues against that. First off I'd like to enable saving a webpage for later viewing. Right now the best we can do is to save a pdf which means that all scrollable areas are lost. Saving a full page with all scripts is rarely going to work result in the user seeing the page they did when saving the page. What I'd like to be able to do is serialize the DOM, strip all script, save all linked stylesheets, images and media and change the references to them to use the correct relative URLs. However this would require also saving the UIWorker and enabling it to run in a reliable way. Otherwise the saved page won't have a working layout. Second, when printing, I'd like to enable the user to change page layout and margins and have that reflow the page and generate a correct layout. This means that we need to enable the UIWorker to work on the cloned document in a reliable way. I think both these usecases would work much better if we ensured that the UIWorker couldn't interact with the outside world. Your use-cases already fail today because many Web pages use scroll event handlers and JS custom layouts. UIWorkers won't make the problem any worse. I agree that it's not making the problem worse. However #1 is making the problem better, so unless we can address this, this becomes an argument for #1 over #2. Furthermore, I think facilitating expressive performant mobile Web apps is much more important than those use-cases, and preventing UIWorkers from interacting with the outside world (including, I assume, the application main thread) would severely limit their usefulness. Personally I think it's futile to try to save a complex app for later viewing and expect anything useful to happen unless the whole app is captured and able to operate offline. The lack of ability to save for later viewing is a big problem on mobile. The fact that native is so much better at retaining content to make it available later when the user is offline is one of the big reasons that the web is having a hard time competing with native. So I do think that this is an important problem to solve. / Jonas ___ dev-servo mailing list dev-se...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
On Thu, Feb 19, 2015 at 7:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. Does that mean also doing the painting/compositing on the thread that handles input? It'd be good to understand which thread the UIWorker proposal intends to run the worker. And if we expect any cross-thread communication to risk causing lag. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2015 09:12 PM, Jonas Sicking wrote: On Fri, Feb 20, 2015 at 8:11 AM, Julien Wajsberg jwajsb...@mozilla.com wrote: Le 20/02/2015 04:25, Robert O'Callahan a écrit : On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. In Firefox OS development, this is part of the biggest issues we have for very specific tasks. To be clear, we're not talking about making the whole DOM async, right? I.e. we're not talking about adding an async element.appendChild() or an async element.innerHTML? What we're specifically talking about are async functions for getting layout information? So async element.getComputedStyle() and element.getClientRects()? These are particular cases that have been suggested, yes. (I've heard suggestions for async innerHTML too, but I don't think that was in the context of Servo.) Either way, while some people have been dreaming about such APIs, we're not actually working on any, or planning to start working on them. HTH Ms2ger -BEGIN PGP SIGNATURE- iQEcBAEBAgAGBQJU57l9AAoJEOXgvIL+s8n2jQMH/1e+/4FTpq/r/7qTJ5MEQ0mL T1MbR+oVBr6xDCX4hu72ZCarmEC992coPnuMC4Y8NI4gD91XpAZhToFraGifXC4Z zEJaqYfUSH8YRMIbwqmH49j1aEQjLW/q0XwHfsE3M7V4+7P7BaccQhotSQQz5Ubo +SGtPbqfTmQr4FeZhXGCOGJAU76tBAdoKr0NanDh6FvuiIz/vB+ZVUSStsK2gwYw FbIdXBPiJLlbW+VQjSz4NDXLrLIfzZ7jb5t43Ad0ljMeKltNSh3X6ad8Wv6DTExM RHx6xGojSYhEvwelfbMhsxI2zCmNrxHoJ2sBSmnG+2kr5BDaUP1c/jneD5zH/Vk= =EYaK -END PGP SIGNATURE- ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
On Fri, Feb 20, 2015 at 3:20 AM, Jonas Sicking jo...@sicking.cc wrote: On Thu, Feb 19, 2015 at 7:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. Does that mean also doing the painting/compositing on the thread that handles input? Not necessarily, the important part is that when the user makes a gesture, an animation is updated with the event as input, and the layout or other properties are changed, all on the same thread. It can be actually painted on a separate thread. The state of the render tree just needs to be updated in sync with the user input. As Anne mentioned, when React Native comes out next month I would highly recommend looking deep into it. They have a lot of good reasons for choosing that architecture, and is something we could think about working towards (long-term). I know the team personally and I can get someone here in touch with them. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
Le 20/02/2015 04:25, Robert O'Callahan a écrit : On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. In Firefox OS development, this is part of the biggest issues we have for very specific tasks. I say 'biggest especially because it's really difficult to work around. And even when we work around we have other issues (see bug 1126391). I don't want to say this is more important than the initial issue you're trying to solve. But yes it's important. -- Julien signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
On Fri, Feb 20, 2015 at 8:11 AM, Julien Wajsberg jwajsb...@mozilla.com wrote: Le 20/02/2015 04:25, Robert O'Callahan a écrit : On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. In Firefox OS development, this is part of the biggest issues we have for very specific tasks. To be clear, we're not talking about making the whole DOM async, right? I.e. we're not talking about adding an async element.appendChild() or an async element.innerHTML? What we're specifically talking about are async functions for getting layout information? So async element.getComputedStyle() and element.getClientRects()? / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
On Thu, Feb 19, 2015 at 10:25 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. You may be right, but it just seems pretty awful that you might accidentally stall the JS event loop because you touched the DOM and you didn't expect it to take so long (say, 30ms, dropping a few frames). The Servo team even said they want to look into making them async: https://news.ycombinator.com/item?id=9011767 However, I have no real evidence this will help anything. Eventually I'd love to see the Servo model in real-world browsers where things are running on separate threads. But that's, like, seriously long-term. On Fri, Feb 20, 2015 at 4:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Thanks, we'll take that into account. If the standards committees are taking a serious look into this, I would really recommend talking to one of the people at Facebook working on React Native. They spent years trying to get this kind of stuff working on the web and React Native is born out of problems they faced with getting 60 FPS. I think they would have tons of good feedback for you, for example like my paragraph above. I'm happy to help someone from that team get involved here if you want. (I'm sure you are well aware of the problems they would tell you, but they have some really good reasons for their solution) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-servo mailing list dev-se...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [dev-servo] UI Workers
Sorry, I may have missed the last part of your email where you say that you could provide touch positions as input too. This would be neat research and I'd like to see more details. Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Anyway, just trying to give feedback from a user. This is really important to me! On Thu, Feb 19, 2015 at 10:02 PM, James Long longs...@gmail.com wrote: I'm not heavily involved in platform work, so take my input as a somewhat naïve outsider. I've been tracking this kind of stuff very closely, especially after I got to play with react native. The place where this is most important is mobile, and I don't think #2 is going to cut it. It's not just scroll position that we're after. We need to handle touch events and perform arbitrary animations based on the gesture of the user. So if the finger moved X pixels in the x direction, and Y in the y direction, native apps can very carefully apply any changes to the layout or properties they want. If we only allow scroll position to be taken into account, I can only imagine the hacks that are going to take place: making random elements secretly scroll in specific ways to apply animations (if that's even possible). Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. On Thu, Feb 19, 2015 at 9:27 PM, Robert O'Callahan rob...@ocallahan.org wrote: Last week in Sydney I spent a lot of time talking to Chrome devs about different approaches for 60fps effects in Web pages. There are three different kinds of approaches being discussed (so far): 1) Apple's animation-timeline proposal, which lets CSS animations use scroll position as an input instead of time. 2) UIWorker: some kind of JS worker that receives callbacks during composition; each callback can take inputs such as time and scroll position(s) as inputs and can update certain CSS properties (e.g. transforms, opacity) on elements that the compositor then uses. 3) Provide a way for pages to turn off async scrolling and make everything fast enough (and isolated enough) for pages to do 60fps updates from their main thread. All of these approaches have problems. Approach #1 is much more limited in its expressiveness than the alternatives. Approach #3 is more fragile and less composable than the alternatives --- sharing your main thread with any JS you don't control could cause jank. I like #2. It's strictly more powerful than #1 without the downsides of #3. Obvious question: how do we stop UIWorkers janking the compositor? We could give them a time budget (say 8ms). If a worker blows its budget, we notify it by sending it an event, we give up on it and composite anyway, and we run it separately from the compositor for a while. This requires an API design that lets UIWorkers still work, with some lag, when the compositor is not blocking on them, but that seems doable. Should UIWorkers have access to the full Worker API? It seems like there's no reason not to give them that. How should we explain the CSS effects of UIWorkers? A promising idea is to extend the Web Animations API to allow adding a new kind of animation effect to DOM elements --- a UIWorker-controlled effect. Essentially the UIWorker would then be responsible for computing the output of the timing function in each frame. The UIWorker could then animate *any* CSS property, though most property updates would require a round trip through main thread layout/rendering before they get rendered. One good thing about UIWorkers is extensibility. We can imagine providing touch input coordinates to UIWorkers to enable 60fps object dragging (with arbitrary effects like resistance, snapping, etc). UIWorkers could render to canvases: this would let you render VR with minimum latency, and let you render to canvases used by CSS masking for 60fps dissolves and clipping effects. If you really want to, you could go all Flipboard and render your entire UI to a canvas in the compositor --- if you keep hitting your deadlines. I like the idea of doing #2 before either #1 or #3. Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo
Re: [dev-servo] UI Workers
On Fri, Feb 20, 2015 at 4:02 PM, James Long longs...@gmail.com wrote: Personally I think what we *really* need to be working on is making all of the DOM APIs asynchronous. That's what Servo needs anyway. That's a step in the right direction for #3, and we can see much more we can get out of the main thread after that. I don't see how making DOM APIs asynchronous really helps you achieve #3. There are some specific cases where custom layouts need to repeatedly measure content, that currently rely on APIs that trigger multiple sequential synchronous reflows, which would be better served by less synchronous APIs. We have nebulous plans for addressing that (it's not as easy as make everything a Promise). But those aren't the main problems mobile Web apps face right now AFAIK. On Fri, Feb 20, 2015 at 4:11 PM, James Long longs...@gmail.com wrote: Note however that people in the native app world believe that it's very important that whatever applies the animations is run on the same thread that handles input. Otherwise it's very easy for animations to get out-of-sync. If you swipe a scroll view on iOS, and it has that momentum scroll, and you place your finger down it immediately stops where it is. I've talked to several engineers that try to do this kind of stuff with CSS animations but because they are async, it's not nearly as reliable as native. Native platforms fire the touch event, and you apply the animation update immediately in the same thread. Thanks, we'll take that into account. Anyway, just trying to give feedback from a user. This is really important to me! Thanks! Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
