> https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&fullscreen > I'm guessing they can be milliseconds? yes sorry I forgot to change that, now it is easier to understand.
On Wed, Jan 20, 2016 at 5:09 PM, Joaquin Oltra Hernandez < [email protected]> wrote: > Thanks for the responses Federico & Peter. > > Federico, I fully agree that a video is not a replacement for data, but > they serve different purposes. Peter expanded on it better than I what I > could do, but basically data driven tests serve a great deal for > development purposes to assessing impact of changes since they give us a > controlled environment to see the impact of changes trying to improve > performance. > > The reality is a lot messier and hard to measure, and I find it greatly > useful to see real experiences on real devices, where the network > conditions vary a lot, and there are multiple applications/services > competing for CPU and bandwidth on the device, making the experience > completely different to what it is on the numbers that the performance > tests provide. > > Surely it is anecdotal data, but it is an anecdotal experience that a lot > of people probably experience, so I end up caring about it. > > --- > > Thanks Peter for the links and the thoughtful response. Great points and > advice. I didn't know about that dashboard, it is great! > > What are the units on the Y axis here for example? > > https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&fullscreen > I'm guessing they can be milliseconds? > > One last thing: We will not pickup things as CPU usage faking real devices > > but to do that, I think we need to look at another tool than WebPageTest > to > > do that correctly. Do we have one already? > > > Not that I know of, we were thinking about doing some searching to see what > we could find. It would be a great addition to have. > > Joaquin > > > On Wed, Jan 20, 2016 at 10:28 AM, Peter Hedenskog <[email protected]> > wrote: > > > I like videos because it's so easy for non technical people to understand > > the value of performance. For me it has been a really good way to show > and > > explain performance changes and what it looks like for the user. > > > > Picking a representative connection is hard and the good thing is I think > > we don't need to. Connections in the real world vary so much: You could > > have 3G but a really bad connection to the nearest station because > there's > > a lot of other phones connected etc. We have tests running with real > > connections 2G/3G from Bangalore and LTE in San Francisco, you can check > it > > out how it looks here: > > > > > https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=30&fullscreen > > I really like that graph because it shows how much variances you can have > > on the same connection type from the same location. Check out how it > looks > > like in Bangalore. > > > > When we throttle the connection for WebPageTest (or other tools) we don't > > get that type of variations, we just use hard coded values for a specific > > connection. And that's good, because it makes it possible for us to > measure > > improvements so that when we do fixes, we can easily see if it makes it > > better or worse. Picking a slow connection like 2G is good because then > it > > is easier to spot what impact we get from our changes. > > > > I like automated testing with real devices, we have automatic tests up > and > > running with a Iphone 6, Ipad Mini and Motorola G (using the ones > provided > > in WebPageTest) and it looks like this: > > > > > https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&fullscreen > > > > You can click on the each row in the table to only show one graph, that > > makes it easier to see what numbers we have. Click > > on Dulles_iPhone6_iPhone_6_iOS_9.render and you can see that the start > > rendering time (when something is first showed on the screen) is pretty > > stable for the Iphone 6 but if you > > choose Dulles_iPhone6_iPhone_6_iOS_9.SpeedIndex (SpeedIndex is a way of > > calculate when content within the viewport is visible for the end user) > > vary a lot. The thing is that we only have limited access to automated > > tests with real phones (we don't have anything setup for ourselves). So > we > > use it now to just get a feeling of what it looks like on real devices. > > > > Most IMPORTANT: When we measure performance it is important that we > always > > do it exactly the same way, so we try to limit the changes to as few as > > possible (=our code change). Using WebPageTest I think it's ok to use > > Mobile 2G Fast, emulating mobile using Chrome. When we run the automated > > tests in Jenkins we test each URL 5 times, it seems to be pretty stable. > > Our tests will not give us the same numbers as the users gets, but it > will > > give us constant numbers so that we can see that the changes we do impact > > the site. When we push these changes to production we can hopefully pick > > them up with our RUM metrics. > > > > One last thing: We will not pickup things as CPU usage faking real > devices > > but to do that, I think we need to look at another tool than WebPageTest > to > > do that correctly. Do we have one already? > > > > On Mon, Jan 18, 2016 at 1:20 PM, Federico Leva (Nemo) < > [email protected]> > > wrote: > > > > > I'm allergic to videos as replacement of data, but does your video > > > substantially differ from what tests give us, e.g. > > > http://www.webpagetest.org/video/view.php?id=160118_YK_HKT.1.0 ? > > > > > > It's easy to test multiple speeds (some tests still pending): > > > > > > http://www.webpagetest.org/video/compare.php?tests=160118_AZ_JH3,160118_VB_JGH,160118_X6_HM1,160118_KC_HM0,160118_YK_HKT,160118_2W_HKD > > > > > > Some of the tests show significant CPU usage after the "everything and > a > > > kitchen sink" ResourceLoader phase in which some stuff for > CentralNotice, > > > Gather etc. etc. is loaded. > > > > > > Mainly, the issue is to pick representative connections. There is some > > > official data, some of which based on actual tests by probes, although > > it's > > > limited: > > > * > > > > > > https://ec.europa.eu/digital-agenda/en/news/quality-broadband-services-eu > > > (no mobile) > > > * > > > > > > https://ec.europa.eu/digital-agenda/en/news/use-commercial-mobile-networks-and-equipment-mission-critical-high-speed-broadband > > > (needs some parsing!) > > > > > > Nemo > > > > > > > > > _______________________________________________ > > > Wikitech-l mailing list > > > [email protected] > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > > _______________________________________________ > > Wikitech-l mailing list > > [email protected] > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > _______________________________________________ > Wikitech-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
