On Wed, Jan 20, 2016 at 1:28 AM, Peter Hedenskog <pe...@wikimedia.org> wrote:
> I like videos because it's so easy for non technical people to understand > the value of performance. For me it has been a really good way to show and > explain performance changes and what it looks like for the user. > Linked to this, for those who might question or not understand performance work, people may want to compare the existing mobile experience on http://en.m.wikipedia.org/wiki/Barack_Obama with http://future-wikipedia.wmflabs.org/wiki/Barack_Obama and http://reading-web-research.wmflabs.org/wiki/Wikipedia on a real 2G connection. I'd be interested if anyone hasn't got time to record their experience to document it as our load times vary around the world depending on how far you are from a data centre among other reasons. > > Picking a representative connection is hard and the good thing is I think > we don't need to. Connections in the real world vary so much: You could > have 3G but a really bad connection to the nearest station because there's > a lot of other phones connected etc. We have tests running with real > connections 2G/3G from Bangalore and LTE in San Francisco, you can check it > out how it looks here: > > https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=30&fullscreen > I really like that graph because it shows how much variances you can have > on the same connection type from the same location. Check out how it looks > like in Bangalore. > > When we throttle the connection for WebPageTest (or other tools) we don't > get that type of variations, we just use hard coded values for a specific > connection. And that's good, because it makes it possible for us to measure > improvements so that when we do fixes, we can easily see if it makes it > better or worse. Picking a slow connection like 2G is good because then it > is easier to spot what impact we get from our changes. > > I like automated testing with real devices, we have automatic tests up and > running with a Iphone 6, Ipad Mini and Motorola G (using the ones provided > in WebPageTest) and it looks like this: > > https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&fullscreen > > You can click on the each row in the table to only show one graph, that > makes it easier to see what numbers we have. Click > on Dulles_iPhone6_iPhone_6_iOS_9.render and you can see that the start > rendering time (when something is first showed on the screen) is pretty > stable for the Iphone 6 but if you > choose Dulles_iPhone6_iPhone_6_iOS_9.SpeedIndex (SpeedIndex is a way of > calculate when content within the viewport is visible for the end user) > vary a lot. The thing is that we only have limited access to automated > tests with real phones (we don't have anything setup for ourselves). So we > use it now to just get a feeling of what it looks like on real devices. > > Most IMPORTANT: When we measure performance it is important that we always > do it exactly the same way, so we try to limit the changes to as few as > possible (=our code change). Using WebPageTest I think it's ok to use > Mobile 2G Fast, emulating mobile using Chrome. When we run the automated > tests in Jenkins we test each URL 5 times, it seems to be pretty stable. > Our tests will not give us the same numbers as the users gets, but it will > give us constant numbers so that we can see that the changes we do impact > the site. When we push these changes to production we can hopefully pick > them up with our RUM metrics. > > One last thing: We will not pickup things as CPU usage faking real devices > but to do that, I think we need to look at another tool than WebPageTest to > do that correctly. Do we have one already? > > On Mon, Jan 18, 2016 at 1:20 PM, Federico Leva (Nemo) <nemow...@gmail.com> > wrote: > > > I'm allergic to videos as replacement of data, but does your video > > substantially differ from what tests give us, e.g. > > http://www.webpagetest.org/video/view.php?id=160118_YK_HKT.1.0 ? > > > > It's easy to test multiple speeds (some tests still pending): > > > http://www.webpagetest.org/video/compare.php?tests=160118_AZ_JH3,160118_VB_JGH,160118_X6_HM1,160118_KC_HM0,160118_YK_HKT,160118_2W_HKD > > > > Some of the tests show significant CPU usage after the "everything and a > > kitchen sink" ResourceLoader phase in which some stuff for CentralNotice, > > Gather etc. etc. is loaded. > > > > Mainly, the issue is to pick representative connections. There is some > > official data, some of which based on actual tests by probes, although > it's > > limited: > > * > > > https://ec.europa.eu/digital-agenda/en/news/quality-broadband-services-eu > > (no mobile) > > * > > > https://ec.europa.eu/digital-agenda/en/news/use-commercial-mobile-networks-and-equipment-mission-critical-high-speed-broadband > > (needs some parsing!) > > > > Nemo > > > > > > _______________________________________________ > > Wikitech-l mailing list > > Wikitech-l@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
