On 11/02/2016 11:38 AM, Valentin Gosu wrote:
TL;DR - Firefox does pretty well when compared to Chrome.
The Presto project is a Mozilla platform initiative that intends to look
into any performance differences between Firefox and other UserAgents in
order to highlight areas that we should look into improving and to clear
any prejudice that may be caused by FUD or past performance differences
that are no longer true.
Bug 1239709 seems related - it was opened in response to a recent
article [1] which shows us as having worse page-load performance than
chrome, particularly with flash involved.
Mark
[1]
http://www.cio.com/article/2974303/browsers/the-best-web-browser-of-2015-firefox-chrome-edge-ie-and-opera-compared.html
We've begun this project by doing a set of performance runs on
WebPageTest.com in order to compare Firefox's performance against Chrome on
the home pages of the Alexa 100 (the top 100 web sites worldwide). We've
made a visualization tool which plots all of the runs on a graph, so we may
easily compare them:
http://valenting.github.io/presto-plot/plot.html
- click on a domain link to see results for each site.
- you are able to view the results sorted or unsorted, or filter by the
browser version
- you can also compare metrics other than load time - such as speed
Index, number of DNS queries, etc
- you can also compare the browsers on several connectivity profiles
(Cable, 3G, Dialup)
- You can click on each individual point to go to the WPT run and view
the results in greater detail
---------------
Initial results:
The results consistently show that Firefox is faster than Chrome for both
of our major metrics (load time and speedIndex). On a majority of domains
Firefox is faster on both first loads and reloads. When we analyze 3G
connectivity runs, Firefox's speedup less substantial, and we encounter
additional domains where Chrome has an edge.
Dial up connectivity is a bit more difficult to analyze. Because of the
large size of most websites, browsers are unable to load the website within
5 minutes, or it might time out. Chrome seems to have more successful data
points, and faster loads on dial up, but it is difficult to reach a
conclusion based on a limited data set.
WPT also has Nightly builds - which default to e10s ON. The several data
points we have show similar performance to non-e10s builds (which is good,
considering Nightly builds have more checks and assertions)
-----------------------
Interesting metrics:
Load time - is measured as the time from the start of the initial
navigation until the beginning of the window load event (onload).
Speed Index - is the metric recommended PageSpeedTest.com. Speed index
measures how fast a page is displayed on screen. Details:
https://goo.gl/7ha6eE
Activity time - measured as the time from the start of the initial
navigation until there was 2 seconds of no network activity after Document
Complete. This will usually include any activity that is triggered by
javascript after the main page loads.
For most metrics a lower score is better (faster).
------------------
Error sources:
Websites may return different content depending on the UA string. While
this optimization makes sense for a lot of websites, in this situation it
is difficult to determine if the browser's performance or the website's
optimizations have more impact on the page load.
Since we do not log onto any sites, the data here for sites which rely on
logins to display full data (Facebook, etc) are not representative of most
users' experience and will generally have a different (much more heavily JS
and XHR-based) profile in reality.
For several data sets we may observe a certain spike in the data points –
runs that take 3x-7x times longer than usual. This may be due to a number
of reasons: a higher load on the network in the data center, a higher load
on the VMs on which the browsers are running, or the fact that websites
serve variable content – different images, different ads.
It may be that some of the page loads don't load the website completely, or
encounter errors. WPT also saves a screenshot of the page, along with data
on all of the resources it loads, so we may look at really fast loads to
make sure they are complete.
WPT has a maximum load time of 5 minutes. Any page load that takes longer
than that will be recorded as a 0 – so while it may seem to load faster,
that is not the case. Other errors may also be recorded as a 0.
WPT only loads one tab page at a time, whereas the load time on a user's
may be affected by his activity in other tabs. e10s certainly helps in this
area.
We only tested the performance of the landing page. It would be possible to
simulate user activity and navigation once the landing page is loaded, but
for this we'd need to write separate scripts for each domain we test.
Location may be an issue, as the RTT to the server or the CDN may vary. All
of the tests we ran were made on the WPT datacenter in Dulles VA.
------------------
The stats are merely informative. All of the data is is publicly available
to anyone who has an inclination towards statistics.
Setup:
webpagetest.meteor.com - an api that returns a list of IDs in JSON
form, representing all the WPT tests run for a given domain (source and API:
https://github.com/valenting/webpagetest )
plot.html - when you click a domain it gets all of the IDs for tests
run against that domains, then parses the results and plots the results.
(source: https://github.com/valenting/presto-plot )
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
