I tried to test my hypothesis using Flame with a build from today (from taskcluster, flame-kk-eng) testing test-startup-limit app (no changes) and FM app (minimal changes):
Here are my results: https://pastebin.mozilla.org/8848224 The bottom line is that I did not reproduce your results for Flame, 319mb for FM. I used Gecko from today and tested FM and test-startup-limit from 2.2 against 2.5. So, one should read my read my results as "if we exclude Gecko changes, that's what should have happened". My results, if anything can be significant, indicate: 1) fullyLoaded improvement in FM on both 319/512 between 2.2 and 2.5 2) visuallyLoaded no change to minimal improvement in FM on both 319/512 between 2.2 and 2.5 3) No noticable performance change between 2.2 and 2.5 in test-startup-limit 4) Potentially a minor memory win in test-startup-limit in 319 scenario 5) Significant memory win in FM between 2.2 and 2.5 in both 319 and 512 scenarios Based on that, I would say that we might be seeing Gecko regression because your results for FM differ significantly from mine and the only change is that you used 2.2 gecko while I used 2.5 gecko. Where full tests show 175ms regression in visuallyLoaded and 140kb regression in memory usage, my results show 140ms improvement in visuallyLoaded and 850kb improvement in memory usage. Also, while your results for 2.5 match mine, your results for FM 2.2 show significant difference that can (only?) be attributed to Gecko changes: Your results (Gecko 2.2 and FM 2.2): FM v2.2 cold launch: 604ms FM v2.2 USS: 10.37MB My results (Gecko 2.5 an FM 2.2): FM v2.2 cold launch: 782ms (visuallyLoaded, median) FM v2.2 USS: 11.31MB (USS, median) The consistency in mine and your results for 2.5 for FM make me feel a bit more confident with the data. The discrepancy in 2.2 results make me worried about Gecko impact on the results. If the same app (FM from 2.2) run on two different geckos (2.2 and 2.5) give 180ms difference in results and 1mb difference in memory usage, it would indicate that quite possibly many of the regressions we observe come from Gecko and not Gaia changes. zb. _______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
