Re: Now measuring Firefox size per-commit. What else should we be tracking?

[Michael Layzell]

I'd love it if we could get better stats for build times. Not only is it
handy for making sure they don't grow out of control, but we can also use
it to make sure that our static analysis doesn't add an unacceptable level
of overhead on compilation.

On Wed, Nov 4, 2015 at 12:10 PM, Andreas Tolfsen  wrote:

> On 4 November 2015 at 15:55, William Lachance 
> wrote:
> > Is there anything we could be tracking as part of our build or test jobs
> > that we should be? Build times are one thing that immediately comes to
> mind.
>
> Measuring build time would almost certainly be of big value, because
> it has a tendency to slip from time to time.
>
> When we start measuring this we should make sure to store the metrics
> for different compiler/linker versions and options separately so we
> can compare slowdown/speedups for different toolchains.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New heap snapshots tool in the developer tools

[Nick Fitzgerald]

​On Wed, Nov 4, 2015 at 11:07 AM, David Rajchenbach-Teller <
dtel...@mozilla.com> wrote:

> That sounds very useful.
>
> Any chance we can get the same kind of output on the console in case of
> mochitest memory leak?
>

​I've filed ​https://bugzilla.mozilla.org/show_bug.cgi?id=1221704 to track
this.

​You can use `[ThreadSafe]Chrome​Utils.saveHeapSnapshot`[0] to
programmatically save a heap snapshot file. This tool is using that under
the covers right now, but we don't yet expose importing and exporting these
files in the UI. It is pretty high priority to add soon.

Follow along with import and export at
https://bugzilla.mozilla.org/show_bug.cgi?id=1215954 and
https://bugzilla.mozilla.org/show_bug.cgi?id=1215953

So yes, we could definitely programmatically save a heap snapshot file at
the end of a mochitest if we detect a leak. A tiny bit more work needed on
the tool's frontend to import a saved snapshot.

​[0] ​
h
​​
ttps://
dxr.mozilla.org/mozilla-central/rev/f742b9412ed5aace90ad863b276faae0641090a8/dom/webidl/ThreadSafeChromeUtils.webidl#13-25,36-66
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New heap snapshots tool in the developer tools

[J. Ryan Stinnett]

On Wed, Nov 4, 2015 at 4:51 AM, Wilson Page  wrote:
> I assume we can use it via the WebIDE Devtools?

Yes, indeed you can! I just tried it on Aries to be sure. :)

You will need a recent device build that includes Gecko rev
601528a16cf9 from 2015-10-29. I used the Aries dogfood-latest channel
in my test.

- Ryan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New heap snapshots tool in the developer tools

[David Rajchenbach-Teller]

That sounds very useful.

Any chance we can get the same kind of output on the console in case of
mochitest memory leak?

Cheers,
 David

On 03/11/15 23:32, Nick Fitzgerald wrote:
> Hi folks!
> 
> (Jordan Santell made an announcement on the devtools mailing list[0],
> but I thought I'd spread the word around.)
> 
> The first iteration of the new heap snapshots memory tools landed on m-c
> in the middle of the last cycle and rode the train to Developer Edition
> with the merge.
> 
> This tool is built on top of JS::ubi::Node[1] which is an API to do
> analyses on the heap graph as the GC sees it, and the SavedFrame JS
> stack recording[2].
> 
> In this first release, the most notable feature is capturing the JS
> stack when objects are allocated so that you can see exactly where your
> retained objects are coming from. The downside is that this incurs
> overhead and you have to start recording allocation stacks *before* you
> start saving heap snapshot. You can also disect the heap into various
> other buckets.
> 
> http://i.imgur.com/QrcrtWq.png
> 
> This initial version is very much just getting something out the door
> and there is a ton more coming down the pipe. Filtering/searching and
> diffing snapshots are in the works at the moment. Currently, we don't
> take advantage of our full access to every node and edge in the heap
> graph. Soon, we will compute dominator trees to find "deep" size /
> retention as well as find the first N shortest paths from GC roots to
> objects of interest. We are also planning on adding support for cycle
> collected nodes and edges to the JS::ubi::Node graph.
> 
> You can enable the new tool by going to the settings panel of the
> devtools and checking the "Memory" checkbox on the left underneath
> "Default Firefox Developer Tools".
> 
> http://i.imgur.com/hEPTqrT.png
> 
> If you would like to use this tool for the whole runtime rather than
> scoped to a single tab, use the browser toolbox. Here are instructions
> for enabling and using the browser toolbox:
> https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox
> 
> Please try it out and file bugs in the "Firefox" product and "Developer
> Tools: Memory" component! We want this to be a useful tool for _you_!
> 
> Thanks,
> 
> Nick
> 
> [0]
> https://groups.google.com/d/msg/mozilla.dev.developer-tools/TIehyWbmXiQ/g-iLjF3eAgAJ
> 
> [1]
> https://dxr.mozilla.org/mozilla-central/rev/59a6ad6a921f4809dfc37d943d765300c65721e5/js/public/UbiNode.h#31-163
> 
> [2]
> https://dxr.mozilla.org/mozilla-central/rev/59a6ad6a921f4809dfc37d943d765300c65721e5/js/src/vm/SavedStacks.h#21-148
> 
> 
> ___
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
> 


-- 
David Rajchenbach-Teller, PhD
 Performance Team, Mozilla



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

[Chris AtLee]

This is really great, thanks for adding support for this!

I'd like to see the size of the complete updates measured as well, in
addition to the installer sizes.

Do we have alerts for these set up yet?

Cheers,
Chris

On Wed, Nov 4, 2015 at 10:55 AM, William Lachance 
wrote:

> Hey, so as described here:
>
> http://wrla.ch/blog/2015/11/perfherder-onward/
>
> ... I recently added tracking for Firefox installer size inside
> Perfherder. This should let us track how bloated (or not) Firefox is on our
> various supported platforms, on a per-commit basis:
>
>
> https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]=[mozilla-inbound,80cac7ef44b76864458627c574af1a18a425f338,1]=[mozilla-inbound,0060252bdfb7632df5877b7594b4d16f1b5ca4c9,1]
>
> As I mentioned in the blog post, it's now *very* easy (maybe too easy?
> heh) to submit "performance" (read: quantitative) data for any job
> reporting to treeherder by outputting a line called "PERFHERDER_DATA" to
> the log.
>
> Is there anything we could be tracking as part of our build or test jobs
> that we should be? Build times are one thing that immediately comes to
> mind. Is there anything else?
>
> In order to be a good candidate for measurement in this kind of system, a
> metric should be:
>
> 1. Relatively deterministic.
> 2. Something people actually care about and are willing to act on, on a
> per-commit basis. If you're only going to look at it once a quarter or so,
> it doesn't need to be in Perfherder.
>
> Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
> add stuff incrementally, to make sure Perfherder can handle the load, but
> I'd love to hear all your ideas.
>
> Will
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Now measuring Firefox size per-commit. What else should we be tracking?

[William Lachance]

Hey, so as described here:

http://wrla.ch/blog/2015/11/perfherder-onward/

... I recently added tracking for Firefox installer size inside 
Perfherder. This should let us track how bloated (or not) Firefox is on 
our various supported platforms, on a per-commit basis:


https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]=[mozilla-inbound,80cac7ef44b76864458627c574af1a18a425f338,1]=[mozilla-inbound,0060252bdfb7632df5877b7594b4d16f1b5ca4c9,1]

As I mentioned in the blog post, it's now *very* easy (maybe too easy? 
heh) to submit "performance" (read: quantitative) data for any job 
reporting to treeherder by outputting a line called "PERFHERDER_DATA" to 
the log.


Is there anything we could be tracking as part of our build or test jobs 
that we should be? Build times are one thing that immediately comes to 
mind. Is there anything else?


In order to be a good candidate for measurement in this kind of system, 
a metric should be:


1. Relatively deterministic.
2. Something people actually care about and are willing to act on, on a 
per-commit basis. If you're only going to look at it once a quarter or 
so, it doesn't need to be in Perfherder.


Anyway, just thought I'd open the floor to brainstorming. I'd prefer to 
add stuff incrementally, to make sure Perfherder can handle the load, 
but I'd love to hear all your ideas.


Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

[William Lachance]

On 2015-11-04 11:15 AM, Chris AtLee wrote:

This is really great, thanks for adding support for this!

I'd like to see the size of the complete updates measured as well, in
addition to the installer sizes.


Where are the updates generated? As part of the build process? If so, 
can you self-serve your way to victory here, by adding something to this 
log line?


http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/building/buildbase.py#l1628

I'm happy to provide guidance where needed/helpful.


Do we have alerts for these set up yet?


Not yet, alert/sheriffing support for Perfherder is being tracked in 
this bug:


https://bugzilla.mozilla.org/show_bug.cgi?id=1150152

I'm hoping to have something working by the end of the quarter.

Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

[William Lachance]

On 2015-11-04 10:55 AM, William Lachance wrote:


1. Relatively deterministic.
2. Something people actually care about and are willing to act on, on a
per-commit basis. If you're only going to look at it once a quarter or
so, it doesn't need to be in Perfherder.

Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
add stuff incrementally, to make sure Perfherder can handle the load,
but I'd love to hear all your ideas.


Someone mentioned "test times" to me in private email.

I do think test times are worth tracking, but probably not in 
Perfherder: test times might not be deterministic depending on where / 
how they're running (which makes it difficult to automatically detect 
regressions and sheriff them on a per commit basis) and regardless 
there's too much data to really be manageable by Perfherder's intended 
interface even if that problem were magically solved.


As a possible alternative, I believe Kyle Lahnakoski's ActiveData 
project (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData) 
already *does* track this type of data but last I heard he was looking 
for more feedback on how to alert/present it to the platform community. 
If you have any ideas on this, please let him know (he's CC'ed). :)


Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

[Andreas Tolfsen]

On 4 November 2015 at 15:55, William Lachance  wrote:
> Is there anything we could be tracking as part of our build or test jobs
> that we should be? Build times are one thing that immediately comes to mind.

Measuring build time would almost certainly be of big value, because
it has a tendency to slip from time to time.

When we start measuring this we should make sure to store the metrics
for different compiler/linker versions and options separately so we
can compare slowdown/speedups for different toolchains.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New heap snapshots tool in the developer tools

[Wilson Page]

This looks like a really valuable tool Nick! It's really cool that you can
reference memory back to real, familiar object names, haven't seen anything
like this before :)

I assume we can use it via the WebIDE Devtools?

Can't wait to try it out!

*W I L S O N  P A G E*

Front-end Developer
Firefox OS (Gaia)
London Office

Twitter: @wilsonpage
IRC: wilsonpage

On Tue, Nov 3, 2015 at 10:32 PM, Nick Fitzgerald 
wrote:

> Hi folks!
>
> (Jordan Santell made an announcement on the devtools mailing list[0], but I
> thought I'd spread the word around.)
>
> The first iteration of the new heap snapshots memory tools landed on m-c in
> the middle of the last cycle and rode the train to Developer Edition with
> the merge.
>
> This tool is built on top of JS::ubi::Node[1] which is an API to do
> analyses on the heap graph as the GC sees it, and the SavedFrame JS stack
> recording[2].
>
> In this first release, the most notable feature is capturing the JS stack
> when objects are allocated so that you can see exactly where your retained
> objects are coming from. The downside is that this incurs overhead and you
> have to start recording allocation stacks *before* you start saving heap
> snapshot. You can also disect the heap into various other buckets.
>
> http://i.imgur.com/QrcrtWq.png
>
> This initial version is very much just getting something out the door and
> there is a ton more coming down the pipe. Filtering/searching and diffing
> snapshots are in the works at the moment. Currently, we don't take
> advantage of our full access to every node and edge in the heap graph.
> Soon, we will compute dominator trees to find "deep" size / retention as
> well as find the first N shortest paths from GC roots to objects of
> interest. We are also planning on adding support for cycle collected nodes
> and edges to the JS::ubi::Node graph.
>
> You can enable the new tool by going to the settings panel of the devtools
> and checking the "Memory" checkbox on the left underneath "Default Firefox
> Developer Tools".
>
> http://i.imgur.com/hEPTqrT.png
>
> If you would like to use this tool for the whole runtime rather than scoped
> to a single tab, use the browser toolbox. Here are instructions for
> enabling and using the browser toolbox:
> https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox
>
> Please try it out and file bugs in the "Firefox" product and "Developer
> Tools: Memory" component! We want this to be a useful tool for _you_!
>
> Thanks,
>
> Nick
>
> [0]
>
> https://groups.google.com/d/msg/mozilla.dev.developer-tools/TIehyWbmXiQ/g-iLjF3eAgAJ
>
> [1]
>
> https://dxr.mozilla.org/mozilla-central/rev/59a6ad6a921f4809dfc37d943d765300c65721e5/js/public/UbiNode.h#31-163
>
> [2]
>
> https://dxr.mozilla.org/mozilla-central/rev/59a6ad6a921f4809dfc37d943d765300c65721e5/js/src/vm/SavedStacks.h#21-148
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform