Re: What data do you want from telemetry (was Re: improving access to telemetry data)

[Patrick McManus]

On Thu, Feb 28, 2013 at 10:36 AM, Benjamin Smedberg
 wrote:

> Cool. Perhaps we should start out with collecting stories/examples:
>

In that spirit:

What I almost always want to do is simply "for the last N days of
variable X show me a CDF (at even just 10 percentile granularity) for
the histogram and let me break that down by sets of build id and or
OS." That's it.

For instance - what is my median time to ready for HTTP vs HTTPs
connections (I've got data for both of those)? What about their tails?
How did they change based on some checkin I'm interested in? Not
rocket science - but incredibly painful to even approximate in the
current front end.. you can kind of do it, but with a bunch of fudging
and manual addition required and it takes forever. I'll admit I get
frustrated with all the talk of EC and Hadoop and what-not when it
really seems a rather straightforward task for me to script on the
data.

Gimme the data set and I can just script it instead of spending an
hour laboriously clicking on things and waiting 15 seconds for every
click.

Reports from the front end seem to indicate that there are 60 Million
submissions in the last month across all channels for one of the
things I'm tracking.. 651K of those from nightly. fwiw.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

What data do you want from telemetry (was Re: improving access to telemetry data)

[Benjamin Smedberg]

On 2/28/2013 10:18 AM, Josh Aas wrote:

On Thursday, February 28, 2013 8:16:50 AM UTC-6, Benjamin Smedberg wrote:


Have we discussed this project with the metrics team yet?

When I started looking into this it wasn't clear to me that people really knew 
what they wanted

Cool. Perhaps we should start out with collecting stories/examples:

* how people currently use the Telemetry UI
* data people wish they had from Telemetry but required either custom 
reports or is unavailable


I myself have the following examples:

Distribution of Flash versions: we wanted to know the distribution of 
Flash versions on various channels, limited on to Windows. This data was 
used to guide the following projects:
* encouraging users on the aurora/beta channels to install the Flash 
beta channel, but making sure that we still had enough users on the 
Flash release channel
* planning the deployment of click-to-play blocklisting and monitoring 
its effectiveness
This data is currently being collected by a custom cronjob, published as 
CSV, and is now being reported via a custom UI here: 
https://crash-analysis.mozilla.com/bsmedberg/flash-distribution.html
I'm also working on correlating this against the crash counts and ADU 
counts for each channel to give us a "crashiness of Flash versions" 
comparison metric that we can use for Flash betas.


Monitoring a specific technical detail about plugin scripting: we 
weren't sure whether plugin elements needed to support direct calling 
via NPAPI. So in bug 827158 bz is adding a telemetry probe which would 
indicate whether a plugin element was ever actually called. We intend to 
check whether there are any hits on this probe once it gets to beta/release.


I am hoping to use telemetry soon to measure the frequency where we hit 
"no active Flash instances running". If users hit 0 instances regularly, 
this will allow us to potentially restart the Flash process more often 
and work around memory leaks and other slow behavior.


I really have never used the telemetry UI frontend. I never quite 
understood how to construct interesting data from it.


--BDS

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform