Firefox data engineering newsletter, Q1 2018

[Georg Fritzsche]

As the Firefox data engineering teams we provide core tools for using data
to other teams. This spans from collection through *Firefox Telemetry*,
storage & processing in our *Data Platform* to making data available in *Data
Tools*.

To make new developments more visible we aim to publish a quarterly
newsletter. As we skipped one, some important items from Q4 are also
highlighted this time.

This year our teams are putting their main focus on:

   -

   Making experimentation easy & powerful.
   -

   Providing a low-latency view into product release health.
   -

   Making it easy to work with events end-to-end.
   -

   Addressing important user issues with our tools.


*Usage improvements*

Last year we started to investigate how our various tools are used by
people working on Firefox in different roles. From that we started
addressing some of the main issues users have.

Most centrally, the Telemetry portal <https://telemetry.mozilla.org/> is
now the main entry point to our tools, documentation and other resources.
When working with Firefox data you will find all the important tools linked
from there.

We added the probe dictionary
<https://telemetry.mozilla.org/probe-dictionary/> to make it easy to find
what data we have about Firefox usage.

For STMO <https://sql.telemetry.mozilla.org/>, our Redash instance, we deployed
a major UI refresh
<https://blog.redash.io/improving-redash-user-experience-36e3ca47d1b3> from
the upstream project.

There is new documentation on prototyping
<https://docs.telemetry.mozilla.org/tools/stmo.html#prototyping-queries>
and optimizing
<https://docs.telemetry.mozilla.org/concepts/sql_optimization.html> STMO
queries.

Our data documentation <https://docs.telemetry.mozilla.org/> saw many other
updates, from cookbooks on how to see your own pings
<https://docs.telemetry.mozilla.org/cookbooks/view_pings_cep.html> and sending
new pings <https://docs.telemetry.mozilla.org/cookbooks/new_ping.html> to
adding more datasets
<https://docs.telemetry.mozilla.org/concepts/choosing_a_dataset.html>. We
also added documentation on how our data pipeline works
<https://docs.telemetry.mozilla.org/concepts/pipeline/data_pipeline.html>.


*Enabling experimentation*

For experimentation, we have focused on improving tooling. Test Tube
<https://firefox-test-tube.herokuapp.com/> will soon be our main experiment
dashboard, replacing experiments viewer. It displays the results of
multivariant experiments that are being conducted within Firefox.

We now have St. Moab <https://github.com/mozilla/stmoab> as a toolkit for
automatically generating experiment dashboards.


*Working with event data*

To make working with events easier, we improved multiple stages in the
pipeline. Our documentation has an overview
<https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html>
of the data flow.

On the Firefox side, events can now be recorded through the events API
<https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html>,
from add-ons
<https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#registerevents>,
and whitelisted Firefox content
<https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/hybrid-content.html>.
From Firefox 61, all recorded events are automatically counted into scalars
<https://bugzilla.mozilla.org/show_bug.cgi?id=1440673>, to easily get
summary statistics.

Event data is available for analysis in Redash in different datasets
<https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html#datasets>.
We can now also connect more event data to Amplitude
<https://amplitude.com/>, a product analytics tool. A connection for some
mobile events to Amplitude is live, for Firefox Desktop events it will be
available soon.


*Low-latency release health data*

To enable low-latency views into release health data, we are working on
improving Mission Control <https://data-missioncontrol.dev.mozaws.net/>,
which will soon replace arewestableyet.com <https://www.arewestableyet.com/>
.

It has new features
<https://wlach.github.io/blog/2018/04/mission-control-update/> that enable
comparing quality measures like crashes release-over-release across
channels.


*Firefox Telemetry tools*

For Firefox instrumentation we expanded on the event recording APIs
<https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html#firefox-event-collection>.
To make build turnaround times faster, we now support adding scalars in
artifact builds
<https://mail.mozilla.org/pipermail/fx-data-dev/2018-February/000114.html>
and will soon extend this to events
<https://bugzilla.mozilla.org/show_bug.cgi?id=1448945>.

Following the recent Firefox data preferences changes, we adopted Telemetry
<ht

PSA: Data preference changes in Firefox 58

[Georg Fritzsche]

*This a shorter summary; if you need more details see this blog post
.*

As part of the Photon UI refresh, the Firefox data preferences were
streamlined. Previously we had two data controls for historic reasons, but
now:

   - There is just one control for data upload for Firefox, which is on by
   default.
   - Individual measurements are now collected from either all users or
   prerelease users.

So for Firefox we now just differentiate between “release data” and
“prerelease data”. *(For some specific features or studies there may still
be explicit opt-in notifications, but this covers all standard Firefox data
collection.)*

As detailed in another post
,
this doesn’t change anything about the type of data being collected while
it makes the data controls less confusing. However, it does have some
technical consequences for Telemetry that we landed in Firefox 58.
What does this mean for me?

While this has broader technical impact on Firefox data collection, the
changes we landed should make this work mostly seamless.

For *Firefox users*, there is no significant change; the data upload
setting users chose is respected as before. The opt-in for additional
Telemetry data went away; instead we always collect less data on Firefox
release.

For most *Firefox development*, nothing should change. Telemetry takes care
of doing the right thing for histograms, scalars & events internally.

Local builds from current mozilla-central should default to recording all
the prerelease data but not upload it, without any extra build flags.
*about:telemetry* allows you to confirm that.

For testing locally how release Telemetry behaves, there is a hidden pref:
*toolkit.telemetry.testing.overridePreRelease*

.

For *Fennec development*, nothing changed. Fennec still has the same
Telemetry behavior and preference handling as before.

For *writing Firefox tests*, *nsITelemetry.canRecordExtended* can be used
to check whether prerelease data is recorded. The pref
*toolkit.telemetry.testing.overridePreRelease* can be used for tests that
need to enable prerelease behavior.

For *QAing Firefox data collection*, you might need to override the
prerelease status using the pref
*toolkit.telemetry.testing.overridePreRelease*. The prerelease and sending
status can be checked in *about:telemetry*.

If you are *looking at Telemetry data*, there are some changes. In Firefox
58, we will not receive the “extended” data from release anymore. This
directly affects some parts of telemetry.mozilla.org that provided release
data from that small *“opt-in”* population:

   - the distribution and evolution dashboards will soon not show release
   data anymore.
   - telemetry.js
   
   and the aggregates service will not provide release data when 58 is
   released.

If you will be affected by these changes and need alternatives, please
reach out to us.
Reaching out

   - If you have any questions reach out
   to :gfritzsche, :chutten, :dexter, :frank or the team
   .
   - For any specific issues, please file a bug
   
.
   We are using bug 1406390
    for tracking.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Firefox data platform & tools update, Q3 2017

[Georg Fritzsche]

*Mission Control showing content crashes per 1k usage hours.*


As the data platform & tools team we provide core tools for using data to
other teams. This spans Firefox Telemetry, data storage and analysis to
some central data viewing tools. To make new developments more visible we
publish a quarterly update on our work.

In the last quarter we continued focusing on decreasing data latency,
supporting analytics and experimentation workflows, improving stability and
building Mission Control.
Let's go faster

To enable faster decision making, we worked on improving latency for
important use-cases.

Most notable is that the main pings now arrive much faster, which power
most of our main dashboards and analysis. The new rule of thumb is 2 days
until 95% of the main ping data
<https://chuttenblog.wordpress.com/2017/09/12/two-days-or-how-long-until-the-data-is-in/>
is available, from activity in the browser to being available for analysis.

In Firefox Telemetry we can now record new probes from add-ons without
having to ride the trains, which greatly reduces shipping times for
instrumentation. This is available first with events
<https://medium.com/georg-fritzsche/recording-new-telemetry-from-add-ons-61d194568212>
in 56 and scalars
<https://www.a2p.it/wordpress/tech-stuff/mozilla/recording-telemetry-scalars-from-add-ons/>
in 58.

The new update ping
<https://chuttenblog.wordpress.com/2017/10/04/anatomy-of-a-firefox-update/>
provides a lower-latency signal for when updates are staged and
successfully applied. It is queryable through the telemetry_update_parquet
dataset
<https://docs.telemetry.mozilla.org/datasets/batch_view/update/reference.html>
.

Similarly, the new-profile ping
<https://www.a2p.it/wordpress/tech-stuff/mozilla/getting-firefox-data-faster-introducing-the-new-profile-ping/>
is a signal for new profiles and installations, which is now queryable
through the telemetry_new_profile_parquet dataset
<https://docs.telemetry.mozilla.org/datasets/batch_view/new_profile/reference.html>
.

The new first-shutdown ping
<https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/first-shutdown-ping.html>
helps us to better understand users that churn after the first session, by
sending the first sessions data of a user immediately on Firefox shutdown.
Enabling experimentation

This year saw a lot of cross-team work on enabling experimentation
workflows in Firefox. A focus was on enabling various SHIELD studies
<https://medium.com/@mgrimes/shield-studies-go-faster-bet-smarter-1010ae8d8e>
.

Here the experiments viewer <https://moz-experiments-viewer.herokuapp.com/>
saw a lot of improvements, which provides a front-end view for inspecting
how various metrics perform in an experiment.

An experiments dataset is now available in Redash and Spark, which includes
data for SHIELD opt-in experiments and is based on the main_summary dataset.

The experiment_aggregates dataset now includes metadata about the
experiment, and its reliability and speed have improved significantly.

Other use-cases can build on the ping data from most experiments using
experiment
annotations
<https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/experiments.html>,
which is available within 15 minutes in the telemetry-cohort data source
<https://gist.github.com/mreid-moz/7c0b32c9b9519f53d372a6c7b7af765d>.
Tools for exploring data

Our data tools make it easier to access and query the data we have. Here
our Redash installation at sql.telemetry.mozilla.org saw many improvements
including:

   -

   Query revision control and reversion.
   -

   Better security and usability for templated queries.
   -

   Schema browser and autocomplete usability and performance improvements.
   -

   Better support for Athena data sources.


Mission Control <https://wlach.github.io/blog/2017/10/mission-control/> is
a new tool, which makes key measures of a Firefox release, like crash
counts, available with low latency. An early version of it is now available
here <https://data-missioncontrol.dev.mozaws.net/#/>.

On the Firefox side, about:telemetry got a major redesign
<https://chuttenblog.wordpress.com/2017/08/30/the-photonization-of-abouttelemetry/>,
which makes it more easy to navigate, added a global search and aligns it
with the photon design.
Powering data analysis

To make analysis more effective, two new datasets were added:

   -

   clients_daily
   
<https://github.com/mreid-moz/firefox-data-docs/blob/6d5076cd2746639874704d8b00af431f4e60c5e7/datasets/mozetl/clients_daily/intro.md>,
   which summarizes main ping data into one row per client and day.
   -

   heavy_users
   <https://mail.mozilla.org/pipermail/fx-data-dev/2017-October/71.html>
   (docs
   
<https://docs.telemetry.mozilla.org/concepts/choosing_a_dataset.html#heavyusers>),
   which has a similar format, but

Firefox data platform & tools update, Q2 2017

[Georg Fritzsche]

The data platform and tools teams are working on our core Telemetry system,
the data pipeline, providing core datasets and maintaining some central
data viewing tools.

To make new work more visible, we provide quarterly updates.
What’s new in the last few months?

A lot of work in the last months was on reducing latency, supporting
experimentation and providing a more reliable experience of the data
platform.




On the data collection side, we have significantly improved reporting
latency from Firefox 55, with preliminary results from Beta showing we receive
95% of the "main" ping within 8 hours

(compared to previously over 90 hours). Curious for more detail? #1

and #2

should have you covered.

We also added a "new-profile" ping
,
which gives a clear and timely signal for new clients.

There is a new API to record active experiments in Firefox
.
This allows annotating experiments or interesting populations in a standard
way.

The record_in_processes is now required for all histograms. This removes
ambiguity about which process they are recorded in.

The data documentation moved to a new home: docs.telemetry.mozilla.org. Are
there gaps in the documentation you want to see filled? Let us know by filing
a bug

.

For datasets, we added telemetry_new_profile_parquet, which makes the data
from the "new-profile" ping available.

Additionally, the main_summary dataset now includes all scalars and uses a
whitelist for histograms, making it easy to add them. Important fields like
active_ticks and Quantum release criteria were also added and backfilled.

For custom analysis on ATMO ,
cluster lifetimes can now be extended self-serve in the UI. The stability
of scheduled job stability also saw major improvements.

There were first steps towards supporting Zeppelin notebooks better; they
can now be rendered as Markdown
 in Python.

The data tools work is focused on making our data available in a more
accessible way. Here, our main tool re:dash
 saw multiple improvements.

Large queries should no longer show the slow script dialog
. A new Athena data source was
introduced, which contains a subset of our Telemetry-based derived
datasets. This brings huge performance and stability improvements over
Presto.

Finally, scheduled queries can now have an expiration date.
What is up next?

For the next few months, interesting projects in the pipeline include:

   -

   The experiments viewer & pipeline, which will make it much easier to run
   pref-flipping experiments in Firefox.
   -

   Recording new probes from add-ons into the main ping (events, scalars,
   histograms).
   -

   We will define and monitor basic guarantees for the Telemetry client
   data (like reporting latency ranges).
   -

   A re-design of about:telemetry is currently on-going, with more
   improvements on the way.
   -

   A first version of Mission Control will be available, a tool for more
   real-time release monitoring.
   -

   Analyzing the results of the Telemetry survey, (thanks everyone!) to
   inform our planning.
   -

   Extending the main_summary dataset to include all histograms.
   -

   Adding a pre-release longitudinal dataset, which will include all
   measures on those channels.
   -

   Looking into additional options to decrease the Firefox data reporting
   latency.

How to contact us.

Please reach out to us with any questions or concerns.

   -

   You can find us on IRC in #telemetry and #datapipeline.
   -

   We are available on slack in #fx-metrics.
   -

   The main mailing list for data topics is fx-data-dev
   .
   -

   Bugs can be filed in one of these components
   .
   -

   You can also find us on Twitter as @MozTelemetry
   .
___
dev-platform 

Data platform & tools update, Q1 2017

[Georg Fritzsche]

The data platform and tools teams are working on our core Telemetry system,
the data pipeline, providing core datasets and maintaining some central
data viewing tools.

To make new work more visible, we intend to provide quarterly updates.

What's new in the last few months?

On the data collection side, scalars

are now supported through the pipeline, so new flag and count histograms
are now disallowed on Desktop in favour of boolean and uint scalars.

Event Telemetry

is now ready for adoption. A general events table
 is available,
a sync events table coming up and further uses are being looked at.

For documentation, we re-worked the guide for adding new Telemetry

and extended the detailed data collection documentation

.

The prototype for making probe history
 more discoverable now has
daily updates and supports Nightly too.

For filing or finding bugs, there is now a new Data Platform and Tools

product. Note that client-side bugs still go into the separate
Toolkit::Telemetry
component

.

The data pipeline work powers results for re:dash
 and custom analysis
 among other things.

Notable recent work here includes:

   -

   Providing efficient lookup of client histories using Hbase
   

   .
   -

   Experimental support for Zeppelin
   , a
   new notebook type that improves Jupyter.
   -

   The Telemetry dashboard
    is now faster
   through a dedicated read replica and client-side caching.
   -

   The Dataset API now has a select method
   

   to return a subset of fields.
   -

   Providing a framework for testable Python ETL jobs
    generated from a template
   .
   -

   Direct-to-parquet
   

   is in production, making easier to build datasets from incoming pings.


The data tools work powers tools that make data analysis more accessible
across Mozilla.

Updates here are:

   -

   For re:dash , the UI improved to
   make the dashboard list more accessible.
   -

   re:dash query issues were reduced by handling failing queries using
   exponential back-off.
   -

   There is also a python re:dash client
    (h/t to emtwo), allowing
   programmatic generation of queries and dashboards.
   -

   The distribution viewer  is now
   live, making distributions of a set of important Firefox metrics available.


   -

   The analysis service  gained
   features
   

   like persistent cluster storage and the ability to extend cluster lifetimes.


Coming soon

For the next few months, interesting projects in the pipeline include:

   -

   Work to decrease data latency, by sending the last ping of a Firefox
   session immediately. We will also start sending timely pings for new users
   and updates.
   -

   Rebooting documentation
   
,
   providing guidance as well as tying existing documentation together.
   -

   Start supporting new data collection from add-ons in Telemetry, starting
   with events.


Contact us

Please reach out to us with any questions or concerns.

You can find us on IRC in #telemetry and #datapipeline.

The main mailing list for data topics is fhr-dev
.

Bugs can be filed in one of these components
.

You can also find us on Twitter as @MozTelemetry
.
___
dev-platform mailing list
dev-platform@lists.mozilla.org

Re: Rationalising Linux audio backend support

[Georg Fritzsche]

Given Ubuntus popularity, we decided to first reach out to that
distribution.
With limited resources and other work, we haven't reached out much further
yet.

Recently, there was some progress:
Fedora is apparently submitting Telemetry
.
Arch Linux now sends Telemetry
.

However, we do depend on work from the distributions.
If you want to work with us on this, please reach out by filing a bug
,
mail to me or fhr-dev, or catch us on IRC in #telemetry.

Thanks,
Georg

On Sat, Apr 1, 2017 at 2:48 PM,  wrote:

> Den fredag 31 mars 2017 kl. 15:49:50 UTC+2 skrev Chris Coulson:
> > On 31/03/17 05:52, burmar...@gmail.com wrote:
> > > Ubuntu just re-enabled ALSA on their latest Firefox 52.0.2 release. Go
> Ubuntu!
> > It's enabled, but please see the small-print in the changelog
> > description at
> > https://launchpad.net/ubuntu/+source/firefox/52.0.2+build1-
> 0ubuntu0.16.04.1.
> > The Firefox package in Ubuntu is maintained by 1 contributor in his
> > spare time and myself who is only able to do the minimum in order to
> > provide updates, so Ubuntu flavors that don't ship Pulseaudio need to
> > step up to maintain this code if they want it to continue working and
> > don't want it to be disabled again in a future update.
> >
>
> On the other hand, Ubuntu is one of the most popular distros. So this
> action should push firefox in the right direction. Does anyone have an idea
> about the Debian policy? Fedora?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Please don't add JS implemented telemetry probes to hot code paths. (eom)

[Georg Fritzsche]

For hot code paths, recording every accumulation from C++ directly into
Telemetry may also not be ideal.
Telemetry::Accumulate() is not very expensive, but it does have costs
(locking, lookups).
It can e.g. be more efficient to accumulate in a local variable and record
the result into Telemetry afterwards.
I will update the docs

to make this clear.

Georg

On Sun, Feb 12, 2017 at 10:22 PM, smaug  wrote:

> On 02/12/2017 08:52 PM, Jared Wein wrote:
>
>> The original email looks to be related to:
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1338891
>> and
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1338902
>>
>> though I agree that the message could have been explained a little better.
>>
>>
> Sorry about that. I thought the subject was clear enough :)
>
> But yes, don't implement probes as JS event listener when the event is
> some rather commonly dispatched.
> Same applies to observer service observers etc.
>
> Bug 1338902 doesn't have much to do with this. It is just about the
> generic overhead probes add, also in C++.
> Not very high overhead, but definitely measurable. So need to be careful
> also in C++ what to measure and how.
>
> -Olli
>
>
> - Jared
>>
>> On Sun, Feb 12, 2017 at 12:41 PM, Gijs Kruitbosch <
>> gijskruitbo...@gmail.com>
>> wrote:
>>
>> Can you clarify what you mean by "JS implemented" here? I was under the
>>> impression the telemetry implementation was in C++...
>>>
>>> Separately, was there a concrete reason for you posting this warning? I
>>> don't wish to shame people, but a concrete example of a 'wrong' pattern
>>> might help in making it more obvious what exactly is problematic about
>>> what
>>> you're suggesting, and what should be done instead.
>>>
>>> ~ Gijs
>>>
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Data platform update

[Georg Fritzsche]

Data platform update

The data platform team is working on our core Telemetry system, the data
pipeline and providing core datasets, with support from the Firefox data
engineering and the Data tools team.

To make new features more visible, we intend to provide quarterly updates,
starting with this one.

What's new in the last few months?

On the data collection side, we added scalars

and added engagement measures

on top. We now support recording histograms in child processes and
added categorical
histograms

.

We improved documentation, starting from the Telemetry wiki page
 and updated onboarding
.

Further work happened on:

   -

   improved client-side documentation
   



   -

   an introduction on analyzing telemetry data
   

   -

   a guide on choosing a dataset
   

   for analysis
   -

   updates to examples for the longitudinal dataset
   

   -

   exploring how to make Firefox data more discoverable
   


The data pipeline work powers results for re:dash
 and custom analysis
 among other things.

Notable recent work here includes:

   -

   the cross-sectional
   

   dataset is now operational in re:dash
   
   -

   socorro crash data
    is now
   available in re:dash
   -

   the new dataset API
   

   improves querying raw ping data
   -

   the self-serve analysis portal 
   was relaunched with improved UX
   -

   provided easy access to self-serve real-time analysis
   
   -

   launched a knowledge repository 
   to make reports discoverable in one place and easier to review


Coming soon

For the next few months, interesting projects in the pipeline include:

   -

   event telemetry , which
   enables recording event data into Telemetry in a common format
   

   -

   work to decrease data latency, allowing us to make decisions faster
   -

   adding Telemetry support for Add-ons
   -

   making it easy for new pings to show up in re:dash thanks to
   direct-to-parquet
   -

   enabling efficient lookup of client histories using Hbase
   -

   improved alerting
   

   for Telemetry probes
   -

   creating a new crash summary dataset, to make it easier to analyze crash
   data


Contact us


Please reach out to us with any questions or concerns.


You can find us on IRC in #telemetry and #datapipeline.

The main mailing list for data topics is fhr-dev
.

Bugs can be filed in one of these components
.

You can also find us on Twitter as @MozTelemetry
.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rationalising Linux audio backend support

[Georg Fritzsche]

On Thu, Jul 14, 2016 at 11:07 AM, Mike Hommey  wrote:

> On Thu, Jul 14, 2016 at 10:33:17AM +0200, Paul Adenot wrote:
> > I just landed some telemetry to measure the usage of all audio backends,
> > we'll have data soon.
>
> Usual question when it comes to Linux: what's the status of telemetry in
> Distro builds?
>

This is a good point. We worked with Ubuntu to make sure that we are
receiving Telemetry from there, but for other distributions it is not very
clear.

This gives an overview of the current incoming Telemetry for Linux (from a
1% sample of our data, "canonical" is the Ubuntu distribution):
https://sql.telemetry.mozilla.org/queries/678#table
Also keep in mind, unless using opt-out probes, that the opt-in rates for
Telemetry are low:
https://sql.telemetry.mozilla.org/queries/682#table

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Is Telemetry::Accumulate threadsafe?

[Georg Fritzsche]

Bug 1141565 made it thread-safe, bug 1258183 is dealing with some remaining
TSan issues.

Georg

On Wed, May 11, 2016 at 7:23 PM, Kyle Huey  wrote:

> And if not, why is this not documented?
>
> - Kyle
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Firefox Hello new data collection

[Georg Fritzsche]

On Mon, Apr 4, 2016 at 5:44 PM, Gijs Kruitbosch 
wrote:

It also seems like you filed the privacy review after the functionality was
> implemented and is now shipping, which per
> https://wiki.mozilla.org/Privacy/Reviews seems like it is too late to
> incorporate meaningful feedback. I'm not on the privacy team, but that
> order looks wrong to me.
>

We have a common data collection review process for Firefox now (with
additionally more intense privacy reviews where needed):
https://wiki.mozilla.org/Firefox/Data_Collection

The idea is definitely to request approval before landing new data
collections.
For more complex collections it is helpful to start communications in the
design phase to catch problems before implementation (maybe this happened
through other channels here?).

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Transitioning from FHR to Telemetry

[Georg Fritzsche]

TL;DR: The old FHR system is going away, use Telemetry.

We have been working on migrating the FHR data sources to Telemetry for a
while now, but realized there has been no proper notice going around so far.
As of Firefox 43, we have disabled FHR upload on Desktop. Instead we are
now collecting opt-out data via Telemetry
 (Firefox for Android will follow along
soon ).

*What about the FHR-based metrics?*
Important data should have been either migrated to Telemetry or have a plan
in place to do so (if not, please bring it up).
FHR-like data is now collected as "opt-out" data in Telemetry, subject to
the data collection policy
.
Note that mobile opt-out Telemetry is following next year.


*Where to get data from?*
The simplest way to investigate the data should be via the main Telemetry
dashboard . This also links to additional
dashboards and further documentation.
For more advanced uses you can either use telemetry.js, run custom analysis

or talk to the measurements team about other options.


*Further information and contact*
IRC: #datapipeline, #telemetry, #metrics
Mailing list: fhr-dev 
If you are heading to Orlando, there is a Telemetry session
.

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Shutdown hangs are very common

[Georg Fritzsche]

Can we fix the UX?
Presumably we will never have zero shutdown hangs and there may be
different/better ways to prompt the user about it.

On Mon, Jul 6, 2015 at 10:48 PM, Vladan D vdje...@mozilla.com wrote:

 KaiRo pointed out another reason to reduce shutdown hang rates on IRC:
 it's lousy UX. The crash-reporter dialog pops up a minute after the user
 closed Firefox

 On Monday, July 6, 2015 at 4:37:41 PM UTC-4, RyanVM wrote:
  On 7/6/2015 4:34 PM, Vladan D wrote:
   Background: Firefox shutdown hangs are turned into shutdown crashes by
 a watchdog thread [1] that forces a crash if shutdown hasn't completed
 within 1 minute. Thanks to the watchdog and the Windows profile unlocker
 [2], shutdown hangs aren't as frustrating as they used to be. However,
 shutdown hangs might still be causing data loss and they are indicative of
 potentially-serious bugs in the code.
  
  
   According to this graph of Firefox crash rate history, shutdown hangs
 (crashes) make up about one third of all browser crashes [3]:
  
  
 https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxrel
  
   I've been told shutdown hangs often don't get enough attention.
   Should fixing shutdown hangs be higher priority?
   And if so, should we allow features with shutdown hangs to be released?
  
  
   Notes:
   1. Force Firefox crash if shutdown hangs
 https://bugzilla.mozilla.org/show_bug.cgi?id=1038342
   2. win32 implementation of nsIProfileUnlocker
 https://bugzilla.mozilla.org/show_bug.cgi?id=286355
   3. The graph above shows that the overall crash rate jumped up by
 roughly a third when the watchdog code shipped in Firefox 36. Hover over
 the 36 box on the blue line
  
 
  Windows mochitest-bc shutdown hangs have been on of the top oranges in
  our automation for months now. See bug 1121145. Would be great if we
  could get more eyes on the problem.
 
  -Ryan

 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Telemetry additions: count histograms keyed histograms

[Georg Fritzsche]

Per bug 1069873 and bug 1069874, Telemetry now has:
* a new histogram type for counts (so there is no need anymore to use boolean 
histograms for that)
* keyed histograms, basically providing a key/value store (mapping string keys 
to histograms)

I updated the documentation accordingly:
https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Telemetry additions: count histograms keyed histograms

[Georg Fritzsche]

Ah, thanks for the catch, i will fix that.
The nsITelemetry page should already up to date for this though.

Georg

On 06 Nov 2014, at 17:11, Jan Odvarko odva...@gmail.com wrote:

 I updated the documentation accordingly:
 https://developer.mozilla.org/en-
 US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe
 Nice!
 
 Note that the page needs to be yet updated since 
 registerAddonHistogram has different order of arguments now:
 https://bugzilla.mozilla.org/show_bug.cgi?id=1069953
 
 Honza
 
 
 
 -Original Message-
 From: dev-platform [mailto:dev-platform-
 bounces+odvarko=gmail@lists.mozilla.org] On Behalf Of Georg Fritzsche
 Sent: Thursday, November 06, 2014 5:05 PM
 To: dev-platform@lists.mozilla.org
 Cc: Firefox Dev
 Subject: Telemetry additions: count histograms  keyed histograms
 
 Per bug 1069873 and bug 1069874, Telemetry now has:
 * a new histogram type for counts (so there is no need anymore to use
 boolean histograms for that)
 * keyed histograms, basically providing a key/value store (mapping string
 keys to histograms)
 
 I updated the documentation accordingly:
 https://developer.mozilla.org/en-
 US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe
 
 Georg
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform
 
 
 ---
 This email is free from viruses and malware because avast! Antivirus 
 protection is active.
 http://www.avast.com
 

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Documenting locations for testing preferences

[Georg Fritzsche]

Hi,

our testing locations are spread out a little much over many different 
locations (see e.g. bug 1023483), so i started documenting some of them:
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing#Need_to_set_preferences_for_test-suites.3F

Feel free to add more locations (and explanations) for things you run into.

Georg


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Can we start using C++ STL containers in Mozilla code?

[Georg Fritzsche]

On 12/10/2013 16:17, Joshua Cranmer  wrote:

I think the right answer is to make the myriad of datatypes we have more
STL-ish--as a first step at least, can we at least add iterator support
for them and get rid of the abominable
EnumerateForwards/EnumerateBackewards methods? This would allow us to
use range-based for once our minimum compiler support gets there.


Yes please.
From what i saw so far we even have a bunch different iteration 
approaches with the current Mozilla containers. We also seem to have a 
lot of algorithmic duplication which seems pretty bad for maintenance 
etc. - getting all, or most, containers to implement compatible 
interfaces would allow for future improvements and cleanups.


Georg

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: how long are we continuing 32-bit OS X support?

[Georg Fritzsche]

On Oct 22, 2013, at 1:27 AM, Mike Hommey m...@glandium.org wrote:

 If there are and we still need to support them despite dropping 10.6
 (which I'm not advocating, but i understand it'd be an option), I wonder
 if it would be possible, and how much work it would be, to make the
 plugin-container not use libxul at all, and just be a small shim doing
 ipc calls. But I really don't know what the plugin container currently
 does on its end that requires libxul code besides ipc.

I'm not sure on what parts are all involved, but that should be a good size win.
I wonder if there is anything preventing us from using the same approach for
the eventual Win64 builds?

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: running tests in HiDPI mode on the build machines

[Georg Fritzsche]

On 11.07.2013, at 09:11, Cameron McCormack c...@mcc.id.au wrote:
 This proposal is just to affect how content is rendered, by setting that 
 pref.  If it's a XUL UI it should render at the higher resolution.  It 
 wouldn't cause native UI to be rendered differently.  I don't really know how 
 we'd achieve that, on any platform, without actual HiDPI hardware.  (Maybe 
 those dongles that fake a monitor can be used on the Mac Minis?)

On OS X, Quartz Debug should help:
https://developer.apple.com/library/mac/ipad/#documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/Testing/Testing.html#//apple_ref/doc/uid/TP40012302-CH6-SW1

Georg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform