The data platform and tools teams are working on our core Telemetry system, the data pipeline, providing core datasets and maintaining some central data viewing tools.
To make new work more visible, we provide quarterly updates. What’s new in the last few months? A lot of work in the last months was on reducing latency, supporting experimentation and providing a more reliable experience of the data platform. On the data collection side, we have significantly improved reporting latency from Firefox 55, with preliminary results from Beta showing we receive 95% of the "main" ping within 8 hours <https://www.a2p.it/wordpress/tech-stuff/mozilla/firefox-data-faster-shutdown-pingsender/> (compared to previously over 90 hours). Curious for more detail? #1 <https://chuttenblog.wordpress.com/2017/07/12/latency-improvements-or-yet-another-satisfying-graph/> and #2 <http://reports.telemetry.mozilla.org/post/projects/mainping_beta_latency.kp> should have you covered. We also added a "new-profile" ping <https://www.a2p.it/wordpress/tech-stuff/mozilla/getting-firefox-data-faster-introducing-the-new-profile-ping/>, which gives a clear and timely signal for new clients. There is a new API to record active experiments in Firefox <https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/experiments.html>. This allows annotating experiments or interesting populations in a standard way. The record_in_processes is now required for all histograms. This removes ambiguity about which process they are recorded in. The data documentation moved to a new home: docs.telemetry.mozilla.org. Are there gaps in the documentation you want to see filled? Let us know by filing a bug <https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cf_fx_iteration=---&cf_fx_points=---&component=Documentation%20and%20Knowledge%20Repo%20%28RTMO%29&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-4=X&flag_type-607=X&flag_type-800=X&flag_type-803=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Linux&priority=P3&product=Data%20Platform%20and%20Tools&rep_platform=x86_64&target_milestone=---&version=unspecified> . For datasets, we added telemetry_new_profile_parquet, which makes the data from the "new-profile" ping available. Additionally, the main_summary dataset now includes all scalars and uses a whitelist for histograms, making it easy to add them. Important fields like active_ticks and Quantum release criteria were also added and backfilled. For custom analysis on ATMO <https://analysis.telemetry.mozilla.org/>, cluster lifetimes can now be extended self-serve in the UI. The stability of scheduled job stability also saw major improvements. There were first steps towards supporting Zeppelin notebooks better; they can now be rendered as Markdown <https://pypi.python.org/pypi/python-zeppelin/1.2> in Python. The data tools work is focused on making our data available in a more accessible way. Here, our main tool re:dash <https://sql.telemetry.mozilla.org/> saw multiple improvements. Large queries should no longer show the slow script dialog <https://github.com/mozilla/redash/issues/82>. A new Athena data source was introduced, which contains a subset of our Telemetry-based derived datasets. This brings huge performance and stability improvements over Presto. Finally, scheduled queries can now have an expiration date. What is up next? For the next few months, interesting projects in the pipeline include: - The experiments viewer & pipeline, which will make it much easier to run pref-flipping experiments in Firefox. - Recording new probes from add-ons into the main ping (events, scalars, histograms). - We will define and monitor basic guarantees for the Telemetry client data (like reporting latency ranges). - A re-design of about:telemetry is currently on-going, with more improvements on the way. - A first version of Mission Control will be available, a tool for more real-time release monitoring. - Analyzing the results of the Telemetry survey, (thanks everyone!) to inform our planning. - Extending the main_summary dataset to include all histograms. - Adding a pre-release longitudinal dataset, which will include all measures on those channels. - Looking into additional options to decrease the Firefox data reporting latency. How to contact us. Please reach out to us with any questions or concerns. - You can find us on IRC in #telemetry and #datapipeline. - We are available on slack in #fx-metrics. - The main mailing list for data topics is fx-data-dev <https://mail.mozilla.org/listinfo/fx-data-dev>. - Bugs can be filed in one of these components <https://wiki.mozilla.org/Telemetry#Filing_Bugs>. - You can also find us on Twitter as @MozTelemetry <https://twitter.com/moztelemetry>. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
