Data platform update The data platform team is working on our core Telemetry system, the data pipeline and providing core datasets, with support from the Firefox data engineering and the Data tools team.
To make new features more visible, we intend to provide quarterly updates, starting with this one. What's new in the last few months? On the data collection side, we added scalars <https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html> and added engagement measures <https://www.a2p.it/wordpress/tech-stuff/mozilla/measuring-tab-and-window-usage-in-firefox/> on top. We now support recording histograms in child processes and added categorical histograms <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type> . We improved documentation, starting from the Telemetry wiki page <https://wiki.mozilla.org/Telemetry> and updated onboarding <https://air.mozilla.org/roberto-vitillo-telemetry-and-other-data-sources/>. Further work happened on: - improved client-side documentation <https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/index.html> - an introduction on analyzing telemetry data <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/analyzing_telemetry_data.md> - a guide on choosing a dataset <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/choosing_a_dataset.md> for analysis - updates to examples for the longitudinal dataset <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/longitudinal_examples.md> - exploring how to make Firefox data more discoverable <http://georgf.github.io/fx-data-explorer/> The data pipeline work powers results for re:dash <https://sql.telemetry.mozilla.org/> and custom analysis <https://analysis.telemetry.mozilla.org/> among other things. Notable recent work here includes: - the cross-sectional <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/choosing_a_dataset.md#cross-sectional> dataset is now operational in re:dash <https://sql.telemetry.mozilla.org/> - socorro crash data <https://sql.telemetry.mozilla.org/queries/1170/source#table> is now available in re:dash - the new dataset API <http://python-moztelemetry.readthedocs.io/en/latest/userguide.html#module-moztelemetry.dataset> improves querying raw ping data - the self-serve analysis portal <https://analysis.telemetry.mozilla.org/> was relaunched with improved UX - provided easy access to self-serve real-time analysis <https://gist.github.com/trink/bf7d6e504b3fd937b73095412612ae5a> - launched a knowledge repository <https://reports.telemetry.mozilla.org> to make reports discoverable in one place and easier to review Coming soon For the next few months, interesting projects in the pipeline include: - event telemetry <https://wiki.mozilla.org/Event_Telemetry>, which enables recording event data into Telemetry in a common format <https://docs.google.com/document/d/1hNuS9lUJMvMqgntZXbFA6xZBU9zBpQgo7x73-sXKRpI/edit#heading=h.62xjq5nppfns> - work to decrease data latency, allowing us to make decisions faster - adding Telemetry support for Add-ons - making it easy for new pings to show up in re:dash thanks to direct-to-parquet - enabling efficient lookup of client histories using Hbase - improved alerting <https://robertovitillo.com/2016/12/15/a-martingale-approach-to-detect-changes-in-histograms/> for Telemetry probes - creating a new crash summary dataset, to make it easier to analyze crash data Contact us Please reach out to us with any questions or concerns. You can find us on IRC in #telemetry and #datapipeline. The main mailing list for data topics is fhr-dev <https://mail.mozilla.org/listinfo/fhr-dev>. Bugs can be filed in one of these components <https://wiki.mozilla.org/Telemetry#Filing_Bugs>. You can also find us on Twitter as @MozTelemetry <https://twitter.com/moztelemetry>. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
