Henri Sivonen wrote:
Do we have a formalized way to do A/B testing with telemetry? That is,
assuming that there is a telemetry probe that measures problem
symptoms and a boolean pref for turning on a potential solution, is
there a way the declare the pref as something that telemetry queries
can be constrained by so that it would be possible to compare the
symptom probe with and without the potential solution?
We don't have anything formal like that. We leave it up to authors(eg
their input telemetry + analysis scripts) to decide on optimal a/b
patterns. Once we have enough of such patterns, might formalize a few
common ones.
If not, is there a better way to do this than duplicating probes and
checking the pref to see which probe should be fed?
I'm a bit late here. This one 'correct' way of doing A/B testing.
Alternatively you can send the value of that pref in the .info section
of telemetry. See one of many fields in
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetryPing.js#332
Taras
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
