Hi! I recently came across a blog post [1] about the ethics of ad-blocking and how websites are optimized for capturing the user's attention, instead of supporting the user's intention. This made me think about our own metrics and whether they might be (unintentionally) biased towards the wrong goals.
For example, app-usage time is one of the things we're interested in, but it seems irrelevant or even misleading: a badly-designed app has a longer app-usage time than an app that let's the user get work done quickly. From this metric's data, we'd choose the wrong design. What we should try to capture is not so much state (app-usage time, number of open apps), but process (usage patterns within an app, patterns when switching among apps). This seems considerably harder and has stricter requirements for the user's privacy. But the results should be a lot better. Best regards Thomas [1] http://blog.practicalethics.ox.ac.uk/2015/10/why-its-ok-to-block-ads/ Am 03.11.2015 um 12:52 schrieb Wilfred Mathanaraj: > Forwarding email from Ravi - while he is waiting for permission to > post to list. > > BR > Wilfred > > --- > FxOS Product Management > Mozilla Corp., UK > > > > >> Begin forwarded message: >> >> *From: *Ravikumar Dandu <[email protected] <mailto:[email protected]>> >> *Date: *3 November 2015 at 11:49:37 GMT >> *To: *[email protected] <mailto:[email protected]> >> *Cc: *Wilfred Mathanaraj <[email protected] >> <mailto:[email protected]>>, Tamara Hills <[email protected] >> <mailto:[email protected]>>, Francisco Jordano <[email protected] >> <mailto:[email protected]>>, David Bialer <[email protected] >> <mailto:[email protected]>>, Dominik Strohmeier >> <[email protected] <mailto:[email protected]>> >> *Subject: **A/B testing on FxOS* >> >> Hi, >> >> This email is for collecting ideas/thoughts for A/B testing on FxOS. >> >> Device metrics is collecting information on app usage and user >> engagement through opt-in. Moving forward, we want to improve our >> processes for design and features based on real usage feedback of >> FxOS. As part of this we want to take action based on feedback from >> users (foxfooders/end users) - within our privacy policies, and >> legal limits - to more clearly guide our development process. >> >> To move this to the next level, A/B testing will play a crucial part. >> A/B testing will enable testing multiple variants of apps/ux against >> each other, so we can determine whether a particular change (a new >> feature or a change to a feature) is improving the >> engagement/experience of user >> >> This is not a simple process and involves various teams to be in sync >> in order to have the right results to guide us. By measuring the >> impact that changes have on metrics such as App Usage, time spent on >> panels etc, we can ensure that changes/features produces positive >> results. We also need to identify processes for targeting user >> segments, and getting the right/random mix for the different >> versions. Also, need to plan to adapt our OS to the results we obtain. >> >> Product team is starting to gather information and requirements on >> how we can implement A/B testing and guide our design and development >> work. We would like to gather your thoughts and feedback on: >> >> 1. >> Results of A/B testing that you would like to see for the >> particular development you are doing >> 2. >> Ideas for A/B testing framework (eg: Addon to deliver the >> proposed change to an app for certain users) >> 3. >> Tools that you've previously used or are using (eg: Optimizely >> for webapps) >> >> >> This will help us help focus our resources in the appropriate place >> to get biggest bang for our development buck. >> >> We've started an initial draft document on FxOS A/B Testing Product >> Requirementsat >> https://docs.google.com/document/d/1BZZ42iWojJa4WJ2AUHMav7aFZ0v_fIZ6ynIYWTUIcYc/edit >> It is editable, so please add/comment on it. >> >> Please also feel free to reach out to me and Wilfred to discuss >> ideas/feedback. We'll organize meetings with individual teams to >> gather more input. >> >> Regards >> Ravi & Wilfred >> <https://docs.google.com/document/d/1BZZ42iWojJa4WJ2AUHMav7aFZ0v_fIZ6ynIYWTUIcYc/edit> > > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos _______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
