A/B testing for major interface changes isn't such a good idea, I think. It's great for small things like login fields that people aren't likely to notice, but once you get into major interface elements you're going to get quite a few people confused or frustrated.
"Why is your Wikipedia different to mine?", "Why does this not look like the help page says?", etc. I'm not sure the side-effects would be worth the data. Andrew. On 16 July 2014 13:01, Erik Zachte <[email protected]> wrote: > Disclaimer: I haven't been following the discussion > > Full disclosure: 1) I personally like the new feature 2) I value community > opinion a lot > > > > Just my 2 cents on methodology: > > It seems to me that if we could present half of a target population with old > and half with new settings from the outset (e.g. by focussing on new users > only), then the outcome would be more convincing. > > In the current proposal if 40% would choose to switch , is that because 60% > like the new settings better, or because they can't be bothered at that very > moment to explore and make up their mind, or for any other reason cling to > what they are offered initially. > > > > Erik > > > > > > From: [email protected] > [mailto:[email protected]] On Behalf Of Gilles Dubuc > Sent: Wednesday, July 16, 2014 14:00 > To: A mailing list for the Analytics Team at WMF and everybody who has an > interest in Wikipedia and analytics. > Subject: Re: [Analytics] Media Viewer User Preference Data > > > > Could we sample only logged-in users? > > > > I think the idea is to measure both separately, because the default might > end up being different for logged-in users and for anons, depending on the > preference ratio for each. > > > > On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz <[email protected]> wrote: > >>a) aim to track total users who enable/disable Media Viewer, rather than >> just events > >>b) switch to a 3-state preference setting: enabled / disabled / default > >>c) try to measure the total number of users in each group (instead of daily >> events) > > > > I assume we are talking about logging stuff for logged in users and not > logged in users. > > If that is the case this strategy will dwarf editor data with reader data, > in the RFC most concerns seem to be about editor workflow. Could we sample > only logged-in users? That way we will get a sample of data that better > represents editor's interaction with media viewer and it will be a lot > easier to quantify our results. > > > > > >>Once we figure out a practical way to implement this request together, we >> will present it to community members to >discuss its impact on the RfC, and >> agree on acceptance criteria (e.g. keep Media Viewer enabled by default >> unless >a 51% majority of users disables the feature on Enwiki by September >> 30?). > > > > Perhaps if we want the analytics to impact the RFC we should work with the > community to come up with a survey strategy they find aceptable? > > > > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin <[email protected]> > wrote: > > Dear Analytics Team, > > > > As discussed with Toby, Dario and Aaron, we would be grateful for your > guidance to collect user preference data for Media Viewer (1), which is > being challenged in RfCs on English (2) and German (3) Wikipedias, as well > as on Wikimedia Commons (4). > > > > The English RfC closed last week, requesting that we disable Media Viewer > right away. The multimedia team, in consultation with community and legal > teams, is not comfortable disabling the feature on the basis of this small > RfC, which we believe does not accurately represent the views of the > millions of users whom we serve. > > > > To address this issue, we would like to conduct a wider outreach to collect > more accurate data about user preferences than either this RfC or our > optional surveys can provide. We propose to develop a more prominent viewing > options panel (5) that would make it very easy to switch quickly to your > preferred viewing mode, as shown in this prototype (6). Right after launch > of this new feature, all users would be shown this prominent panel and asked > to select their favorite viewing option. Once we have collected and analyzed > that data over a period of a month or two, we would be able to make more > informed decisions with our community on whether or not to keep the feature > enabled -- based on actual responses from all users, rather than > speculation. > > > > With your help, we have already developed a basic dashboard that tracks > Media Viewer opt-in/out events (8), based on clicks on the less prominent > ‘Disable’ feature at the bottom of the metadata panel. We propose to modify > this dashboard to support this new initiative, as outlined in this ticket > (7), with these changes: > > a) aim to track total users who enable/disable Media Viewer, rather than > just events > > b) switch to a 3-state preference setting: enabled / disabled / default > > c) try to measure the total number of users in each group (instead of daily > events) > > > > Also note that most users would start in the default state, showing them > Media Viewer, since it is enabled by default — but after 10 or so image > views, the viewing options panel would appear automatically, asking them to > either enable or disable the feature. After they have made their selection, > the panel would remain accessible (but much less prominent) and they can > still switch state (but can't go back to default). > > > > We could use your guidance on the right way of logging total logged-out > users, so we can use them as a basis for percentages: should we log > enabled/disabled state on some specified event/action: page load? thumbnail > click? first site visit of the day? — or do you recommend another method? We > would want to collect this data for a month or two, so we don’t only capture > the immediate responses from the most active users, but also those of less > frequent visitors. > > > > We would also appreciate your comments on the specific tickets (5) and (6), > with any recommendations for improvement. Keep in mind that we would like to > respond quickly to our community’s concerns and have limited resources, so > we would prefer to get this work done in the next week or two. And it will > take a couple months to develop, release and collect enough data -- so even > if we start tomorrow, we may not have conclusive data to share until > September. So time is of the essence. :) > > > > Gergo is spearheading this project, and may have more technical questions > for you. But I wanted to give an overview from a product perspective, before > we dive in to the implementation details. Once we figure out a practical way > to implement this request together, we will present it to community members > to discuss its impact on the RfC, and agree on acceptance criteria (e.g. > keep Media Viewer enabled by default unless a 51% majority of users disables > the feature on Enwiki by September 30?). > > > > Thanks in advance for your advice. We look forward to working with you soon > on this important project, which could impact other features now in > development. > > > > All the best, > > > > > > > > Fabrice > > > > > > (1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer > > (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC > > (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter > > (4) > https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer_software_feature > > (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 > > (6) > http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/index.html > > (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793 > > > > > > _______________________________ > > > Fabrice Florin > > Product Manager, Multimedia > > Wikimedia Foundation > > > > https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF) > > > > > > > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > > > > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > > > > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > -- - Andrew Gray [email protected] _______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
