I do, however, have concerns about any research that expects to contact 40,000 editors and involve 1500 of them; that is a very significant portion of our active editorship on the English Wikipedia project.
Commenting on this: out of those targeted 40,000 editors, 30,000 or so are *newly registered users*, so that the sample remains somewhat representative of the diversity we find on en:wp. The rest of it indeed are active contributors. Regards, Jérôme. 2011/12/10 Risker <[email protected]> > On 9 December 2011 22:51, Dario Taraborelli <[email protected] > >wrote: > > > I’d like to give everybody on this list some information on the > > Berkman/Sciences Po research project that many of you have been > discussing > > here. > > > > On Thursday the Wikimedia Foundation announced the launch of a banner to > > support a study led by a team at the Berkman Center/Sciences Po and > > recruiting participants from the English Wikipedia editor community [1]. > > The banner was taken down within hours of its launch after concerns > raised > > in various community forums (the Admin Noticeboard [2], the Village Pump > > Tech [3], various IRC channels and mailing lists such as foundation-l [4] > > and internal-l [5]) that the design was confusing, that it was perceived > as > > a commercial ad and that the community approval process and privacy terms > > were unclear and hardly visible. > > > > Here’s what happened until the launch, what went wrong after the launch > > and what we are planning to do next. > > > > ==The prequel== > > This proposal went through a long review process, involving community > > forums, the Research Committee and various WMF departments since early > 2010. > > > > The Berkman research team first approached WMF to discuss this study in > > January 2010. They suggested a protocol to recruit English Wikipedia > > contributors to participate in an early version of this study by March > 2010 > > and posted a proposal to the Administrators’ noticeboard to get community > > feedback [6]. The community response at that time opposed the proposed > > recruitment protocol (posting individual invitation messages on user talk > > pages). It was suggested instead that the recruitment should be handled > > through a CentralNotice banner to be displayed to registered editors, but > > concerns were raised on how to minimize the disruption. > > > > To address these concerns, the proposal went through a full review with > > the Wikimedia Research Committee, that was completed in July 2011. The > RCom > > evaluated the methods, the recruitment strategy, the language used in the > > survey and approved the proposal pending a final solution for the > > recruitment taking into account the concerns expressed by the community > [7]. > > > > Based on suggestions made by community members (e.g. [8]) the research > > team started to work on a technical solution to selectively display a > > banner to a subset of registered editors of the English Wikipedia meeting > > certain eligibility conditions. WMF agreed to invest engineering effort > > into a system that would allow CentralNotice to serve contents to a > > specific set of editors – functionality that would benefit future > > campaigns run by the community, chapters or the Foundation [9] [10]. > > > > A new CentralNotice backend was then designed to look up various editor > > metrics (i.e. number of contributions, account registration date and > editor > > privileges) – all public information available from our database – and to > > perform a participant eligibility check against these metrics. A banner > > would then be displayed to eligible participants, posting the above data > > (user ID + editor metrics) along with a unique token to the server > hosting > > the survey upon clicking. On the landing page of the survey, participants > > would have the possibility to read the privacy terms of the survey and > > decide whether to take it or not. > > > > Throughout the review process of this recruitment protocol, the research > > team received constant feedback from the Foundation’s legal team, the > > community department, the tech department and the communication team > before > > the campaign went live. > > > > The campaign was announced in the CentralNotice calendar one month before > > its launch [11] and the launch was with a post on the Foundation’s blog. > > The banner was enabled on December 8 at 11:00pm UTC. 800+ participants > > completed the study within a few hours since its launch. The banner was > > then taken down by a meta-admin a few hours after the launch due to the > > concerns described above. > > > > So what went wrong? > > > > ==A few explanations we owe you== > > > > • Is the Foundation running ads? > > No, this banner is a recruitment campaign for a research project that has > > been thoroughly reviewed by the Research Committee. We have a long > > tradition of supporting recruitment for research about our communities > via > > various sitenotices. The methodology of this project is sound and the > > recruitment method less invasive than thousands of individual messages > > posted on user talk pages. We believe this research will help advance our > > understanding of the dynamics of participation in our projects. Receiving > > support by the Research Committee implies that all published output and > > anonymized data produced by this study will be made available under open > > licenses. [12] The banner also received full Wikimedia Foundation > approval > > before its launch. > > > > • Is this campaign conflicting with the fundraiser? > > No, this banner is running only for a subset of logged-in editors for > whom > > the main fundraiser campaign has already been taken down. We carefully > > timed this campaign to minimize the impact on the fundraiser and we > > scheduled it on the CentralNotice calendar with a month notice for this > > reason. > > > > • Is this campaign running at 100% on the English Wikipedia? > > No, the banner has been designed to target a subsample of the English > > Wikipedia registered editor population. Based on estimates by the > research > > team, the eligibility criteria apply to about 10,000 very active > > contributors and about 30,000 new editors of the English Wikipedia. The > > target number of completed responses is 1500. > > > > • Why does the banner include logos of organizations not affiliated with > > Wikimedia? > > The design of the banner was based on the decision to give participants > as > > much information as possible about the research team running the project > > and to set accurate expectations about the study. > > > > > > ==What we are doing now== > > > > We realize that despite an extensive review, the launch of this project > > was not fully advertised on community forums. We plan to shortly resume > the > > campaign (for the time needed by the researchers to complete their > > responses) after a full redesign of the recruitment protocol in order to > > address the concerns raised by many of you over the last 24 hours. Here’s > > what we are doing: > > > > • Provide you with better information about the project > > We asked the research team to promptly set up a FAQ section on the > project > > page on Meta [13], and to be available to address any concern about the > > study on the discussion page of this project. The project page on Meta > will > > be linked from the recruitment banner itself. > > > > • Redesign the banner > > We understand that the banner design has been interpreted by some as > > ad-like (even if the goal was to make clear that this study was not being > > run by WMF, as it implied a redirection to a third party website for > > performing the experiment). In coordination with the research team, we > will > > come up with a banner design that will be more in line with the concerns > > expressed by the community (for instance by removing the logos from the > > banner). > > > > • Make privacy terms as transparent as possible > > Upon clicking on the banner, participants accept to share their username, > > edit count and user privileges with the research team. The previous > version > > didn’t make it explicit and we are working to address this problem. To > make > > the process totally transparent we will make the acceptance of these > terms > > explicit in the banner itself. > > > > Once redirected to the landing page, participants will have to accept the > > terms of participation in order to enter the study. The project is funded > > by the European Research Council: the data collected in this study is > > subject to strict European privacy protocols. The research team will use > > this data for research purposes only. The research team is not exposed to > > and does not record participants’ IP addresses. > > > > ==How you can help== > > > > We would like to hear from you on the redesign of the banner to make sure > > it meets the expectations of the community and doesn’t lend itself to any > > kind of confusion. We will post the new banners to Meta and try to > address > > all pending questions before we resume the campaign. > > > > This is one of the first times we’re supporting a complex, important > > research initiative like this one, and I apologize for the bumps in the > > road. We believe that supporting research is part of our mission: it > helps > > advance our understanding of ourselves. So thanks again for all support > you > > can give in making this a success. > > > > > > Dario Taraborelli > > Senior Research Analyst, Wikimedia Foundation > > > > [1] http://blog.wikimedia.org/2011/12/08/experiment-decision-making/ > > [2] > > > http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents#Harvard.2FScience_Po_Adverts > > [3] > > > http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_banner_Wikipedia_Research_Committee > > [4] > > > http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html > > [5] > > > https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842.html > > [6] > > > http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive222#Researchers_requesting_administrators.E2.80.99_advices_to_launch_a_study > > [7] > > > http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions_and_Behavior#RCom_review > > [8] > http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html > > [9] > http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html > > [10] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines > > [11] > > > http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldid=3056067 > > [12] http://meta.wikimedia.org/wiki/Research:Subject_recruitment > > [13] > > > meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavior > > > > > > Dario, nobody in any of the discussions on the English Wikipedia (whose > editors are the target of this research project) suggested that a *central > site notice* be used for this or any other research project. The > discussion in April 2011 showed consensus opposition to bot-delivered talk > page notices. One editor involved in the discussion suggested "site > notices" (which I believe were interpreted by the participants to mean a > local site notice) and two others mentioned watchlist notices. The > subsequent discussion about central notices discussed the possibility of > developing a narrowcasting ability for such notices, and discussed > specifically notices directly related to WMF projects or activities. It > did not, in any way, address the concept of using a central notice to > promote a non-WMF activity (such as this research project). Indeed, this is > the first use of a central notice for anything not directly related to an > obviously WMF-related activity. > > The ability to narrowcast central notices is a positive advancement; > however, the processes for proposing and determining the appropriateness of > a narrowcast are poorly publicized, and some of them don't appear to have > even existed until after this notice was taken down. There are still no > community-approved guidelines for the use of central notices, although a > draft one is currently up for comment.[1] An RFC initiated in August 2010 > with respect to "global banners"/central notices, well in advance of the > development of the narrowcasting ability, strongly supported consensus > approval on Meta for non-fundraising global banners.[2] Now that there is > the ability to target central notices to only one project or community, it > is extremely important that that community be directly notified of such > discussion - a discussion that never took place in any public forum that I > can see in advance of this central notice being activated. > > The links above include one to a private mailing list that the majority of > readers of this list have no access to. You may want to consider asking the > persons whose contributions are contained in that particular message to > grant permission for it to be reproduced here so that the rest of us aren't > left in the dark about who said what. > > I don't begridge scholars carrying out approved research with Wiki?edians > who volunteer to do so; in fact, I've responded to several requests myself. > I do, however, have concerns about any research that expects to contact > 40,000 editors and involve 1500 of them; that is a very significant portion > of our active editorship on the English Wikipedia project. I'm curious to > know if scholars have shown much interest in studying some of the other > projects as much as they've initiated studies on enwp. > > Risker/Anne > > > > [1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines > http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners > _______________________________________________ > foundation-l mailing list > [email protected] > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > _______________________________________________ foundation-l mailing list [email protected] Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
