Hey guys,
I posted some thoughts to my own blog and am linking to those posts
below. Everything I say on my blog is captured in the summary below,
so feel free to not click through.
----
My biggest worry is that researchers who recruit human subjects assume
that there are huge numbers of Wikipedians for them to survey, and
consequently, they do not need to do a lot of advance survey
preparation because there is no harm from distracting Wikipedians from
their usual volunteer work. This assumption is wrong because actually
almost every researcher recruiting human subjects wants Wikipedians
who are in very short supply. Consequently, researchers do cause harm
to the community by soliciting for volunteer time, and Wikipedia
community benefit is dubious when researchers do not do sufficient
preparation for their work. This is not quite accurate, but if there
were one message I could convey to researchers, it would be "Your
research participant pool only consists of about 30 super busy people
and many other volunteers greatly depend on getting their time. When
you take time from a Wikipedian, you are taking that time away from
other volunteers who really need it, so be respectful of your
intervention in our communities."
----
I do not want a lot of gatekeeping between researchers and the
Wikipedia community, but at the same time, researchers should take
professional pride in their work and take care not to disrupt
Wikimedia community activities.
<http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-on-the-wikimedia-community/>
<http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/>
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli
<[email protected] <mailto:[email protected]>> wrote:
Hi all,
I am a bit late in the game, but since so many questions were
raised about RCom, its scope, its goals, the source of its
authority etc. and I helped coordinate it in the early days I
thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and
best practices around research on Wikimedia projects. People who
joined the committee did so on a volunteer basis and with a
variety of interests by responding to a call for participation
issued by WMF. Despite the fact that the original initiative came
from WMF, its membership almost entirely consisted of non-WMF
researchers and community members (those of us who are now with
Wikimedia had no affiliation with the Foundation when RCom was
launched [1]). RCom work was and remains 100% volunteer-driven,
even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest
identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago/as a/ /group meeting on a regular
basis to discuss joint initiatives/. However, it spawned a large
number of initiatives and workgroups that are still alive and
kicking, some of which have evolved into other projects that are
now only loosely associated with RCom. These include reviewing
subject recruitment requests, but also the Research Newsletter,
which has been published monthly for the last 3 years; countless
initiatives in the area of open access; initiatives to facilitate
Wikimedia data documentation and data discoverability; hackathons
and outreach events aimed at bringing together researchers and
Wikimedia contributors. Subject recruitment reviews and
discussions are still happening, and I believe they provide a
valuable service when you consider that they are entirely run by a
microscopic number of volunteers. I don’t think that the
alternative between “either RCom exists and it functions
effectively or reviews should immediately stop” is well framed or
even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject
recruitment requests?*
Despite the perception that one of RCom’s duties would be to
provide formal approval for research projects, it was never
designed to do so and it never had the power to enforce formal
review decisions. Instead, it was offered as a volunteer support
service in an effort to help minimize disruption, improve the
relevance of research involving Wikimedia contributors, sanity
check the credentials of the researchers, create collaborations
between researchers working on the same topic. The lack of
community or WMF policies to back subject recruitment caused in
the past few years quite some headaches, particularly in those
cases in which recruitment attempts were blocked and referred to
the RCom in order to “obtain formal approval”. The review process
itself was meant to be as inclusive as possible and not restricted
to RCom participants and researchers having their proposal
reviewed were explicitly invited to address any questions or
concerns raised by community members on the talk page. I totally
agree that the way in which the project templates and forms were
designed needs some serious overhaul to remove any indication of a
binding review process or a commitment for reviews to be delivered
within a fixed time frame. I cannot think of any example in which
the review process discriminated some type of projects (say
qualitative research) in favor of other types of research, but I
am sure different research proposals attracted different levels of
participation and interest in the review process. My
recommendation to anyone interested in designing future subject
recruitment processes is to focus on a lightweight review process
open to the largest possible number of community members but
backed by transparent and /enforceable/ policies. It’s a really
hard problem and there is simply no obvious silver bullet solution
that can be found without some experimentation and fault tolerance.
*What about requests for **private data**?*
Private data and technical support requests from WMF are a
different story: they were folded into the list of frequently
asked questions hosted on the RCom section of Meta, but by
definition they require a direct and substantial involvement from
the Foundation: (1) they involve WMF as the legal entity that
would be held liable for disclosing data in breech of its privacy
policies and (2) they involve paid staff resources and need to be
prioritized against a lot of other requests. There are now
dedicated sections on private data on the Wikimedia Privacy Policy
[4] and Data Retention guidelines [5]. Many people, including
myself and other members of the Foundation’s Analytics team,
believe that we should try and collect the minimum amount of
private data that we need in order to operate and study our
projects and make all those types of aggregate/sanitized data that
we can retain indefinitely publicly available to everyone under
open licenses. We’ve already started a process to do so and to
ensure that more data (for example, data collected via site
instrumentation [6]) be exposed via Labs or other APIs, in the
respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make
sure we could get back as much as possible from research involving
WMF resources. It’s been hard to implement these requirements
without policies to enforce them. The suggestion of having more
researchers apply for a slot at the Research Showcase to present
their work is an excellent idea that we should consider. In
general, the Research team at WMF is always interested in hearing
about incentives to drive more interest towards actionable
research on Wikimedia projects.
Dario
[1]
https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=2094818
[2]
https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3] https://meta.wikimedia.org/wiki/Research:WMF_support
[4] https://meta.wikimedia.org/wiki/Privacy_policy
[5] https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker
<[email protected] <mailto:[email protected]>> wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees
still. In other words, there are initiatives that RCOM started
that are alive and successful, but we no longer coordinate as a
larger group. I don't know how else to explain it. I guess you
could say that RCOM is still functioning and that we no longer
require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says
that researchers "must" obtain approval through the process
described. If the wording now needs to be changed to "ought
to" then surely this requires more consensus than your single
message here?
That's a proposed policy. Until it is passed by consensus, the
"must" is a proposed term. I think that it should be "must", but
until that consensus is reached, I'll continue to say that it
"ought to".
Regarding researchers stating what should be regulated, I think
there is a big difference between *deciding what should be
regulated* and *being involved in the discussion of *how* it
should be regulated*. Hence why I welcome participation. What
I'm saying is that you have a vested interest in not being
regulated, but I'd still like to discuss how your activities can
be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to
obtain clarity and I supported this but it was met with
silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth
their time, but they might disagree.
But what is clear is that clarification is required -
especially on the distribution of tasks between Foundation
employees, the research community and Wikimedia editors. And
this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that
quite well. That's why I directed people there. Also, again, I
am not working on RCOM or subject recruitment as a WMF employee.
I do this in my volunteer time. This is true of all of RCOM who
happen to also be staff.
if you want process to be more clearly documented, you also have
to address people like Poitr who would rather not have processes
described in detail. When you guys work out how clearly you want
a process to be described, please let me know. I'm tired of
re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages,
then someone from RCom (or whatever it is now called) should
be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia.
It is not part of RCOM. It would render RCOM irrelevant for
subject recruitment concerns. That's why I started it. I don't
think that RCOM/WMF/researchers should own subject recruitment
review. I think the community being studied should own it and
that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I
don't have much of it, so I focus on keeping the system running
-- and it is -- and improving the system -- which is the proposal
I linked to. If you want something done and other volunteers
don't have time to do it. Do it yourself.
<https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT>
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[email protected]
<mailto:[email protected]>> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker
<[email protected]
<mailto:[email protected]>> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the
beginning of this thread e.g. when others suggested this
might be the case and you refuted them? Also, I'm not sure
what 'functioning as a complete group' actually means. Either
its functioning or its not, surely?
However, we split into sub-committees while we were
still a functioning group. The subject recruitment
sub-committee and newsletter sub-committees are
performing vital functions still.
I never stated that research recruiting needs RCOM
approval. I definitely said that it "ought to" have
RCOM approval.
So, does that mean that is what the policy *ought to* be now?
And do you believe that this should this be the way that the
policy gets decided? Because it isn't right now as far as I
can see. As Kerry noted earlier on, the policy as it stands
[1] says that researchers "must" obtain approval through the
process described. If the wording now needs to be changed to
"ought to" then surely this requires more consensus than your
single message here?
re. the comment that I (and the other researchers?) on this
list shouldn't be the ones to decide what the regulation
should be, I disagree on two counts. a) It seems on the one
hand that you want this to be "self-regulation" i.e. you
invited researchers on this list to join R-COM at the
beginning of this thread, but that you don't think that the
researchers here should be able to determine what to
regulate. I know that you're looking for an inclusive process
but you can't have it both ways: if we are going to help
regulate, then we need to at least help decide how to
regulate. b) Pine suggested a board decision on this earlier
one to obtain clarity and I supported this but it was met
with silence, which is why I followed up.
There are also more than two "review coordinators"
(not not "reviewers") -- it's just that DarTar and I
have accepted the burden of distributing work. When
people are busy, we often coordinate the reviews
ourselves.
I can understand your frustration; I really can! I know that
you've done a lot of really great, prior work on this and I
don't think any of us are saying that we need to throw the
baby out with the bathwater. But what is clear is that
clarification is required - especially on the distribution of
tasks between Foundation employees, the research community
and Wikimedia editors. And this is *especially* true for
people outside this list.
I welcome your edits to make it clear that review is
optional. As you might imagine, I have plenty of
work to do and I appreciate your good-faith
collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly
described on the policy pages, then someone from RCom (or
whatever it is now called) should be the one to change this -
preferably with some discussion. I find it frustrating that
WMF employees are often the ones who make the final policy
pronouncements but then tell others to implement it. And if
we don't do the work, then we're apparently not assuming good
faith.
This is a great opportunity to rejuvenate the process;
hopefully it will eventually be seen that way :)
Best,
Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
_______________________________________________
Wiki-research-l mailing list
[email protected]
<mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
[email protected]
<mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
[email protected]
<mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
[email protected]
<mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814
[email protected] <mailto:[email protected]>
_______________________________________________
Wiki-research-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
