https://bugzilla.wikimedia.org/show_bug.cgi?id=49142
Web browser: ---
Bug ID: 49142
Summary: Notifications Productivity A/B Test
Product: MediaWiki extensions
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Echo
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected],
[email protected]
Classification: Unclassified
Mobile Platform: ---
To understand how Notifications impact the behavior of new editors in
Wikipedia, we are planning an A/B test the week of June 10, which will let us
study their productivity by comparing two cohorts, as described below, and on
this page:
http://meta.wikimedia.org/wiki/Research:Notifications/Experiment_1
The research question we're trying to answer is "Do the new Echo Notifications
help people become more productive?" This answer will be determined primarily
by analyzing total edits from each cohort -- and number of edits that were not
reverted within a week.
To that end, we need to accomplish two goals this week:
1) divide new users who register after June 11 into two buckets:
a) Pre-Echo: All notification types disabled (legacy talk notification enabled)
b) Echo-Current: Existing Echo notification defaults for new users
Users will be bucketed based on the last digit of their userID (even digits =
pre-Echo cohort; odd digits = Echo-Current cohort).
2) modify the 'hidden preference' to disable all email notifications from Echo,
and to restore the previous talk page messages users were getting before Echo.
As a result, the first 'Pre-Echo' cohort will have both web and email
preferences disabled (including Echo emails), using a 'hidden preference' that
is currently part of the Echo code; they will see the OBOD for a one-week
period, to match conditions prior to Echo. The second 'Echo-Current' cohort
will get exactly what they get today (with edit reverts disabled for new users,
both on web and email).
After a one-month period, we will change the 'hidden preference' for the first
cohort, so that they get the same 'Echo-Current' settings as new users do
today.
Note that this study needs to be conducted the week of June 10, before the
Visual Editor research starts impacting our ability to bucket users into
cohorts, which is why this ticket has a high priority. The reason we want to
conduct this study right away is that we want to have some productivity data
available when discussing the deployment of Echo on other language wikis later
this summer.
Dario Taraborelli and Aaron Halfaker are the primary investigators for this
research, and may provide more details on this thread, as needed.
P.S.: If clicktracking data is available by next week, we would like to also
study the clickthrough rate of the two cohorts as part of this test, though
that is not a hard requirement.
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
