Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Asaf Bartov]

On Sat, Jan 25, 2014 at 6:10 AM, Federico Leva (Nemo) wrote:

> Asaf Bartov, 25/01/2014 03:12:
>
>  Thanks!  The most interesting datum I've noticed so far is the rate of
>> active editor (5+ edits/month) retention among new contributors after
>> six months, which is about 0.4%.
>>
>
> Is "active editor retention" a metric defined somewhere?
> It's not in  standardized_editor_classes>.


I guess it isn't, and it seems to me it should be.  But I'm not a
researcher, and I leave it to the research community to figure out if this
is an interesting metric for them.  I can tell you it's an interesting
metric to _me_, because making 1 edit some time within six months of
registration is _much_ less meaningful than making at least 5 edits a month
(which is the active editor definition).

(I note that https://meta.wikimedia.org/wiki/Research:Active_editor is not
in the above category, and is also not defined on Meta.  Once again, I'm
not taking it upon myself to fix those pages, as I feel they are !owned by
others.  But it behooves us to get these a little more aligned.  CCing
Dario and crossing my fingers.

   Asaf
-- 
Asaf Bartov
Wikimedia Foundation 

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
https://donate.wikimedia.org
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Jaime Anstee]

Hello Nemo,

Sorry for the confusing in-line, I entered the conversation late, and
appreciate your patience. I do not think I am understanding what definition
information you are asking for.  The definition of  how we defined the
windows for 3 and 6 month follow-up is written on the overview page in
reference note number 1 and I have now added a second reference to the
mediawiki reference page I shared as that is what the 5+/month is
referenced from.

The way the quarters are defined is also referenced in the main text of the
report with some additional note references. As we are looking at retention
of "active"editors at different follow-up points defined at 3 and 6 months
(as indication of short and intermediate term outcomes) we have outlined
how this was done as a more-or-less quarterly follow-up following the end
of program events.  So yes, the first quarter post event is October through
December and the second January through March.
Assessments are from Wikimetrics and based on a start day and time and end
day and time, not on account creation as these other metrics do as we are
evaluating outreach programs that have event dates and variable account
creation dates. The goal of the metric that was was mainly to assess the
number of "active" contributors that are actually recruited, survival has
been included mostly for added understanding of the how this "active"
requirement compares to survival (any activity in the window of interest).

Still, I am not sure if this is answering you either. If you are simply
suggesting we not look at retention at 3 and 6 month intervals but rather
at 1 week, 2 months, and 6 months, as exampled on your page link, I must
say that this is the first time for that suggestion  and that would be
something to discuss on the report talk page (or overview talk page).

If you have additional questions or comments, please share them on the
report talk pages where we would like these conversations to take place and
be recorded.

Thank you,

Jaime


-- 

Jaime Anstee, Ph.D
Program Evaluation Specialist
Wikimedia Foundation
+1.415.839.6885 ext 6869
www.wikimediafoundation.org

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
*https://donate.wikimedia.org *



On Sat, Jan 25, 2014 at 2:18 PM, Federico Leva (Nemo) wrote:

> Hi Jaime.
> Your inline replying format highly confuses me, but I tried to reconstruct
> what you meant to say and I believe it's what below.
>
> Jaime Anstee, 25/01/2014 20:19:
>
>  On Sat, Jan 25, 2014 at 6:10 AM, Federico Leva (Nemo)
>>
> >
>
>> Is "active editor retention" a metric defined somewhere?
>>>
>>> It's not in >> standardized_editor_classes>.  [...]
>>>
>>
>> Yes -
>> https://www.mediawiki.org/wiki/Analytics/Metric_definitions#Active_editor
>>
>> Thank you for asking, and mentioning that it is not linked in the
>> section.  I will reference that link there.
>>
>
> I know what an "active editor" is, you'll see in the history that I
> contributed to the section you linked. I asked about the "rate of active
> editor retention" as Asaf called it, or... "Six-month follow-up Active
> second quarter of retention follow-up" (??) as mentioned in the page.
> Apart from the confusing terminology, you'll note that it doesn't match
> with the (draft?) definition  wiki/Research:Surviving_new_editor> even with non-standard n=1, m=15,
> t_2=90 days and t_1 < 0 unless you arbitrarily pose T = end of September.
> You may want to align definitions a bit.
>
>
>  I would like to point out that
>> the overview page to the set of reports does contain a lot of important
>> information about how things were defined and are reported in each
>> programs
>> evaluation report.
>>
>
> If you mean  portal/Library/Overview#Recruitment_and_retention>, it doesn't contain
> any definition for the metric.
>
> Nemo
>
>
> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Federico Leva (Nemo)]

Hi Jaime.
Your inline replying format highly confuses me, but I tried to 
reconstruct what you meant to say and I believe it's what below.


Jaime Anstee, 25/01/2014 20:19:

On Sat, Jan 25, 2014 at 6:10 AM, Federico Leva (Nemo)

>

Is "active editor retention" a metric defined somewhere?
It's not in .  [...]


Yes -
https://www.mediawiki.org/wiki/Analytics/Metric_definitions#Active_editor

Thank you for asking, and mentioning that it is not linked in the
section.  I will reference that link there.


I know what an "active editor" is, you'll see in the history that I 
contributed to the section you linked. I asked about the "rate of active 
editor retention" as Asaf called it, or... "Six-month follow-up Active 
second quarter of retention follow-up" (??) as mentioned in the page.
Apart from the confusing terminology, you'll note that it doesn't match 
with the (draft?) definition 
 even 
with non-standard n=1, m=15, t_2=90 days and t_1 < 0 unless you 
arbitrarily pose T = end of September. You may want to align definitions 
a bit.



I would like to point out that
the overview page to the set of reports does contain a lot of important
information about how things were defined and are reported in each programs
evaluation report.


If you mean 
, 
it doesn't contain any definition for the metric.


Nemo

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Jaime Anstee]

On Sat, Jan 25, 2014 at 9:06 AM, Strainu  wrote:

> 2014-01-25 rupert THURNER :
> > On Sat, Jan 25, 2014 at 4:37 PM, LiAnna Davis 
> wrote:
> >
> >> On Sat, Jan 25, 2014 at 4:45 AM, rupert THURNER <
> rupert.thur...@gmail.com>wrote:
> >>>
> >>> lianna, where did you get that number from?
> >>>
> >>>... i of course ment how you calculated the number.
> > there must be some tools available to do this, and ideally they are
> online
> > for everybody. 60'000 images used is such an impressive number ... it
> still
> > leaves me speachless :)
> >
>

 For image use counts you can use
http://tools.wmflabs.org/glamtools/glamorous.php to look up usage for any
commons category name
For quality ratings and image use you can use
https://tools.wmflabs.org/catscan2/catscan2.php to look for template
inclusion on the file pages, here is a link to a learning pattern we posted
about it:
https://meta.wikimedia.org/wiki/Grants:Learning_patterns/Counting_Featured_pictures,_Quality_and_Valued_images_in_Wikimedia_Commons


Thanks,

Jaime

-- 

Jaime Anstee, Ph.D
Program Evaluation Specialist
Wikimedia Foundation
+1.415.839.6885 ext 6869
www.wikimediafoundation.org

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
*https://donate.wikimedia.org *




> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Jaime Anstee]

On Sat, Jan 25, 2014 at 6:10 AM, Federico Leva (Nemo) 
 wrote:

> Asaf Bartov, 25/01/2014 03:12:
>
> Thanks!  The most interesting datum I've noticed so far is the rate of
>> active editor (5+ edits/month) retention among new contributors after
>> six months, which is about 0.4%.
>>
>
> Is "active editor retention" a metric defined somewhere?
> It's not in  standardized_editor_classes>.


>> https://meta.wikimedia.org/wiki/Programs:Evaluation_
>> portal/Library/WLM#New_users
>>
>> >>Yes -
https://www.mediawiki.org/wiki/Analytics/Metric_definitions#Active_editor

>>Thank you for asking, and mentioning that it is not linked in the
section.  I will reference that link there.  I would like to point out that
the overview page to the set of reports does contain a lot of important
information about how things were defined and are reported in each programs
evaluation report.



> That (along with the easy-to-measure and already known sheer quanities
>> of photos generated) seems to me to be the most robust number to use in
>> evaluating WLM (and comparable) funding requests.  0.4% may be
>> considered worthwhile or not enough, depending on local contexts, so I'm
>> not drawing a universal conclusion from it, just noting it.
>>
>
> I find the return rate for the next event even more interesting!

Yes, it is important to remember that there are many other potential
outcomes that have not been measured and are not included in this first
round of evaluation reporting.

See http://www.wikilovesmonuments.org/wiki-loves-monuments-footprint/ for
> more.
>
> >>Thank you for sharing this link

Jaime

-- 

Jaime Anstee, Ph.D
Program Evaluation Specialist
Wikimedia Foundation
+1.415.839.6885 ext 6869
www.wikimediafoundation.org

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
*https://donate.wikimedia.org *
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Strainu]

2014-01-25 rupert THURNER :
> On Sat, Jan 25, 2014 at 4:37 PM, LiAnna Davis  wrote:
>
>> On Sat, Jan 25, 2014 at 4:45 AM, rupert THURNER 
>> wrote:
>>>
>>> lianna, where did you get that number from?
>>>
>>> I want credit for this report to go where it belongs -- the Program
>> Evaluation team! I'm just assisting with publishing and distributing the
>> report; the work in it is a whole team effort, including contributions from
>> Frank Schulenburg, Dr. Jaime Anstee, Yuan Li, and Edward Galvez. Their hard
>> work went into data collection, reporting, and analysis; I just did some
>> minor copy editing and publicizing. :)
>>
>> But that being said, I encourage everyone to post comments and questions
>> on the talk page, which is where the program evaluation team will be more
>> likely to see and answer them, rather than having parallel discussions on
>> multiple lists:
>> https://meta.wikimedia.org/wiki/Programs_talk:Evaluation_portal/Library/WLM
>>
>>
>> oh, my english is bad, i of course ment how you calculated the number.
> there must be some tools available to do this, and ideally they are online
> for everybody. 60'000 images used is such an impressive number ... it still
> leaves me speachless :)
>


Do you mean the number of images? You might like to analyse the stats page:
https://toolserver.org/~emijrp/wlm/stats.php

It might not match 100% what the report says, as some images were
deleted, but it should be pretty close. You also have raw data if you
want to make your own statistics.

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[rupert THURNER]

On Sat, Jan 25, 2014 at 4:37 PM, LiAnna Davis  wrote:

> On Sat, Jan 25, 2014 at 4:45 AM, rupert THURNER 
> wrote:
>>
>> lianna, where did you get that number from?
>>
>> I want credit for this report to go where it belongs -- the Program
> Evaluation team! I'm just assisting with publishing and distributing the
> report; the work in it is a whole team effort, including contributions from
> Frank Schulenburg, Dr. Jaime Anstee, Yuan Li, and Edward Galvez. Their hard
> work went into data collection, reporting, and analysis; I just did some
> minor copy editing and publicizing. :)
>
> But that being said, I encourage everyone to post comments and questions
> on the talk page, which is where the program evaluation team will be more
> likely to see and answer them, rather than having parallel discussions on
> multiple lists:
> https://meta.wikimedia.org/wiki/Programs_talk:Evaluation_portal/Library/WLM
>
>
> oh, my english is bad, i of course ment how you calculated the number.
there must be some tools available to do this, and ideally they are online
for everybody. 60'000 images used is such an impressive number ... it still
leaves me speachless :)

rupert.
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[LiAnna Davis]

On Sat, Jan 25, 2014 at 4:45 AM, rupert THURNER
wrote:
>
> lianna, where did you get that number from?
>
> I want credit for this report to go where it belongs -- the Program
Evaluation team! I'm just assisting with publishing and distributing the
report; the work in it is a whole team effort, including contributions from
Frank Schulenburg, Dr. Jaime Anstee, Yuan Li, and Edward Galvez. Their hard
work went into data collection, reporting, and analysis; I just did some
minor copy editing and publicizing. :)

But that being said, I encourage everyone to post comments and questions on
the talk page, which is where the program evaluation team will be more
likely to see and answer them, rather than having parallel discussions on
multiple lists:
https://meta.wikimedia.org/wiki/Programs_talk:Evaluation_portal/Library/WLM

LiAnna


-- 
LiAnna Davis
Wikipedia Education Program
Wikimedia Foundation
http://education.wikimedia.org
+1-(415) 839-6885 x6649
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Federico Leva (Nemo)]

Asaf Bartov, 25/01/2014 03:12:

Thanks!  The most interesting datum I've noticed so far is the rate of
active editor (5+ edits/month) retention among new contributors after
six months, which is about 0.4%.


Is "active editor retention" a metric defined somewhere?
It's not in 
.




https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/WLM#New_users

That (along with the easy-to-measure and already known sheer quanities
of photos generated) seems to me to be the most robust number to use in
evaluating WLM (and comparable) funding requests.  0.4% may be
considered worthwhile or not enough, depending on local contexts, so I'm
not drawing a universal conclusion from it, just noting it.


See http://www.wikilovesmonuments.org/wiki-loves-monuments-footprint/ 
for more.


Nemo

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [Wiki Loves Monuments] New Program Evaluation report: Wiki Loves Monuments

[Asaf Bartov]

Thanks!  The most interesting datum I've noticed so far is the rate of
active editor (5+ edits/month) retention among new contributors after six
months, which is about 0.4%.

https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/WLM#New_users

That (along with the easy-to-measure and already known sheer quanities of
photos generated) seems to me to be the most robust number to use in
evaluating WLM (and comparable) funding requests.  0.4% may be considered
worthwhile or not enough, depending on local contexts, so I'm not drawing a
universal conclusion from it, just noting it.

(just thinking out loud here.)

   A.


On Fri, Jan 24, 2014 at 4:14 PM, Samuel Klein  wrote:

> Thanks!  Great to see this summary.
>
>
> On Fri, Jan 24, 2014 at 6:00 PM, LiAnna Davis wrote:
>
>> (please excuse cross-posting)
>>
>> Hi everyone!
>>
>> I posted the latest in the string of program evaluation reports on Meta
>> today, on Wiki Loves Monuments:
>> https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/WLM
>>
>> Highlights of the report include:
>> * About 17% of the images uploaded through Wiki Loves Monuments 2012 are
>> in use on our projects.
>> * The majority of Wiki Loves Monuments participants are new users;
>> however, the survival rate of new users is low (1.7% of the 2012
>> participants made at least one edit and 1.4% uploaded at least one new file
>> to Commons six months after the event).
>> * Half of the existing editors who participated in Wiki Loves Monuments
>> 2012 also participated in Wiki Loves Monuments 2013, with new users making
>> up for the other half.
>> * The global Wiki Loves Monuments organizing team helps support Wiki
>> Loves Monuments organizers around the world, providing replication
>> opportunities via direct support. This makes it easier for someone to
>> organize their own Wiki Loves Monuments contest.
>>
>> Questions are welcomed and encouraged on the talk page.
>>
>> On behalf of the Program Evaluation team,
>>
>> LiAnna
>>
>> --
>> LiAnna Davis
>> Wikipedia Education Program
>> Wikimedia Foundation
>> http://education.wikimedia.org
>> +1-(415) 839-6885 x6649
>>
>>
>> ___
>> Wiki Loves Monuments mailing list
>> wikilovesmonume...@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
>> http://www.wikilovesmonuments.org
>>
>
>
>
> --
> Samuel Klein  @metasj   w:user:sj  +1 617 529 4266
>
> ___
> Wiki Loves Monuments mailing list
> wikilovesmonume...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
> http://www.wikilovesmonuments.org
>



-- 
Asaf Bartov
Wikimedia Foundation 

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
https://donate.wikimedia.org
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,