Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi Sam, The main misconception (which is understandable, but also often pointed out already) is that Wiki Loves Monuments can be fundamentally different projects from a goals-and-outcomes point of view, based on the interests and strenghts of the local organizers and the local situation. In some countries, the main outcome of the competition is that it brings together organizers for a first project, that can then move on, and leverage their collaboration in other projects. In other countries it fosters collaborations with other organizations. In some countries, it is a very grassroots competition, with low budget and big focus on getting a lot of photos. In other countries, there is a lot of effort (and funding) going into catching editors, setting up structures or overcoming the local challenges or making concepts better aware. Aside from the fact that many of these outcomes are qualitative, which seems to get no attention in the (summaries of the) reports, but do get described in the reports of the individual contests, the local competitions are too diverse to try and catch as one group. This is a fundamental flaw (pointed out before) in the approach. The work is appreciated of course, the numbers can be useful - the way they are presented is however very sensitive for major misunderstandings. Besides this, there are several very specific flaws in the number crunching that have been pointed out, which are for example messing up the numbers on editor retention. I hope that at some point WLM organizers can be given the tools, enthusiasm and support to create their own evaluation on a larger scale. That way I hope that some of the flaws can be avoided thanks to a better understanding of the collaborations, structures and the projects in general. All in all it is good to have something 'to shoot at' but I would prefer that these reports are produces more in concert with the stakeholders involved and affected, rather than 'announced' and 'presented' to the wide community. Best, Lodewijk (effeietsanders) member of the international coordinating team 2011-2013 On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Claudia, I share your concerns about reducing subtle things to a few numbers. Data can also be used in context-sensitive ways. So I'm wondering if there are any existing quantitative summaries that you find useful? Or qualitative descriptions that draw from more than one project? Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. I'm glad to see data being shared, and again it might help to have many different datasets, to limit conceptual bias in what sort of data is relevant. On May 6, 2015 9:59 AM, Claudia Garád claudia.ga...@wikimedia.at wrote: Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1]
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
On Thu, May 7, 2015 at 3:14 PM, Maria Cruz mc...@wikimedia.org wrote: snip All in all it is good to have something 'to shoot at' but I would prefer that these reports are produces more in concert with the stakeholders involved and affected, rather than 'announced' and 'presented' to the wide community. This isn't true. We always reach out to program leaders to engage in data collection. Further, had you taken part of the event, or even watched it, or read the blog we wrote [6], you would have seen nothing is presented or announced, rather, open for discussion and conversation. snip Sure, the team did reach out in the collection phase - after all, without the data such evaluation would be impossible. But after that, the conclusions were drafted and shared with the wide community, rather than with the stakeholders involved to discuss interpretation. And I do admit for not watching the full video (the event itself was during working hours in Europe - not compatible with my job) but only watching parts of it - and it had a high presentation level to me. But maybe I was unlucky in that. Either way, all communication seemed to be aimed to announce the evaluation, rather than to ask active input on whether the analysis made sense, whether there were misunderstandings, etc. But maybe you have had a lot of follow-up discussions with the people you collected data from on a 1-to-1 level, which would be admirable. Again, I do appreciate the effort, I don't agree with the approach and process. Best, Lodewijk ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi Lodewijk, Thanks for your feedback about the process. It's been very valuable. I have a few follow up questions below: Sure, the team did reach out in the collection phase - after all, without the data such evaluation would be impossible. But after that, the conclusions were drafted and shared with the wide community, rather than with the stakeholders involved to discuss interpretation. Can you say more about which stakeholders? Do you have ideas how we might include them in the future, for example, through the Wiki Loves Monuments mailing list, or were you thinking in some other way? Either way, all communication seemed to be aimed to announce the evaluation, rather than to ask active input on whether the analysis made sense, whether there were misunderstandings, etc. But maybe you have had a lot of follow-up discussions with the people you collected data from on a 1-to-1 level, which would be admirable. We tried to encourage input and questions through the next steps and in the talk page, but it sounds like this might not have been enough. How do you think we can do this better next time? Anything specific that stands out to you, beyond sharing with stakeholders beforehand? Thanks so much, Edward Again, I do appreciate the effort, I don't agree with the approach and process. Best, Lodewijk ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org https://meta.wikimedia.org/wiki/Mailing_lists/guidelineswikimedi...@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe -- Edward Galvez Program Evaluation Associate Wikimedia Foundation ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Regaring measuerment of editor retention - this is tricky - as in fact many participant created new accounts only to join the contest. Some of them had accounts on Wikipedia (but different) - some others - abandoned their accounts and created a new ones for various reasons (the most trival - they have forgoten passwords). There are also user who are active only during contensts - also for various reasons - not only due to possibility to win attractive prizes, but also because the normal upload process is too tricky for them, or they don't know what to photograph if there is no easy to use list of objects. In fact measurement of editor retention is tricky even for workshops if it is only based on list of nicknames. I saw this many times - that people create the accounts during the workshop and then abandon them, but create later a new ones. The only effective way to follow the retention of users after workshop is to collect their e-mails and then survey them some time after the workshop. It might produce completely different picture that studies based on following the activity of accounts created during workshops... 2015-05-07 11:34 GMT+02:00 Lodewijk lodew...@effeietsanders.org: Hi Sam, The main misconception (which is understandable, but also often pointed out already) is that Wiki Loves Monuments can be fundamentally different projects from a goals-and-outcomes point of view, based on the interests and strenghts of the local organizers and the local situation. In some countries, the main outcome of the competition is that it brings together organizers for a first project, that can then move on, and leverage their collaboration in other projects. In other countries it fosters collaborations with other organizations. In some countries, it is a very grassroots competition, with low budget and big focus on getting a lot of photos. In other countries, there is a lot of effort (and funding) going into catching editors, setting up structures or overcoming the local challenges or making concepts better aware. Aside from the fact that many of these outcomes are qualitative, which seems to get no attention in the (summaries of the) reports, but do get described in the reports of the individual contests, the local competitions are too diverse to try and catch as one group. This is a fundamental flaw (pointed out before) in the approach. The work is appreciated of course, the numbers can be useful - the way they are presented is however very sensitive for major misunderstandings. Besides this, there are several very specific flaws in the number crunching that have been pointed out, which are for example messing up the numbers on editor retention. I hope that at some point WLM organizers can be given the tools, enthusiasm and support to create their own evaluation on a larger scale. That way I hope that some of the flaws can be avoided thanks to a better understanding of the collaborations, structures and the projects in general. All in all it is good to have something 'to shoot at' but I would prefer that these reports are produces more in concert with the stakeholders involved and affected, rather than 'announced' and 'presented' to the wide community. Best, Lodewijk (effeietsanders) member of the international coordinating team 2011-2013 On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Claudia, I share your concerns about reducing subtle things to a few numbers. Data can also be used in context-sensitive ways. So I'm wondering if there are any existing quantitative summaries that you find useful? Or qualitative descriptions that draw from more than one project? Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. I'm glad to see data being shared, and again it might help to have many different datasets, to limit conceptual bias in what sort of data is relevant. On May 6, 2015 9:59 AM, Claudia Garád claudia.ga...@wikimedia.at wrote: Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Editor retention really consists of three components *new temporary contributors. WLM helps here, and even if they leave after a few edits this is of value for the projects. They have learned to edit, and will be more open to correct an error or complement an article very much later when using Wikipedia *New regular contributors. Low impact from WLM, but this is the key and only parameter being measured *Make regular contributors stay on (longer). Here too WLM has a positive effect. It is a stimulus for longtimers to see the new images, the (IRL)activities around the WLM and that something of value is happening. This is of course impossible to measure. Personally I believe that making the work environment fun and stimulating is the most cost effective way to keep up the editor base. The Thanks notification is a wonderful example of high-effect on retention by a very limited investment in software Anders Tomasz Ganicz skrev den 2015-05-07 13:06: Regaring measuerment of editor retention - this is tricky - as in fact many participant created new accounts only to join the contest. Some of them had accounts on Wikipedia (but different) - some others - abandoned their accounts and created a new ones for various reasons (the most trival - they have forgoten passwords). There are also user who are active only during contensts - also for various reasons - not only due to possibility to win attractive prizes, but also because the normal upload process is too tricky for them, or they don't know what to photograph if there is no easy to use list of objects. In fact measurement of editor retention is tricky even for workshops if it is only based on list of nicknames. I saw this many times - that people create the accounts during the workshop and then abandon them, but create later a new ones. The only effective way to follow the retention of users after workshop is to collect their e-mails and then survey them some time after the workshop. It might produce completely different picture that studies based on following the activity of accounts created during workshops... 2015-05-07 11:34 GMT+02:00 Lodewijk lodew...@effeietsanders.org: Hi Sam, The main misconception (which is understandable, but also often pointed out already) is that Wiki Loves Monuments can be fundamentally different projects from a goals-and-outcomes point of view, based on the interests and strenghts of the local organizers and the local situation. In some countries, the main outcome of the competition is that it brings together organizers for a first project, that can then move on, and leverage their collaboration in other projects. In other countries it fosters collaborations with other organizations. In some countries, it is a very grassroots competition, with low budget and big focus on getting a lot of photos. In other countries, there is a lot of effort (and funding) going into catching editors, setting up structures or overcoming the local challenges or making concepts better aware. Aside from the fact that many of these outcomes are qualitative, which seems to get no attention in the (summaries of the) reports, but do get described in the reports of the individual contests, the local competitions are too diverse to try and catch as one group. This is a fundamental flaw (pointed out before) in the approach. The work is appreciated of course, the numbers can be useful - the way they are presented is however very sensitive for major misunderstandings. Besides this, there are several very specific flaws in the number crunching that have been pointed out, which are for example messing up the numbers on editor retention. I hope that at some point WLM organizers can be given the tools, enthusiasm and support to create their own evaluation on a larger scale. That way I hope that some of the flaws can be avoided thanks to a better understanding of the collaborations, structures and the projects in general. All in all it is good to have something 'to shoot at' but I would prefer that these reports are produces more in concert with the stakeholders involved and affected, rather than 'announced' and 'presented' to the wide community. Best, Lodewijk (effeietsanders) member of the international coordinating team 2011-2013 On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Claudia, I share your concerns about reducing subtle things to a few numbers. Data can also be used in context-sensitive ways. So I'm wondering if there are any existing quantitative summaries that you find useful? Or qualitative descriptions that draw from more than one project? Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. I'm glad to see data being shared, and again it might help to have many different datasets, to limit conceptual bias in what sort of data is relevant. On May 6, 2015
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi, I wasn't involved in this evaluation, but I would like to say that, as someone who recently worked for WMF Learning and Evaluation, I believe that the LE team is interested in producing useful and accurate reports. So, I am optimistic that feedback from the community about methodology and communications will be carefully considered in future work plans for the LE team. Also, I will mention that Cascadia Wikimedians plans to participate in Summer of Monuments, and we will look at LE reports for ideas and data about effective practices in Wiki Loves Monuments and other programmatic work. These reports will be, I hope, not just about numerical accountability but also about sharing stories, ideas, and qualitative information. Regards, Pine ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi Edward, Thanks for the questions. The Wiki Loves Monuments mailing list would have made a very logical starting place to ask for initial feedback. But also sending an email to the people who shared their data with you to work with in the first place, or people who worked on internal evaluations in these projects before. The feeling has been created that right now, the 'damage is done': the report is published, you have done all you could to make sure that all community members are as much aware as possible of what you consider the conclusions. That means that any feedback now, becomes somewhat moot. We have seen this before with Foundation publications (i.e. statistics on the chapters), once it is announced to the community at large, feedback often doesn't get incorporated any more (I hope this time it does!) and even if it is, the facts already found their place into other publications like the signpost. Asking feedback is most valuable *before* you announce it, and proactively. You could (even better) consider involving those stakeholders even earlier in the process, which makes it less of a black box. I strongly believe that it would improve the quality of the work you do. Still some of the basic flaws will remain due to the basic setup of the evaluation framework (assumptions that all WLM are comparable etc.) but others could be managed better. Best, Lodewijk On Fri, May 8, 2015 at 3:47 AM, Edward Galvez egal...@wikimedia.org wrote: Hi Lodewijk, Thanks for your feedback about the process. It's been very valuable. I have a few follow up questions below: Sure, the team did reach out in the collection phase - after all, without the data such evaluation would be impossible. But after that, the conclusions were drafted and shared with the wide community, rather than with the stakeholders involved to discuss interpretation. Can you say more about which stakeholders? Do you have ideas how we might include them in the future, for example, through the Wiki Loves Monuments mailing list, or were you thinking in some other way? Either way, all communication seemed to be aimed to announce the evaluation, rather than to ask active input on whether the analysis made sense, whether there were misunderstandings, etc. But maybe you have had a lot of follow-up discussions with the people you collected data from on a 1-to-1 level, which would be admirable. We tried to encourage input and questions through the next steps and in the talk page, but it sounds like this might not have been enough. How do you think we can do this better next time? Anything specific that stands out to you, beyond sharing with stakeholders beforehand? Thanks so much, Edward Again, I do appreciate the effort, I don't agree with the approach and process. Best, Lodewijk ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org https://meta.wikimedia.org/wiki/Mailing_lists/guidelineswikimedi...@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe -- Edward Galvez Program Evaluation Associate Wikimedia Foundation ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
On Thu, May 7, 2015 at 6:34 AM, Lodewijk lodew...@effeietsanders.org wrote: I hope that at some point WLM organizers can be given the tools, enthusiasm and support to create their own evaluation on a larger scale. That way I hope that some of the flaws can be avoided thanks to a better understanding of the collaborations, structures and the projects in general. The Evaluation portal on Meta [1] has all the resources we use, open to organizers of any program. There is a guide to using the portal resources [2]. We also host virtual meet ups regularly to develop capacity around evaluation, that are recorded and available on our Youtube channel [3] under CC license. The Learning and Evaluation team is open to have conversations one on one as well! =) We are always encouraging program leaders to engage in this conversation: what metrics matter to this program, what is relevant to measure. Happily, this is the conversation we had with some WLM organizers yesterday [4], which is also taking place on WLM Report talk page [5]. All in all it is good to have something 'to shoot at' but I would prefer that these reports are produces more in concert with the stakeholders involved and affected, rather than 'announced' and 'presented' to the wide community. This isn't true. We always reach out to program leaders to engage in data collection. Further, had you taken part of the event, or even watched it, or read the blog we wrote [6], you would have seen nothing is presented or announced, rather, open for discussion and conversation. *María Cruz * \\ Community Coordinator, PED Team \\ Wikimedia Foundation, Inc. mc...@wikimedia.org | : @marianarra_ https://twitter.com/marianarra_ [1] https://meta.wikimedia.org/wiki/Grants:Evaluation [2] https://meta.wikimedia.org/wiki/Grants:Evaluation/Introduction [3] https://www.youtube.com/user/WikiEvaluation/ [4] https://www.youtube.com/watch?v=PN3TN4wrFZs [5] https://meta.wikimedia.org/wiki/Grants_talk:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments [6] http://blog.wikimedia.org/2015/04/22/first-2015-wikimedia-programs-evaluations/ Best, Lodewijk (effeietsanders) member of the international coordinating team 2011-2013 On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Claudia, I share your concerns about reducing subtle things to a few numbers. Data can also be used in context-sensitive ways. So I'm wondering if there are any existing quantitative summaries that you find useful? Or qualitative descriptions that draw from more than one project? Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. I'm glad to see data being shared, and again it might help to have many different datasets, to limit conceptual bias in what sort of data is relevant. On May 6, 2015 9:59 AM, Claudia Garád claudia.ga...@wikimedia.at wrote: Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads
[Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
I organized a contest Wiki Loves Monuments wiki and loves earth in algeria and coordinate on the rest of each arabic country who has organised the contest I had a lot of fun to organize during 2013 2014 till 2015 now In algeria ;with astonishment ;many do not know what that meant wikipedia; those who knew wikipedia ;they were discovered commons and more he was able to me; to establish relationships that are allowed me to create WMUG Algeriait; is a great chalenge for me all these offline activities help to enhance the experience and conaissance of wikimedia projects Wikipedia is an encyclopedia of knowledge; and yet it is a great unknown Wiki loves what is good; May it must be renewed I suggest only one thing: a single contest each year with a commission that reflects, and proposes the theme of the year it is more focus; if the Commission believes that this year we need to photograph flowers or rare plants or animals in commons then this will be the case International Commitee or Commission or the way you want and the weak link was missing the comunity wikimedia must help to the elaboration of this infrastructure without influencing it best -- *Mohammed Bachounda* Leader Wikimedia Algérie UG ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
[Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe -- Samuel Klein @metasj w:user:sj +1 617 529 4266 ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Claudia, I share your concerns about reducing subtle things to a few numbers. Data can also be used in context-sensitive ways. So I'm wondering if there are any existing quantitative summaries that you find useful? Or qualitative descriptions that draw from more than one project? Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. I'm glad to see data being shared, and again it might help to have many different datasets, to limit conceptual bias in what sort of data is relevant. On May 6, 2015 9:59 AM, Claudia Garád claudia.ga...@wikimedia.at wrote: Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hello my friends, I didn't have the opportunity to organize a WLM contest yet, but I had the opportunity to organize the Brazilian WLE last year and I'm promoting that same contest here in Brasil this year again. Quantitative analysis are always easier to do than qualitative analysis. In that case WMF is always trying to show how the money was spent, how much and the direct impact of that effort. So, basically we are always measuring using metrics of the conversion of money/time to edits/new users engagement and retention. IMHO, WLE and WLM are bigger than this and these projects are very complex with many types of results and direct and mainly, indirect impacts to the Wikimedia Projects and movement in general. By example, last year we organized an exhibition https://pt.wikipedia.org/wiki/Wikip%C3%A9dia:Wiki_Loves_Earth_2014/Brasil/Exposi%C3%A7%C3%A3o with the TOP500 photos collected during the WLE Brasil 2014. So, during that exhibition so many people from many classes, ages, cities and level of knowledge passed by the exhibition and read the contest descriptions and they got more information regarding the Photo Contest, WLE, Wikipedia, Commons and the Wikimedia movement. Since we can't track the visitors behavior after the exhibition, we cant say anything regarding the results of that activity based on the regular metrics, used by default. So, based in things like that we cant publish any final report with the real impact of the exhibition translated in some numbers.IMHO that report regarding WLM don't reflect the real impact of the contest, we can see there only some simple numbers and conversions and this kind of report can generate a lot of misunderstanding when accessed by the regular media. Best regards Rodrigo Padula Wikimedia Brazilian Group of Education and Research PPGI/UFRJ 2015-05-06 10:59 GMT-03:00 Claudia Garád claudia.ga...@wikimedia.at: Hi Sam, I am sure there are figures and stories that the various orgs collect and publish. But they are spread across different wikis and websites and/or languages. E.g. many of the FDC orgs are looking into ways to demonstrate these more qualitative aspects of our work (e.g. by storytelling) in their reports. But these information does not get the same attention and publicity in the wider community as the evaluation done by the WMF. Many WMAT volunteers and I myself share the concerns expressed by Romaine that these unidimensional numbers and lack of context foster misconceptions or even prejudices especially in the parts of the community that are not closely involved in the work of the respective groups and orgs. Best Claudia Am 06.05.2015 um 13:40 schrieb Sam Klein: Hi Romaine, Are there other evals of WLM projects that capture the complexity you want? Perhaps single-community evaluations done by the WLM organizers there? Sam On Wed, May 6, 2015 at 7:21 AM, Romaine Wiki romaine.w...@gmail.com wrote: Hi all, In the past months the Wikimedia Foundation has been writing an evaluation about Wiki Loves Monuments. [1] At such it is fine that WMF is writing an evaluation, however they fail in actual understanding Wiki Loves Monuments, and that is shown in the evaluation report. As a result on the Wiki Loves Monuments mailing list a discussion grows about the various problems the evaluation has. As the Learning and Evaluation team at the Wikimedia Foundation already had released the first Programs Reports for Wiki Loves Monuments, we are now put as fait accompli with this evaluation report. Therefore I am writing here so that the rest of the worldwide Wikimedia community is informed that this is not going right. Wiki Loves Monuments is not just a bunch of uploads done in September, the report is too simplified without actual understanding how the community is doing this project. Romaine [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Hi all, Thanks for the comments on the first two program evaluation reports. This is the kind of feedback we are looking for coming from the community, and for that reason, we want to continue this conversation and learn more about what goals and metrics make more sense to program leaders. As many of you know, today we held an open virtual event to introduce the first Wikimedia Programs Evaluation Reports 2015. You can now watch the recorded event online [1]. We have also captured some of the conversation that started on Wiki Loves Monuments list on the report's Talk Page [2]. Many community members have already contributed their views there. We want to encourage everyone to keep the conversation on the talk page, which will allow us to document all the feedback and keep track of it. Looking forward to your feedback and happy editing! *María Cruz * \\ Community Coordinator, PED Team \\ Wikimedia Foundation, Inc. mc...@wikimedia.org | : @marianarra_ https://twitter.com/marianarra_ [1] Video of the reports presentation https://youtu.be/PN3TN4wrFZs [2] Wiki Loves Monuments Evaluation Report - Talk Page https://meta.wikimedia.org/wiki/Grants_talk:Evaluation/Evaluation_reports/2015/Wiki_Loves_Monuments On Wed, May 6, 2015 at 12:11 PM, Ilario Valdelli valde...@gmail.com wrote: Yes, I think that this may be considered the central problem. It's easier to compare two different scenarios with a standard measure and to use kilos to compare apples and oranges, for instance. The problem is to understand that oranges will continue to be oranges after this measure, and apples will continue to be apples. This is an example to say that several countries focus their contest in quality, some others in quantity. The prize and the contest, anyway, is focused to select the better photo and not the biggest uploaders. It means that there is no sense to force the quantitative parameters while the incentives are focused to increase quality. Personally I find the same measure costs/uploads a lot far from the most correct measure costs/benefits because we cannot consider a single upload automatically as a benefit. In my opinion the most critical point is how measure costs (the workload of a community is it a cost?) and the benefits (a huge amount of worst photos is it a benefit?) because it involves several not measurable parameters. Regards On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. -- Ilario Valdelli Wikimedia CH Verein zur Förderung Freien Wissens Association pour l’avancement des connaissances libre Associazione per il sostegno alla conoscenza libera Switzerland - 8008 Zürich Wikipedia: Ilario https://meta.wikimedia.org/wiki/User:Ilario Skype: valdelli Facebook: Ilario Valdelli https://www.facebook.com/ivaldelli Twitter: Ilario Valdelli https://twitter.com/ilariovaldelli Linkedin: Ilario Valdelli http://www.linkedin.com/profile/view?id=6724469 Tel: +41764821371 http://www.wikimedia.ch ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
Re: [Wikimedia-l] Evaluation by WMF of Wiki Loves Monuments is failing to understand the community
Yes, I think that this may be considered the central problem. It's easier to compare two different scenarios with a standard measure and to use kilos to compare apples and oranges, for instance. The problem is to understand that oranges will continue to be oranges after this measure, and apples will continue to be apples. This is an example to say that several countries focus their contest in quality, some others in quantity. The prize and the contest, anyway, is focused to select the better photo and not the biggest uploaders. It means that there is no sense to force the quantitative parameters while the incentives are focused to increase quality. Personally I find the same measure costs/uploads a lot far from the most correct measure costs/benefits because we cannot consider a single upload automatically as a benefit. In my opinion the most critical point is how measure costs (the workload of a community is it a cost?) and the benefits (a huge amount of worst photos is it a benefit?) because it involves several not measurable parameters. Regards On Wed, May 6, 2015 at 4:40 PM, Samuel Klein meta...@gmail.com wrote: Figuring out what ideas are repeatable, scalable, or awesome but one-time only, is complex. We probably need many different approaches, not one central approach, to understand and compare. -- Ilario Valdelli Wikimedia CH Verein zur Förderung Freien Wissens Association pour l’avancement des connaissances libre Associazione per il sostegno alla conoscenza libera Switzerland - 8008 Zürich Wikipedia: Ilario https://meta.wikimedia.org/wiki/User:Ilario Skype: valdelli Facebook: Ilario Valdelli https://www.facebook.com/ivaldelli Twitter: Ilario Valdelli https://twitter.com/ilariovaldelli Linkedin: Ilario Valdelli http://www.linkedin.com/profile/view?id=6724469 Tel: +41764821371 http://www.wikimedia.ch ___ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe
