Hi Pine, Thank you for your bringing this page to our attention and for raising these interesting questions. I would have to agree that the “Program evaluation basics” page is not well-designed and should be revisited. We are actually going to be redesigning the entire evaluation portal soon and this page will likely be revised and included in the new design in some way. We are also continuing to build tools and learning resources (like the learning modules [1]) on evaluation to help explain some of these concepts.
I also agree that we need to think more about how we can define “impact” within the context of Wikimedia. Before we can reach a final “impact”, there are different layers of success in terms of outputs and short-, intermediate-, and long-term outcomes that help to measure success along the way. We have been working on this approach to evaluation—we have developed resources for mapping a program’s theory of change in order to identify measurable outcomes, both near and far. Specifically, logic models are a useful tool for drawing out the steps needed to reach long-term impact and identifying more immediate indicators for evaluation; there is a resource page within the Evaluation portal on logic models [2] and I am working on a learning module that will guide anyone through what a logic model is and how to create one. As far as the term “impact”, it is very jargonistic and can be used in many ways which can be confusing. Since we began last year, we have been working to generate a growing glossary of a shared language around evaluation [3]. That glossary page is more current and inclusive than the original “Program Evaluation basics” page you linked to. Please feel free to discuss this and any other of those terms and definitions there on the portal. Coincidentally, we are asking the community to provide feedback on some of the initial evaluation capacity building efforts our team has engaged in thus far. We’d like to hear feedback on the metrics and methods used so we can continue towards a shared understanding of Wikimedia programs and their impacts. We invite you (or anyone!) to read about the Community Dialogue [4] and join in the discussion on the Evaluation portal Parlor [5]. As always, I’m available for any questions! Best, Edward [1] https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Learning_modules [2] https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/Logic_models [3] https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/Glossary [4] https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Parlor/Dialogue [5] https://meta.wikimedia.org/wiki/Programs_talk:Evaluation_portal/Parlor/Dialogue On Sun, May 18, 2014 at 6:23 PM, ENWP Pine <deyntest...@hotmail.com> wrote: > Hi, I spent a few minutes searching on Meta for how "impact" is defined. > What is the WMF definition? > > Some examples of places where "impact" is used: > * https://meta.wikimedia.org/wiki/Grants:APG/FDC_portal/Impact_report_form > * https://meta.wikimedia.org/wiki/Grants:APG/Impact_report_form_Q%26A > * https://meta.wikimedia.org/wiki/Grants:IEG/Learning/Round_1_2013/Impact > * > https://blog.wikimedia.org/2014/05/02/beginning-understand-what-works-measuring-impact-programs/ > * > https://meta.wikimedia.org/wiki/Program_evaluation_basics:_efficiency,_effectiveness_and_impact > > I am not fond of the Boulmetis / Dutwin definition used in that last > reference because short-term effects can be important and much easier to > measure than long-term effects. For example, an administrator protecting a > page can have the short-term effect of preventing editing and preventing an > edit war, and the long term effects of that can be impossible to know, such > as whether preventing an edit war prevented the situation from escalating > to an Arbcom case with imposition of long-term blocks, and also whether > preventing editing prevented important information from being added to the > page by an occasional IP editor. > > I might suggest a rewrite of that entire page on "program evaluation > basics" to make it simple. Right now it's a wall of text that's difficult > to follow and, I feel, at least partly wrong. I think that Edward Galvez is > working on some of these issues and I would be happy to have him or someone > else in Evaluation thoughtfully redesign and rewrite that page to make it > easy to follow for everyone including non-native English speakers. If I > have a hard time with that page, you can imagine how difficult it is for > someone who only understands English at an intermediate level. I would like > to start with having a clear and simple definition of "impact" that makes > sense in Wikimedia contexts, and some examples that are easy to follow. > > Thanks, > > Pine > > _______________________________________________ > Wikimedia-l mailing list, guidelines at: > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > Wikimedia-l@lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe> -- Edward Galvez Program Evaluation Associate Wikimedia Foundation _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>
