Re: [Wikimedia-l] [Wikimedia Announcements] Be part of the most epic data collection and analysis of Wikimedia programs!
Thank you, Maria. By the way, I like this poster's colorful high-level overview of previous findings: https://commons.wikimedia.org/wiki/File:Evaluation_Report_%28beta%29_Poster_Wikimania_2014.pdf I am looking forward to seeing the findings from the new survey, especially the portion about conferences. I also am glad to hear that a variety of language responses will be accommodated, and I hope that this will improve the quantity and diversity of data collection. Atentamente, Pine On Thu, Sep 25, 2014 at 1:29 PM, Maria Cruz mc...@wikimedia.org wrote: Hi Pine, thank you for your detailed response. Answers in line! On Wed, Sep 24, 2014 at 8:24 PM, Pine W wiki.p...@gmail.com wrote: Hi Anasuya, Thanks for this announcement. It will be interesting to see what we learn from this round. I have a few questions. * People may feel more comfortable with providing reports in languages that are familiar to them. Are responses allowed in languages other than English? Yes, they are. Open-ended text responses may be answered in different languages, as well. Text boxes are dominant in the survey to make the process less rigid. While we may need to follow up for clarification after translation, we definitely are willing to work with those reporting to make the process as light as possible. Further, category names should be reported as they would appear in the project they exist in, not translated to English. * The Start Date and End Date fields allow free-form text. Will that create any difficulties for the people who compile and analyze the information from these reports? Actually, the fields are open text, because, if we add validation, it requires the answer and will not permit a reporter who does not have the exact dates, or someone who only has a date and not hour, to advance through the rest of the reporting form. Instead, we do offer an open box, optional to complete, and instruct responses to be in the MM/DD/YY:00:00 (UTC) format. We will clean up the data along with the rest of the analysis. * What is the definition of quality content? There are many definitions of quality content, many of which have no easy measure. In this question set , we expect the indicators listed to assess how much content was affected: Number of bytes added (Bytes added, positive sum in Wikimetrics), Number of bytes removed (Bytes added, negative sum in Wikimetics), Number of new articles that were created (Pages created, namespace 0 in Wikimetrics), Number of photos/media uploaded to Wikimedia Commons, and Number of Wikimedia pages improved. Later on in the survey, along with other outcomes rather than outputs, we also ask for metrics indicating “quality”. For images, these metrics include number of uniques images used, number of Wikimedia projects using images, and the number of images which are awarded “quality image,” “valued image,” or “featured picture” status. For article content, these metrics include number of articles which are awarded “featured article” or “quality article” status. * The statement Please upload your txt or csv file of your participant usernames. raises an interesting privacy question. Should program leaders be uploading those usernames if consent forms were not obtained? Also, do different standards apply if, at the end of the Qualtrics report, the program leader who is completing the survey selects the option to allow all data from the survey form to be public? Yes, that is an important clarification we can make. Importantly, opt-in procedures should be followed in order to collect usernames at in-person events for use in Wikimetrics. The use of this tool, housed on servers in the US, automatically transfers data internationally (for anyone outside of the U.S.) [1]. If usernames for a program exist publically (eg,on an event page, or online elsewhere) we suggest an opt-out procedure for Wikimetrics as well, but it is as a courtesy as the data have already been exchanged internationally and publically via the internet in such a case. Regardless, no individual usernames will appear in any level of reporting. However, we need to have username list used for reporting. This will allow us to validate the data, as well as have the ability to pull additional metrics, if needed, as we proceed with analysis and reporting. We have now added further clarification. We have changed the language from: Please upload your txt or csv file of your participant usernames. to: Please upload your txt or csv file of your participant usernames. Note: If you have collected usernames from an in-person event as the only record of participation, there are some important steps that may apply [1]. If you have any questions or concerns about confidentiality of user names please reach out to the evaluation team and we can help you determine what is appropriate. * I am confused by the question
Re: [Wikimedia-l] [Wikimedia Announcements] Be part of the most epic data collection and analysis of Wikimedia programs!
Hi Pine, thank you for your detailed response. Answers in line! On Wed, Sep 24, 2014 at 8:24 PM, Pine W wiki.p...@gmail.com wrote: Hi Anasuya, Thanks for this announcement. It will be interesting to see what we learn from this round. I have a few questions. * People may feel more comfortable with providing reports in languages that are familiar to them. Are responses allowed in languages other than English? Yes, they are. Open-ended text responses may be answered in different languages, as well. Text boxes are dominant in the survey to make the process less rigid. While we may need to follow up for clarification after translation, we definitely are willing to work with those reporting to make the process as light as possible. Further, category names should be reported as they would appear in the project they exist in, not translated to English. * The Start Date and End Date fields allow free-form text. Will that create any difficulties for the people who compile and analyze the information from these reports? Actually, the fields are open text, because, if we add validation, it requires the answer and will not permit a reporter who does not have the exact dates, or someone who only has a date and not hour, to advance through the rest of the reporting form. Instead, we do offer an open box, optional to complete, and instruct responses to be in the MM/DD/YY:00:00 (UTC) format. We will clean up the data along with the rest of the analysis. * What is the definition of quality content? There are many definitions of quality content, many of which have no easy measure. In this question set , we expect the indicators listed to assess how much content was affected: Number of bytes added (Bytes added, positive sum in Wikimetrics), Number of bytes removed (Bytes added, negative sum in Wikimetics), Number of new articles that were created (Pages created, namespace 0 in Wikimetrics), Number of photos/media uploaded to Wikimedia Commons, and Number of Wikimedia pages improved. Later on in the survey, along with other outcomes rather than outputs, we also ask for metrics indicating “quality”. For images, these metrics include number of uniques images used, number of Wikimedia projects using images, and the number of images which are awarded “quality image,” “valued image,” or “featured picture” status. For article content, these metrics include number of articles which are awarded “featured article” or “quality article” status. * The statement Please upload your txt or csv file of your participant usernames. raises an interesting privacy question. Should program leaders be uploading those usernames if consent forms were not obtained? Also, do different standards apply if, at the end of the Qualtrics report, the program leader who is completing the survey selects the option to allow all data from the survey form to be public? Yes, that is an important clarification we can make. Importantly, opt-in procedures should be followed in order to collect usernames at in-person events for use in Wikimetrics. The use of this tool, housed on servers in the US, automatically transfers data internationally (for anyone outside of the U.S.) [1]. If usernames for a program exist publically (eg,on an event page, or online elsewhere) we suggest an opt-out procedure for Wikimetrics as well, but it is as a courtesy as the data have already been exchanged internationally and publically via the internet in such a case. Regardless, no individual usernames will appear in any level of reporting. However, we need to have username list used for reporting. This will allow us to validate the data, as well as have the ability to pull additional metrics, if needed, as we proceed with analysis and reporting. We have now added further clarification. We have changed the language from: Please upload your txt or csv file of your participant usernames. to: Please upload your txt or csv file of your participant usernames. Note: If you have collected usernames from an in-person event as the only record of participation, there are some important steps that may apply [1]. If you have any questions or concerns about confidentiality of user names please reach out to the evaluation team and we can help you determine what is appropriate. * I am confused by the question that ends with the report. It starts with the statement that Although we will not share reporting data in an identifiable way... and then proceeds to ask if the person completing the report will allow WMF to share the name of my program along with my reported data so that people can see how we did. Can you clarify this situation? Last round of reporting we used the standard privacy language to assure complete anonymity of direct reporters as a general privacy decision to maximize comfort and accuracy in reporting. When we published our first round of reports with such a high level of anonymity, a number of community members protested
Re: [Wikimedia-l] [Wikimedia Announcements] Be part of the most epic data collection and analysis of Wikimedia programs!
Hi Anasuya, Thanks for this announcement. It will be interesting to see what we learn from this round. I have a few questions. * People may feel more comfortable with providing reports in languages that are familiar to them. Are responses allowed in languages other than English? * The Start Date and End Date fields allow free-form text. Will that create any difficulties for the people who compile and analyze the information from these reports? * What is the definition of quality content? * The statement Please upload your txt or csv file of your participant usernames. raises an interesting privacy question. Should program leaders be uploading those usernames if consent forms were not obtained? Also, do different standards apply if, at the end of the Qualtrics report, the program leader who is completing the survey selects the option to allow all data from the survey form to be public? * I am confused by the question that ends with the report. It starts with the statement that Although we will not share reporting data in an identifiable way... and then proceeds to ask if the person completing the report will allow WMF to share the name of my program along with my reported data so that people can see how we did. Can you clarify this situation? Thanks, Pine On Wed, Sep 24, 2014 at 4:29 PM, Anasuya Sengupta asengu...@wikimedia.org wrote: Dear Wikimedian friends and colleagues, tl;dr We have just launched our second round of voluntary reporting. This is the most epic data collection and analysis of Wikimedia programs we've done so far as a movement, and all program leaders are invited to take part. :-) You can do so here: https://wikimedia.qualtrics.com/SE/?SID=SV_0B3azKpdZ7ggCtD (or get in touch with the LE team for support). As we did in the Fall of 2013, we invite community members leading and evaluating Wikimedia programs to share their data with the rest of the movement (i.e., Edit-a-thons, Editing Workshops, On-wiki Writing Contests, Photo Events, etc.). Last year’s data was collected and analysed in a series of reports that was the beginning of telling the Wikimedia story of impact: the incredible work of over 60 program leaders implementing 119 programs or projects in 30 countries across the world. This helped us start building a set of good and best practices for effective programs across our movement.[1] This year’s data drive will be critical to help us continue to do and learn better from each other. To best prepare, program leaders can review the reporting items [1] and start gathering that data you have filed away about your programs since the last reporting round. We are looking for data on programs completed any time from September 1, 2013 through September 30, 2014. You can ask questions directly on the reporting form preview [2] or on our portal talk page [3]. If you are planning to report and may need support from us, do let us know so that we can help in any way needed. When ready, you will find the reporting collector at: https://wikimedia.qualtrics.com/SE/?SID=SV_0B3azKpdZ7ggCtD We also welcome your data in different formats. For example, if you have already reported data elsewhere, we are happy to work with you to make the process as easy as possible. Message e...@wikimedia.org and we can work out the easiest way to include your data. We are expanding the number of programs covered in the reporting this year, and extend the reporting window longer for some new programs, GLAM, and Wiki Loves Monuments. See the schedule below for timelines for reporting for each program type. Data submission deadlines by program: Due by October 20th - Edit-a-thons/editing parties - Editing Workshops Due by November 3rd - On-wiki Writing Contests - Photo Events (Wiki Loves Earth, WikiExpeditions, WikiTakes, etc.) - Wikipedia Education Program Due by November 17th - Conferences - GLAM Content Donation - Hackathons - Wiki Loves Monuments (2013 and 2014) - Wikimedian in Residence Remember, reporting is voluntary but the more people do it, the better representation of programs we can make. This voluntary reporting allows us to come together and generate a bird’s eye view of programs [4]. We want to understand the impact of programs across different contexts, to examine both more broadly, and more deeply, what works best to meet our shared goals for Wikimedia and to, together, grow the awesome [5] in Wikimedia programs! On behalf of the Program Evaluation and Design team, thank you for your time and support in this initiative. Warmly, Anasuya Resource links: [1] https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2013 [2] https://docs.google.com/document/d/1CG-K8I1d9JPqyRRyHTIQ5x7fASQXcHZsEMKutdAGuEY/edit?usp=sharing [3] https://meta.wikimedia.org/wiki/Grants_talk:Evaluation [4]
