This was an awesome read. Thank you for sharing it!
On Tue, Apr 16, 2013 at 4:35 PM, Laura Hale <[email protected]> wrote: > This is currently published on my user space on Wikinews at > http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis and > as done in order to determine some of the issues that students are facing > with the review process at Wikinews in light of student complaints to their > instructor. I am hoping to do a follow up for it at some point in the > future to provide more detailed analysis on review processes on project, > but thought in the interim it might be of interest to both researchers and > people in education. > > Wikinews Review Analysis > < User:LauraHale <http://en.wikinews.org/wiki/User:LauraHale> > <http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#mw-navigation><http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#p-search> > > One of the greatest challenges in attracting and retaining new > contributors to English Wikinews is the project's review process. This > process requires the journalist to have a symbiotic relationship with the > reviewer as the pair both work towards getting an article published. The > review process involves the submitted news article being checked for > copyright, newsworthiness, verifiability, neutral point of view and > compliance with the style guide. This process is compacted ideally into a > window of opportunity of no longer than 24 to 48 hours. It can be extremely > challenging for those not used to the style requirements, neutrality > requirements, verification requirements and above all doing this sort of > writing quickly. > > For those from Wikipedia, the review process on English Wikinews is close > in scope to English Wikipedia's Good Article > criteria<http://en.wikipedia.org/wiki/WP:GAN> only > on a more compressed timetable. For those who have been to secondary school > or university, it mirrors a teacher giving you an assignment with your > grade being based on meeting the criteria stated on a rubric, and failure > to pass requires re-submission. If you do not work to the rubric, you just > do not pass and on English Wikinews, that means not getting published. > > When you review regularly, you begin to notice certain patterns that > frequently occur in the review process. Most often, these appear to be a > failure to understand what is news or a failure to read and try to write to > the style guide specifications. Observational analysis only gets you so far > though when trying to determine a problem and how to develop solutions for > what appears to be a problem of articles not passing review or people being > discouraged from submitting. > > From the period between January 1, 2013 and April 12, 2013, 203 failed > reviews were examined to determine which criteria were the biggest > stumbling block. The articles reviewed included published articles, > non-published articles moved to user space and deleted articles that were > not published. For each review, the primary author was assessed as either > accredited reporter, regular contributor with 10 or more published > articles, new reporter with 9 or fewer articles, or University of > Wollongong student. Of the 202 reviews examined, 104 were for articles by > new contributors, 47 by University of Wollongong students, 40 by regular > contributors and 11 by accredited reporters. Each review looked at also > noted if the article finally reached a published state. There were 110 > different articles reviewed, of which 33 were published and 77 were not. > > What were problems for articles did not pass review? Bearing in mind > articles can be marked not published for multiple reasons, 23 articles were > not passed for copyright reasons, 103 for newsworthiness issues, 70 for > verifiability issues, 43 for neutrality issues and 96 for style issues. > [image: English Wikinews Review > Issues.png]<http://en.wikinews.org/wiki/File:English_Wikinews_Review_Issues.png> > > The different cohorts appear to have different sets of issues. Accredited > reporters did not have any problems with copyright or plagiarism. 57% of > University of Wollongong reviews and 51% of new reporters had not passing > reviews because of newsworthiness concerns. Accredited reporters had > problems with verifiability at 45% versus 26% for University of Wollongong > reporters. 25% of new contributors and 29% of University of Wollongong > reporters had problems with neutrality. 60% of University of Wollongong had > problems complying with the style guide compared to 33% of regular > reporters. > > Some of these patterns are explainable. There are times when it is > difficult to get reviewers, and an article may languish for 24 to 48 hours. > By that time, the article is no longer news or requires more and new > information in order to stay fresh. This does not explain all of it though. > Observational analysis suggests that at least two thirds of these articles > fail newsworthiness because of a lack of a clear focus on the news topic, > writing a news article as the topic itself is set to go stale, taking too > long to address problems with previous reviews or going through the review > process repeatedly to the point where by the time everything else gets > fixed, the story is no longer news. > > In many cases, reviewers look at some things before others. Newsworthiness > and style guide compliance are two of the most easily visible problems. > They do not require looking at external links and doing intensive > examination of the text to look for more systemic, underlying problems. The > article does not state when an event happened or makes clear it happened 4 > days ago? The article is written using lots of non-relative dating? There > are lots of external links inside the article? The article is clearly not > written in inverted pyramid style? The title lacks a verb? The first > paragraph does not answer who, what, when, how or why this is news? There > is no reason to look beyond newsworthiness and style as these obvious > problems need to be fixed before going forward. Copyright, NPOV and > verification can come later. > > For accredited reporters, many of them are accredited because they do > original reporting. This often requires sending things to reviewers via > e-mail or posting extensive notes, pictures, audio, video on the talk page. > Verification is also often one of the last steps in the review process. > Thus, it makes sense that reporters with a track record of success are > likely to get caught up here. > > Multiple problems identified on a review decreases the likelihood of > publication. Only 22 failed reviews were present on articles that > subsequently were published. This compares 74 reviews that identified > multiple problems where the article was never published. The only review > for an accredited reporter which identified multiple problems was > subsequently published. Regular contributors had 11 total reviews with > multiple problems, of which 7 of those reviews were done on articles that > were subsequently published. 14 reviews identifying multiple problems for > articles by new contributors went on to be published, with 45 reviews on > articles that were not published. All 24 of the UoW student articles with > reviews indicating multiple problems failed to reach a published state. The > percentages form an almost predictable slope based on experience for > chances of an article becoming published at 100% published for accredited > reporters reviewed with multiple issues, 63% for regulars, 23% for new > contributors, and 0% for University of Wollongong students. > > On the other hand, there is no significant difference between articles > with only one issue identified being on an article subsequently getting > published. 81 reviews identifying only one problem were on articles that > were not subsequently published versus 24 reviews identifying only one > problem on articles that were subsequently published. This puts a > publishing rate at 22.8% for 1 problem reviews versus 22.9% for 2+ problems > reviews. There are differences in cohort performance when only one problem > is identified: 70% of accredited reporter reviews are on articles > subsequently published, 36% for regulars, 11% for new contributors and 4% > for University of Wollongong reporters. With the exception of the > University of Wollongong cohort, reviews that identify only one problem are > less likely to lead to an article eventually arriving at a published state. > > With newsworthiness a major reason for all cohorts as a reason for a > failed review, there are distinct differences in the likelihood of this > problem being overcome based on cohort. Overall, 15% of all articles with > newsworthiness cited as a reason for an article not being published > subsequently becoming published. 40% of accredited reporter reviews > indicating this problem were on articles that eventually became published. > This rate is comparable to regular reporter reviews, with 38% of that > cohort becoming published after a failed review citing newsworthiness. New > reporters have an 11% rate of later publishing. 3% of University of > Wollongong reporters reviews indicating newsworthiness problems reach a > published state. > > 54% of the time when newsworthiness is a problem, a reviewer indicates > some other problem with the article. For articles with multiple problems > including newsworthiness, 93% of the time there is also a style problem, > 46% of the time there is a verifiability problem , 44% of the time there is > a point of view problem and 4% of the time there is a copyright problem. > [image: English Wikinews Average > Submissions.png]<http://en.wikinews.org/wiki/File:English_Wikinews_Average_Submissions.png> > > For articles that are not passed on their first attempt, there are > different continuing progress responses for each cohort. For accredited > reporters, they have one failed review before either submitting > successfully on their second attempt or before abandoning their work. This > suggests that accredited reporters are able to successfully respond to > feedback or understand when an article has systemic problems that will > result in it never being published. New and University of Wollongong > reporters are much more likely to continue to try to resubmit multiple > times, both successfully and unsuccessfully, than their regular reporter > counterparts. 4 new reporters out of 46 submitted their work 4 or more > times unsuccessfully. This contrasts to 3 out of 10 for regular reporters > and 4 out of 18 for University of Wollongong students : 8% to 30% to 22%. > High number of submissions for rereview are unlikely to lead to publication > of the article. Of the articles finally published, only one had failed at > review more than three times, United States deportation policies > challenged in Santa Clara County, > California<http://en.wikinews.org/wiki/United_States_deportation_policies_challenged_in_Santa_Clara_County,_California>, > which had 6 failed reviews before being published. In this particular case, > the article was failed 3 times for copyright reasons, once for > newsworthiness, 3 times for verifiability, 4 times for neutrality and 2 > times for style. > > Accredited reporters were the most successful as a percentage of total > articles with failed initial reviews subsequently getting published, with 8 > articles published after only 1 failed review. Regular reporters had 15 > published articles out of 27 for a 55% success rate, new reporters had 9 > published out of 57 for a 15% success rate, and University of Wollongong > students had 1 out of 21 for a success rate of 4%. As reporters become more > acclimatized, they are more likely to translate failed reviews into > successfully published articles. > > This confirms observational bias that English Wikinews has a high barrier > of entry in terms of adapting to the local review process. It also confirms > that the feedback system for the review system works for established > contributors who have figured out the basics of preparing an article for a > published state. New reporters and University of Wollongong students have > problems that are similar, but new reporters are either more willing to > work through failure to accomplish a goal or more likely to find community > members who are willing to assist them getting the article over the line. > That the percentage of new and University of Wollongong students getting > not publish ready reviews for style suggests they are unfamiliar with the > style guide. How this can be addressed is difficult because it appears as > if they are not reading the style guide. One thought for increasing the > likelihood of getting an article published is to provide a form of > motivation that will encourage a reporter to keep with it until their work > —though not necessarily a specific article— is published. This may need to > be coupled with an improved feedback system, though how this would work > with a cohort of editors who are unmotivated to read existing materials > designed to increase their chances of getting published or interact with > contributors to seek advice in getting published, calls an over reliance on > an improved feedback system as a primary method to increase chances of > getting published into question. > > > > -- > twitter: purplepopple > blog: ozziesport.com > > _______________________________________________ > Wiki-research-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > >
_______________________________________________ Education mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/education
