Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Hence the one comment on the Wikimedia blog article (http://blog.wikimedia.org/2011/07/15/%E2%80%9Crate-this-page%E2%80%9D-is-coming-to-the-english-wikipedia/) about the survey poll: http://www.vizu.com//poll-results.html?n=138785 50.5% It will be griefed like YouTube comments. -MuZemike On 7/27/2011 5:21 AM, Thomas Morton wrote: I'm cynical about this article feedback system for several reasons, chiefly the worry that it could exacerbate the templating trend of commenting on lots of articles rather than actually improving a few. I'm also slightly circumspect about the idea (though not outright opposed or anything). The issue I've noted is that it is being used as a warfare tool on controversial articles. I've not seen it mentioned on a talk page yet; but one contentious article (on a subject with a large online following, entrenched *readers* on either side of the issue) has had the bars swinging between about 1.5 and 4 in the last week. Not a huge issue, but I suspect that on certain articles the ratings are to be trusted less than usual :) Tom ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 07/16/11 4:42 PM, Dan Dascalescu wrote: On Fri, Jul 15, 2011 at 02:28, Ray Saintongesainto...@telus.net wrote: On 07/14/11 5:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? It's difficult to see any logical connection between an article rating system, and encouraging new editors. After rating an article, there is this link asking Did you know you could edit this page. Just saying that is not enough to inspire people to edit. Ec ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 27 July 2011 08:34, Ray Saintonge sainto...@telus.net wrote: On 07/16/11 4:42 PM, Dan Dascalescu wrote: After rating an article, there is this link asking Did you know you could edit this page. Just saying that is not enough to inspire people to edit. It turns out it is: http://blog.wikimedia.org/2011/07/15/%e2%80%9crate-this-page%e2%80%9d-is-coming-to-the-english-wikipedia/ The feature brings in editors. One of the main Strategic Goals for the upcoming year is to increase the number of active editors contributing to WMF projects. The initial data from the Article Feedback tool suggests that reader feedback could become a meaningful point of entry for future editors. Once users have successfully submitted a rating, a randomly selected subset of them are shown an invitation to edit the page. Of the users that were invited to edit, 17% attempted to edit the page. 15% of those ended up successfully completing an edit. These results strongly suggest that a feedback tool could successfully convert passive readers into active contributors of Wikipedia. A rich text editor could make this path to editing even more promising. While these initial results are certainly encouraging, we need to assess whether these editors are, in fact, improving Wikipedia. We need to measure their level of activity, the quality of their contributions, their longevity, and other characteristics. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Wed, Jul 27, 2011 at 9:08 AM, David Gerard dger...@gmail.com wrote: http://blog.wikimedia.org/2011/07/15/%e2%80%9crate-this-page%e2%80%9d-is-coming-to-the-english-wikipedia/ snip While these initial results are certainly encouraging, we need to assess whether these editors are, in fact, improving Wikipedia. We need to measure their level of activity, the quality of their contributions, their longevity, and other characteristics. There is little point assessing the level of activity, the quality of their contributions, their longevity, and other characteristics for editors that start editing after using the 'Article Feedback tool' if there is no corresponding effort made to assess the *same* characteristics for editors who start editing for other reasons. In particular, I'm referring to other motivations for editing Wikipedia (advertising, pushing an agenda) and editors whose contributions are of poor quality and don't improve over time even when this is pointed out. In other words, are the editors that Wikipedia currently has improving Wikipedia? It is quite conceivable that different sorts of editors are needed at different stages, or are expending their efforts in the wrong places. Carcharoth ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Actually there are a number of other tests we need to run before we know whether Article Rating really is a net positive or a net negative. I hoped they would compare the 100,000 with a control sample to see which gets more edits: http://www.mediawiki.org/wiki/Thread:Talk:Article_feedback/Is_this_a_positive_or_a_negative%3F It should still be possible to keep a control sample of 100,000 articles without the rating system to see if their average quality improves more or less quickly than those with that huge AFT template. And while I can appreciate the excitement that 15% of raters could be tempted to edit, I'd like to see that broken down between: 1 Article improvements 2 Useful talkpage comments 3 Is awesome type comments I'm cynical about this article feedback system for several reasons, chiefly the worry that it could exacerbate the templating trend of commenting on lots of articles rather than actually improving a few. But I accept it is a another great test of the theory that people are basically nice and constructive as opposed to the theory that people are better behaved if they feel they have a reputation at stake. Though judging from the proportion of vandalism amongst IP editors as opposed to registered editors I think we know how that one will pan out. WSC On 27 July 2011 09:08, David Gerard dger...@gmail.com wrote: On 27 July 2011 08:34, Ray Saintonge sainto...@telus.net wrote: On 07/16/11 4:42 PM, Dan Dascalescu wrote: After rating an article, there is this link asking Did you know you could edit this page. Just saying that is not enough to inspire people to edit. It turns out it is: http://blog.wikimedia.org/2011/07/15/%e2%80%9crate-this-page%e2%80%9d-is-coming-to-the-english-wikipedia/ The feature brings in editors. One of the main Strategic Goals for the upcoming year is to increase the number of active editors contributing to WMF projects. The initial data from the Article Feedback tool suggests that reader feedback could become a meaningful point of entry for future editors. Once users have successfully submitted a rating, a randomly selected subset of them are shown an invitation to edit the page. Of the users that were invited to edit, 17% attempted to edit the page. 15% of those ended up successfully completing an edit. These results strongly suggest that a feedback tool could successfully convert passive readers into active contributors of Wikipedia. A rich text editor could make this path to editing even more promising. While these initial results are certainly encouraging, we need to assess whether these editors are, in fact, improving Wikipedia. We need to measure their level of activity, the quality of their contributions, their longevity, and other characteristics. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 27/07/2011, Thomas Morton morton.tho...@googlemail.com wrote: The issue I've noted is that it is being used as a warfare tool on controversial articles. I've not seen it mentioned on a talk page yet; but one contentious article (on a subject with a large online following, entrenched *readers* on either side of the issue) has had the bars swinging between about 1.5 and 4 in the last week. Not a huge issue, but I suspect that on certain articles the ratings are to be trusted less than usual :) The average is not very trustworthy. But the bar graph of how many people have actually voted each score is a bit more interesting. If it's bimodal, with two peaks, then that often tells you something. But the tool doesn't currently give you that, it probably should. Tom -- -Ian Woollard ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Re Ray's comment: It's difficult to see any logical connection between an article rating system, and encouraging new editors. I'm not convinced that we fully understand all the different things that made Wikipedia work, and especially what are the elements that didn't motivate us individually but are important to others. Two aspects that really attracted me to the pedia were firstly the SoFixIt approach, rather than write a paragraph somewhere explaining why I thought something was wrong I could just make a change and see if others accepted it. Secondly the correction of my errors. Instead of someone red-penning my work I much prefer that they just fix isolated errors - if I'm watchlisting the article I have the opportunity to see what gets changed. Similarly I don't whinge at people who make the typos that I fix, I just fix them. I suspect this is part of the motivation for many of our editors who edit in a language other than their native one, editing Wikipedia gives them an opportunity to practice a language in a collaborative environment where their mistakes are fixed in a non-judgmental way. I'm not convinced that such editors would benefit if there was a switch from collaborative editing where they have a chance to improve their English to article assessment where they are marked down without getting specific feedback as to how to improve. So for me there are two close and logical connections between article rating and editing. But I appreciate that others may see them as unconnected, and I agree that without testing we can't easily find out how important these connections are or indeed whether this will divert people from collaborative editing to critiquing or simply attract additional involvement.. My concern about the article rating system is that it could undermine two important parts of what I perceive to be our foundations. I'm reassured that there will be testing to see whether this does in practice what I suspect it will, and indeed whether we can do sufficient call to actions to persuade some of our new article assessors to actually edit. WSC On 15 July 2011 10:28, Ray Saintonge sainto...@telus.net wrote: On 07/14/11 5:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? It's difficult to see any logical connection between an article rating system, and encouraging new editors. Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; If templates were subject to a similar rating system as articles we would soon see which are being ignored by users, and are thus of no value. So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. This dream has been around since the stone age. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. I seriously doubt that it will head in that direction. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. It's not a problem if they do. If many readers do this for a single article it's worth paying attention to these articles. A claim from a single person can be suspected eccentric. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. It's all a matter of statistical trends, and for this a 100-point scale would have been more useful than a 5-point scale. I actually suggested a 10-point scale many years ago. The first statistical measure that should develop is a cumulative rating for all articles. The mean in that will be the measure of the average article, and any article falling within a certain deviation from that could be judged average. As overall quality of WP increases so too will the average rating, but only extremely slowly. Other measures could be developed from there. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 07/14/11 5:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? It's difficult to see any logical connection between an article rating system, and encouraging new editors. Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; If templates were subject to a similar rating system as articles we would soon see which are being ignored by users, and are thus of no value. So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. This dream has been around since the stone age. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. I seriously doubt that it will head in that direction. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. It's not a problem if they do. If many readers do this for a single article it's worth paying attention to these articles. A claim from a single person can be suspected eccentric. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. It's all a matter of statistical trends, and for this a 100-point scale would have been more useful than a 5-point scale. I actually suggested a 10-point scale many years ago. The first statistical measure that should develop is a cumulative rating for all articles. The mean in that will be the measure of the average article, and any article falling within a certain deviation from that could be judged average. As overall quality of WP increases so too will the average rating, but only extremely slowly. Other measures could be developed from there. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. This isn't a problem either. The number of ratings given is just as important as what those ratings are. It should be reported right along with the rating on the article page Users could then be reminded that a small number of ratings is just not statistically significant; they could even be color-coded to that effect. Short samples are also more volatile. They would easily be driven into the top or bottom decile of the data, and that alone would bring attention to them. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? This FUD gives undue weight to sockpuppetry or other hostile editing. Ideally such practices should be marginalised to a point where they don't matter. Mounting a successful campaign to influence the rating of an article would take a tremendous amount of sustained effort. I played with trying to affect the page views of one of the Bomis girl articles in the early days by going repeatedly to that page; the effects were minimal. Now, with a much bigger encyclopedic corpus this would be proportionally more difficult. Random unsigned votes are perfectly consistent with wikiness, and will also trend toward statistical norms. Building safeguards against agenda based ratings would be a waste of time and effort. Ec ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 07/14/11 10:01 AM, MuZemike wrote: However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, Such an abomination on an article would stick out like a sore thumb, begging to be corrected. Ec ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerard dger...@gmail.com wrote: On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
A couple of fair points. However, I would disagree that everyone is interested in editing or improving the encyclopedia; some are perfectly content on reading the content therein and, if given the chance, say what they think about out (not necessarily on Wikipedia, but could be anywhere on the Web). I mean, we cannot point a gun to their head and make them edit something, as this is a purely volunteer project. However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. -MuZemike On 7/14/2011 7:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerarddger...@gmail.com wrote: On 14 July 2011 00:40, Howie Funghf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. There are various ways to mitigate these effects, e.g. cut off the top and bottom 10% of ratings when calculating the displayed numbers. But the essential problem is [[Goodhart's law]]: once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting policy, then it will lose the information content that would qualify it to play such a role. So the answer is not to take the ratings *too* seriously for purposes of writing the encyclopedia. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Cutting off the top or bottom 10% wouldn't work if 4chan targets the articles written by one of our editors, if anything the non4chan votes will be in the top 10% that you discard. To be honest I'm not particularly worried if people canvass their mates to give straight 5s to an obscure article that only a few hundred people will ever notice. I would anticipate that will happen whenever someone files an AFD on an article that is of interest to a particular fansite, and if anything it will be less disruptive to have a bunch of fans boost the articles ratings than it will be to deal with those same fans at the AFD. The positive ratings that really matter to editors on this site are things like FA and GA and I don't see this system replacing that. I'm more concerned that this will give people an underhand way to get back at an editor they dislike. Unless I'm missing something and this has already been anticipated, this system needs a mechanism to spot when a group of editors anonymously rate everything another editor has done as rubbish. WSC On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: A couple of fair points. However, I would disagree that everyone is interested in editing or improving the encyclopedia; some are perfectly content on reading the content therein and, if given the chance, say what they think about out (not necessarily on Wikipedia, but could be anywhere on the Web). I mean, we cannot point a gun to their head and make them edit something, as this is a purely volunteer project. However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. -MuZemike On 7/14/2011 7:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerarddger...@gmail.com wrote: On 14 July 2011 00:40, Howie Funghf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
WereSpielChequers, thanks for the great feedback. We are going to analyze the overall effect of AFT on article edit volume. More generally, for all retention features we are currently deploying, we will be studying both how they affect edit activity at article-level and how they affect individual editor contributions. Updates will be posted as usual on the AFT research page http://www.mediawiki.org/wiki/Article_feedback/Research Dario On Jul 14, 2011, at 5:56 AM, WereSpielChequers wrote: Do we have stats yet that measure whether this is encouraging editing, or diverting even more people from improving the pedia to critiquing it? Remember there is a risk that this could exacerbate the templating trend. Just as we need to value edits that fix problems and remove templates above edits that add to the hundreds of thousands of maintenance templates on the pedia; So we need to value a talkpage comment that explains why someone has a specific concern about an article over a bunch of feedback that says people like or dislike an article without indicating why. Better still we should be encouraging readers to improve articles that they see as flawed. So we need to measure this tool in terms of its success at getting readers to edit, not in terms of its success at getting readers to rate articles. I hope it is successful, and I'm happy to take the long view and measure a trial over months to see how effectively we convert article raters into article editors. But we do need to be prepared to remove this if it has a net effect of diverting potential editors into merely rating articles for others to fix. We also need to be careful how we compare this 374k to the other 90%, not least because with 3,682,158 articles on En wiki as I write, 374k is about 6k more than a random 10% sample would be. We also need to learn from one of the lessons of the Strategy wiki where we had a similar rating system. Many of the proposals there had so few ratings that they were close to being individual views and few had sufficient responses to be genuinely collective to the point where one maverick couldn't skew them - even without sockpuppetry. On average our articles get one or two edits a month, many get far less. I would not be surprised if 100,000 of the 374k in the trial had less than ten ratings even if trialled for a couple of months. Lastly we need to be prepared for sockpuppetry, especially as these are random unsigned votes with no rationale. Can we have assurances that something is being built into the scheme to combat this? Regards WereSpielChequers On 14 July 2011 10:08, David Gerard dger...@gmail.com wrote: On 14 July 2011 00:40, Howie Fung hf...@wikimedia.org wrote: Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. Is there anywhere we can read articles' ratings? - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Jul 14, 2011, at 10:11 AM, David Gerard wrote: On 14 July 2011 18:01, MuZemike muzem...@gmail.com wrote: However, you've made a good point there about gaming the system and intentionally trying to garner high ratings. For example, one could create a horrid piece of crap article which would have no chance of staying on Wikipedia and canvass his/her buddies to flood said piece of crap with 5.0's across the board. This thing precisely happens from time to time on YouTube. I don't know how this could be prevented, but I acknowledge that even this feedback system, as with all others, are not perfect and comes with systemic flaws. There are various ways to mitigate these effects, e.g. cut off the top and bottom 10% of ratings when calculating the displayed numbers. But the essential problem is [[Goodhart's law]]: once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting policy, then it will lose the information content that would qualify it to play such a role. So the answer is not to take the ratings *too* seriously for purposes of writing the encyclopedia. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On 14 July 2011 18:22, WereSpielChequers werespielchequ...@gmail.com wrote: Cutting off the top or bottom 10% wouldn't work if 4chan targets the articles written by one of our editors, if anything the non4chan votes will be in the top 10% that you discard. [...] Unless I'm missing something and this has already been anticipated, this system needs a mechanism to spot when a group of editors anonymously rate everything another editor has done as rubbish. Again, this is a problem of taking the numbers too seriously. There is *no* system *anywhere* that can't be gamed. Before saying we need to deal with exploit x, we should see what exploits actually happen. Making all the rating data publicly available for analysis (with no usernames or IPs attached, of course) is a first step. Before proposing solutions to problems in the data, look at the data ;-) - d. - d. ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Thu, Jul 14, 2011 at 2:31 PM, David Gerard dger...@gmail.com wrote: Making all the rating data publicly available for analysis (with no usernames or IPs attached, of course) is a first step. Before proposing solutions to problems in the data, look at the data ;-) A sound recommendation from the psychology literature on problem solving. To quote Eliezer Yudkowsky ( http://lesswrong.com/lw/ka/hold_off_on_proposing_solutions/ ) quoting Robyn Dawes (_Rational Choice in an Uncertain World_) expanding Norman R. F. Maier: ...when a group faces a problem, the natural tendency of its members is to propose possible solutions as they begin to discuss the problem. Consequently, the group interaction focuses on the merits and problems of the proposed solutions, people become emotionally attached to the ones they have suggested, and superior solutions are not suggested. Maier enacted an edict to enhance group problem solving: Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any. It is easy to show that this edict works in contexts where there are objectively defined good solutions to problems. -- gwern http://www.gwern.net ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
Re: [WikiEN-l] Article Feedback - Ramp up to 10% of Articles
On Thu, Jul 14, 2011 at 18:22, WereSpielChequers werespielchequ...@gmail.com wrote: To be honest I'm not particularly worried if people canvass their mates to give straight 5s to an obscure article that only a few hundred people will ever notice. I would anticipate that will happen whenever someone files an AFD on an article that is of interest to a particular fansite, and if anything it will be less disruptive to have a bunch of fans boost the articles ratings than it will be to deal with those same fans at the AFD. The positive ratings that really matter to editors on this site are things like FA and GA and I don't see this system replacing that. I think the important think about the article feedback tool is that hopefully it will allow WikiProjects to prioritise article improvements. Let's say you are involved with WikiProject Philosophy: it'd be really useful to get a list of all the philosophy articles with article feedback statistics mixed in. If we have an article that is getting very variable ratings, going up and down all over the place, that's a useful measure for having passionate readers. If there's an article with organically occurring high ratings from the readers, that is something the WikiProject should collectively consider pushing towards Good Article or Featured Article. The problem is we get the 'Bieber problem': people voting on the basis of their views of the article's subject rather than the article, so people who love Justin Bieber upvote it and people who loathe him downvote it, even though we are asking whether they think the article is good. The negative side is worse here: people downvoting the article as a kind of 'delete' vote - they think that saying the article is poor quality because we are giving too much coverage to a subject we shouldn't be giving coverage to. There is a good side though: we can use the different categories quite usefully. If we have an article that is highly rated in three of the four criteria but not so well rated in another, that's potentially something we could flag up to WikiProjects as an area for improvement. The article feedback tool is just that... a tool we can use to feed back into the project. It shouldn't ever be an end in itself. -- Tom Morris http://tommorris.org/ ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
[WikiEN-l] Article Feedback - Ramp up to 10% of Articles
Everyone, Just wanted to pass along a note to let everyone know that earlier today, we ramped up the Article Feedback Tool to 10% of articles on the English Wikipedia. That brings the total to approximately 374K articles with the tool deployed. We'll be posting additional information on the Foundation blog soon, but I wanted to keep everyone in the loop regarding the ramp-up earlier today. Howie ___ WikiEN-l mailing list WikiEN-l@lists.wikimedia.org To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
