Re: [Wiki-research-l] Research Showcase April 18, 2018 (11:30 AM PDT| 18:30 UTC)

[Sarah R]

Hi Everyone,

Just a reminder that the Research Showcase will begin in a half hour!

Kindly,

Sarah R.



On Thu, Apr 12, 2018 at 7:30 PM, Sarah R  wrote:

> Hi All,
>
> A quick correction.* "*The Critical Relationship of Volunteer Created
> Wikipedia Content to Large-Scale Online Communities" will be presented by 
> *Nicholas
> Vincent.*
>
> Kind regards,
>
> Sarah R.
>
> On Thu, Apr 12, 2018 at 6:47 PM, Sarah R  wrote:
>
>> Hi Everyone,
>>
>> The next Research Showcase will be live-streamed this Wednesday, April
>> 18, 2018 at 11:30 AM (PDT) 18:30 UTC.
>>
>> YouTube stream: https://www.youtube.com/watch?v=Z1pa-pr6xis
>>
>> As usual, you can join the conversation on IRC at #wikimedia-research.
>> And, you can watch our past research showcases here.
>> 
>>
>> The Critical Relationship of Volunteer Created Wikipedia Content to
>> Large-Scale Online CommunitiesBy *Nate TeBlunthuis*The extensive
>> Wikipedia literature has largely considered Wikipedia in isolation, outside
>> of the context of its broader Internet ecosystem. Very recent research has
>> demonstrated the significance of this limitation, identifying critical
>> relationships between Google and Wikipedia that are highly relevant to many
>> areas of Wikipedia-based research and practice. In this talk, I will
>> present a study which extends this recent research beyond search engines to
>> examine Wikipedia’s relationships with large-scale online communities,
>> Stack Overflow and Reddit in particular. I will discuss evidence of
>> consequential, albeit unidirectional relationships. Wikipedia provides
>> substantial value to both communities, with Wikipedia content increasing
>> visitation, engagement, and revenue, but we find little evidence that these
>> websites contribute to Wikipedia in return. Overall, these findings
>> highlight important connections between Wikipedia and its broader ecosystem
>> that should be considered by researchers studying Wikipedia. Overall, this
>> talk will emphasize the key role that volunteer-created Wikipedia content
>> plays in improving other websites, even contributing to revenue generation.
>>
>>
>> The Rise and Decline of an Open Collaboration System, a Closer LookBy *Nate
>> TeBlunthuis*Do patterns of growth and stabilization found in large peer
>> production systems such as Wikipedia occur in other communities? This study
>> assesses the generalizability of Halfaker etal.’s influential 2013 paper on
>> “The Rise and Decline of an Open Collaboration System.” We replicate its
>> tests of several theories related to newcomer retention and norm
>> entrenchment using a dataset of hundreds of active peer production wikis
>> from Wikia. We reproduce the subset of the findings from Halfaker and
>> colleagues that we are able to test, comparing both the estimated signs and
>> magnitudes of our models. Our results support the external validity of
>> Halfaker et al.’s claims that quality control systems may limit the growth
>> of peer production communities by deterring new contributors and that norms
>> tend to become entrenched over time.
>>
>> Kindest regards,
>>
>> Sarah R. Rodlund
>> Senior Project Coordinator-Product & Technology, Wikimedia Foundation |
>> Hic sunt leones
>> srodl...@wikimedia.org
>>
>>
___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] bias relative to accuracy or populist centrism? (was re: The complete program of Wiki Workshop 2018 is live)

[James Salsman]

Christoph,

Thank you for your reply. I am happy to help and glad you have made
good decisions.

Are you familiar with Priyanka's work on Accuracy Review?
 https://priyankamandikal.github.io/posts/gsoc-2016-project-overview/

Fabian Floeck and

On Wed, Apr 18, 2018 at 7:07 AM, Christoph Hube  wrote:
> Hi James,
>
> thanks a lot for your interest in our work!
>
> The problem of crowdworkers being biased is a problem definitely not to be
> neglected. Majority vote can help to sort out single extremist views of
> workers but if many workers are strongly biased then I agree that this might
> not be enough. We are actually already thinking about methods to improve
> future crowdsourced bias datasets. One way to improve the quality is to have
> a very well defined task that leaves only little room for subjective
> interpretation. For example, instead of letting the workers decide whether a
> statement is biased or not, we asked more specifically whether the statement
> reflects an opinion or contains bias words. Of course, the decision if a
> statement reflects a fact or an opinion is still subjective in many cases.
>
> Given your example it is hard to make a decision (even when being unbiased)
> without having the proper background knowledge. That is why our work mostly
> focuses on language bias, i.e. bias that is introduced through the use of
> judgemental language. Since there are many cases of bias without using
> judgemental language, we are definitely interested to come up with good
> approaches that cover these cases as well. Ideas and suggestions are always
> welcome!
>
> One other thing that we are planning to do for future crowdsourcing jobs is
> to ask workers for their political opinions and to take this background
> information into account when creating ground truth data.
>
> Best regards,
> Christoph
>
>
>
> Am 4/18/2018 um 2:22 PM schrieb James Salsman:
>>>
>>> ... Accepted papers
>>> Christoph Hube and Besnik Fetahu
>>> Detecting Biased Statements in Wikipedia
>>> http://wikiworkshop.org/2018/papers/wikiworkshop2018_paper_1.pdf
>>> ...
>>
>> Hi Christoph and Besnik,
>>
>> Having worked with several thousand of Amazon Mechanical Turkers over
>> the past year, I am not convinced that their opinions of bias, even in
>> aggregate, are not biased.  Did you take any steps to measure the bias
>> against accuracy in your crowdworkers?
>>
>> Here is an example of what I expect they would get wrong:
>>
>> "Tax cuts allow consumers to increase their spending, which boosts
>> aggregate demand."
>>
>> That statement, added by en:User:Bkwillwm in 2012,[1] is still part of
>> the English Wikipedia's Economics article. However, the statement is
>> strictly inaccurate, and heavily biased in favor of trickle-down
>> economics and austerity policy.[2] It and statements like it,
>> pervasive through many if not most of the popular language Wikipedias,
>> directly support increases in income inequality, which in turn is a
>> terrible scourge affecting both health[3] and economic growth.[4]
>>
>> How can you measure whether your crowdworkers are truly unbiased
>> relative to accuracy, instead of just reflecting the
>> propaganda-influenced whims of the populist center?
>>
>> Sincerely,
>> James Salsman
>>
>> [1]
>> https://en.wikipedia.org/w/index.php?title=Economics=prev=511580566
>>
>> [2]
>> https://en.wikipedia.org/wiki/Talk:Economics/Archive_7#Tax_cut_claim_in_Fiscal_policy_section
>>
>> [3] http://talknicer.com/ehip.pdf
>>
>> [4] http://talknicer.com/egma.pdf
>
>

___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] bias relative to accuracy or populist centrism? (was re: The complete program of Wiki Workshop 2018 is live)

[Christoph Hube]

Hi James,

thanks a lot for your interest in our work!

The problem of crowdworkers being biased is a problem definitely not to 
be neglected. Majority vote can help to sort out single extremist views 
of workers but if many workers are strongly biased then I agree that 
this might not be enough. We are actually already thinking about methods 
to improve future crowdsourced bias datasets. One way to improve the 
quality is to have a very well defined task that leaves only little room 
for subjective interpretation. For example, instead of letting the 
workers decide whether a statement is biased or not, we asked more 
specifically whether the statement reflects an opinion or contains bias 
words. Of course, the decision if a statement reflects a fact or an 
opinion is still subjective in many cases.


Given your example it is hard to make a decision (even when being 
unbiased) without having the proper background knowledge. That is why 
our work mostly focuses on language bias, i.e. bias that is introduced 
through the use of judgemental language. Since there are many cases of 
bias without using judgemental language, we are definitely interested to 
come up with good approaches that cover these cases as well. Ideas and 
suggestions are always welcome!


One other thing that we are planning to do for future crowdsourcing jobs 
is to ask workers for their political opinions and to take this 
background information into account when creating ground truth data.


Best regards,
Christoph


Am 4/18/2018 um 2:22 PM schrieb James Salsman:

... Accepted papers
Christoph Hube and Besnik Fetahu
Detecting Biased Statements in Wikipedia
http://wikiworkshop.org/2018/papers/wikiworkshop2018_paper_1.pdf
...

Hi Christoph and Besnik,

Having worked with several thousand of Amazon Mechanical Turkers over
the past year, I am not convinced that their opinions of bias, even in
aggregate, are not biased.  Did you take any steps to measure the bias
against accuracy in your crowdworkers?

Here is an example of what I expect they would get wrong:

"Tax cuts allow consumers to increase their spending, which boosts
aggregate demand."

That statement, added by en:User:Bkwillwm in 2012,[1] is still part of
the English Wikipedia's Economics article. However, the statement is
strictly inaccurate, and heavily biased in favor of trickle-down
economics and austerity policy.[2] It and statements like it,
pervasive through many if not most of the popular language Wikipedias,
directly support increases in income inequality, which in turn is a
terrible scourge affecting both health[3] and economic growth.[4]

How can you measure whether your crowdworkers are truly unbiased
relative to accuracy, instead of just reflecting the
propaganda-influenced whims of the populist center?

Sincerely,
James Salsman

[1] 
https://en.wikipedia.org/w/index.php?title=Economics=prev=511580566

[2] 
https://en.wikipedia.org/wiki/Talk:Economics/Archive_7#Tax_cut_claim_in_Fiscal_policy_section

[3] http://talknicer.com/ehip.pdf

[4] http://talknicer.com/egma.pdf



___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Wiki-research-l] bias relative to accuracy or populist centrism? (was re: The complete program of Wiki Workshop 2018 is live)

[James Salsman]

>... Accepted papers
> Christoph Hube and Besnik Fetahu
> Detecting Biased Statements in Wikipedia
> http://wikiworkshop.org/2018/papers/wikiworkshop2018_paper_1.pdf
>...

Hi Christoph and Besnik,

Having worked with several thousand of Amazon Mechanical Turkers over
the past year, I am not convinced that their opinions of bias, even in
aggregate, are not biased.  Did you take any steps to measure the bias
against accuracy in your crowdworkers?

Here is an example of what I expect they would get wrong:

"Tax cuts allow consumers to increase their spending, which boosts
aggregate demand."

That statement, added by en:User:Bkwillwm in 2012,[1] is still part of
the English Wikipedia's Economics article. However, the statement is
strictly inaccurate, and heavily biased in favor of trickle-down
economics and austerity policy.[2] It and statements like it,
pervasive through many if not most of the popular language Wikipedias,
directly support increases in income inequality, which in turn is a
terrible scourge affecting both health[3] and economic growth.[4]

How can you measure whether your crowdworkers are truly unbiased
relative to accuracy, instead of just reflecting the
propaganda-influenced whims of the populist center?

Sincerely,
James Salsman

[1] 
https://en.wikipedia.org/w/index.php?title=Economics=prev=511580566

[2] 
https://en.wikipedia.org/wiki/Talk:Economics/Archive_7#Tax_cut_claim_in_Fiscal_policy_section

[3] http://talknicer.com/ehip.pdf

[4] http://talknicer.com/egma.pdf

___
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l