Title: Re: Wisdom of Crowds
Gottfried:
I just started listening to The Wisdom of
Crowds by James Surowiecki (2004). He describes some old
observations by Galton[...]: There was a contest to estimate the
weight of an ox and he found that the average of all the guesses was
very close to the correct value. Apparently this has been confirmed
by many experiments of estimating the number of jellybeans in a
jar.
[...]So I was wondering if this is just some simple
statistical property of random guesses or what the current status of
research is on these issues (the book seems to be more anecdotal than
scientific).
At 20:49 +0700 12/5/04, Korakot Chaovavanich wrote:
Suppose that everyone is likely to guess
it correctly (no inherent bias to this
specific problem) and solution is only one dimension (in this
case,
just a number).
According to central limit theorem, the eventual average will have
the same
mean (u) with deviation reduced by square root of n times.
So, if everybody is likely to guess with the deviation 2 (+-), 16
people averaged
will reduce the deviation to 2/root(16) = 0.5
(approximately).
But the assumption may not hold. For example, if there is systemic
bias from a large
porportion of the crowd, the average will be affected, and they
may
not appear so wise.
Averaging the guesses of many different people will come closer
and closer to the optimal solution if we assume that individual
guesses deviate from the correct one by a random number. In that
case, as Korakot pointed out, the statistical law of large
numbers will lead the deviations to cancel each other out the
larger the number of guesses that are averaged. This wisdom of
the crowds or collective intelligence phenomenon
has many useful applications.
For example, Craig Kaplan, in the paper he presented at our
Global Brain Workshop (http://pespmc1.vub.ac.be/Conf/GB-0-abs.html#Kaplan), used it to successfully forecast stock prices.
Norman Johnson from the Los Alamos National Laboratory made a nice
simulation demonstrating how the average decision of many agents
trying to find their way through a maze is better than that of any
individual agent. (the simulation and my interpretation are described
below in a quote from my paper on Collective Intelligence and
its Implementation on the Web
(http://pespmc1.vub.ac.be/papers/CollectiveWebIntelligence.pdf)
However, this assumes that there is no collective bias,
i.e. a common factor that makes people systematically overestimate or
underestimate the true value. Of course, we know of plenty such
cognitive and social biases (e.g. people tend to overestimate the
size of an object surrounded by smaller objects, and to underestimate
it when surrounded by bigger objects). But since these biases are
common to all of us, the wisdom of the crowds won't do
worse than the guess of an individual.
More dangerous are the situations were the bias is of an
inherently social nature, i.e. engendered by the interactions between
individuals. For example, social psychologists have shown that groups
often take more extreme decisions than individuals, because the
individual opinions reinforce each other (conformity), and people
feel more confident making a doubtful decision when supported by
others. That problem can be avoided by making people vote
independently on the preferred outcome, a feature of many group
decision support systems.
An important research issue in collective intelligence would be
to systematically list all these different individual and social
biases, so that we could take them into account, or try to avoid
them, when making collective decisions.
In the examples about the weight of an ox or the number of
candies in a jar, there probably aren't any specific individual
biases (e.g. we can assume that the ox is not surrounded by unusually
small members of the same species), while the independent guessing
eliminates social biases, so that is why it works. But it would be
interesting to do the experiment with many different types of
questions and settings to see under what circumstances biases
appear.
Since our cognitive apparatus has been finetuned by evolution to
be as accurate as possible, conditional to our limited capacity for
perception and information processing, shared individual biases are
probably the exception rather than the rule. While that is the case,
individuals still have idiosyncratic biases, that depend wholly on
their personal experience (e.g. having encountered mostly heavy,
respectively light oxen until now). But since everyone's
experience is different, these biases can be assumed to be random,
and therefore they will be reduced and eventually eliminated through
the averaging of an increasing number of guesses.
That would seem to imply that we just need a sufficiently large
number of people voting independently to come to good solutions. But
that assumes that these people have a sufficient diversity of
relevant experiences. Democracy shows that this is not at all
obvious.
