[ECCO] Re: Wisdom of Crowds

[Francis Heylighen]

Title: Re: Wisdom of Crowds

Gottfried:

I just started listening to " The Wisdom of Crowds"  by James Surowiecki (2004). He describes some old observations by Galton[...]: There was a contest to estimate the weight of an ox and he found that the average of all the guesses was very close to the correct value. Apparently this has been confirmed by many experiments of estimating the number of "jellybeans in a jar".

[...]So I was wondering if this is just some simple statistical property of random guesses or what the current status of research is on these issues (the book seems to be more anecdotal than scientific).

At 20:49 +0700 12/5/04, Korakot Chaovavanich wrote:

Suppose that everyone is likely to guess it correctly (no inherent bias to this
specific problem) and solution is only one dimension (in this case,
just a number).
According to central limit theorem, the eventual average will have the same
mean (u) with deviation reduced by square root of n times.

So, if everybody is likely to guess with the deviation 2 (+-), 16
people averaged
will reduce the deviation to 2/root(16) = 0.5  (approximately).

But the assumption may not hold. For example, if there is systemic
bias from a large
porportion of the crowd, the average will be affected, and they may

not appear so wise.

Averaging the guesses of many different people will come closer and closer to the optimal solution if we assume that individual guesses deviate from the correct one by a random number. In that case, as Korakot pointed out, the statistical "law of large numbers" will lead the deviations to cancel each other out the larger the number of guesses that are averaged. This "wisdom of the crowds" or "collective intelligence" phenomenon has many useful applications.

For example, Craig Kaplan, in the paper he presented at our Global Brain Workshop (http://pespmc1.vub.ac.be/Conf/GB-0-abs.html#Kaplan), used it to successfully forecast stock prices. Norman Johnson from the Los Alamos National Laboratory made a nice simulation demonstrating how the average decision of many agents trying to find their way through a maze is better than that of any individual agent. (the simulation and my interpretation are described below in a quote from my paper on "Collective Intelligence and its Implementation on the Web" (http://pespmc1.vub.ac.be/papers/CollectiveWebIntelligence.pdf)

 However, this assumes that there is no collective bias, i.e. a common factor that makes people systematically overestimate or underestimate the true value. Of course, we know of plenty such cognitive and social biases (e.g. people tend to overestimate the size of an object surrounded by smaller objects, and to underestimate it when surrounded by bigger objects). But since these biases are common to all of us, the "wisdom of the crowds" won't do worse than the guess of an individual.

More dangerous are the situations were the bias is of an inherently social nature, i.e. engendered by the interactions between individuals. For example, social psychologists have shown that groups often take more extreme decisions than individuals, because the individual opinions reinforce each other (conformity), and people feel more confident making a doubtful decision when supported by others. That problem can be avoided by making people vote independently on the preferred outcome, a feature of many group decision support systems.

An important research issue in collective intelligence would be to systematically list all these different individual and social biases, so that we could take them into account, or try to avoid them, when making collective decisions.

In the examples about the weight of an ox or the number of candies in a jar, there probably aren't any specific individual biases (e.g. we can assume that the ox is not surrounded by unusually small members of the same species), while the independent guessing eliminates social biases, so that is why it works. But it would be interesting to do the experiment with many different types of questions and settings to see under what circumstances biases appear.

Since our cognitive apparatus has been finetuned by evolution to be as accurate as possible, conditional to our limited capacity for perception and information processing, shared individual biases are probably the exception rather than the rule. While that is the case, individuals still have idiosyncratic biases, that depend wholly on their personal experience (e.g. having encountered mostly heavy, respectively  light oxen until now). But since everyone's experience is different, these biases can be assumed to be random, and therefore they will be reduced and eventually eliminated through the averaging of an increasing number of guesses.

That would seem to imply that we just need a sufficiently large number of people voting independently to come to good solutions. But that assumes that these people have a sufficient diversity of relevant experiences. Democracy shows that this is not at all obvious. When it comes to electing a new president or party, you cannot use any previous experience to guess how good that president will be when confronted by novel problems. And it is clear to everybody that electors can be make very serious mistakes (remember that Hitler came to power through democratic elections...).

One possible improvement (used among others by Caplan in his latest system, if I understood well) is to use a weighted average, where individuals who have proven to be good guessers in the past (implying a broader experience) get a larger weight in the vote. There are also other methods to differentially "weigh" individual contributions, such as the one implicit in PageRank (Google's algorithm to estimate the importance of a web page), where an individual's "weight" or authority increases the more s/he is trusted by other authorities, or the one being explored by my PhD student Marko Rodriguez (http://www.soe.ucsc.edu/~okram), where an individual's "weight" increases the more people with similar opinions s/he represents. This is a very interesting research area, with crucial relevance to the Global Brain.

Quote from my 1999 paper:

Johnson's (1998; see also Johnson et al. 1998) simulation of collective problem-solving illustrates the power of this intrinsically simple averaging procedure. In the simulation, a number of agents try to find a route through a "maze", from a fixed initial position to a fixed goal position. The maze consists of nodes randomly connected by links. In a first phase, the agents "learn" the layout of the maze by exploring it in a random order until they reach the goal. They do this by building up a preference function which attaches a weight to every link in the network they tried, but such that the last link used (before exiting the maze) in any given node gets the highest weight. In a second, "application" phase, they use this knowledge to find a short route, which now avoids all needless loops and dead-ends encountered during the learning phase. Since different agents have learned different preference functions, they will not all be successful to the same degree, and their best routes can greatly differ in length. However, Johnson (1998) showed that if the preference functions for a large number of agents are averaged, the route selected by that "collective" preference was significantly shorter than the average route found by a typical individual agent. In fact, if the collective consisted of a sufficiently large number of agents, the collective solution was better than the best individual solution.

        This phenomenon might be explained by assuming that the different routes learned by the agents are all variations on the globally shortest route through the maze. Because of the time spent learning how to avoid poor routes, the preferred option at any node is more likely to be the optimal choice than any other choice. However, since the agents have no global understanding of the maze, most of their choices will still be less than optimal. These deviations from the optimum are caused by random factors, and therefore have no systematic bias in any particular direction. Because of the law of large numbers, we can expect these "fluctuations" to cancel each other out when many different preferences are averaged. This will only leave what the different routes have in common, namely their bias towards the optimal solution. It seems that a similar mechanism may apply to human decision-making, under the same condition that there is no systematic bias away from the optimum. Therefore, we may expect the average of the choices made by a large group to be effectively better than the choice made by a random individual.

        Although Johnson's simulation exhibits collective intelligence the way we have defined it, the solutions found by a large collective (of the order of 100 individuals or more) are only somewhat better than the solutions found by a single agent. In real life, it would seldom seem worth the trouble employing a hundred people to solve a problem if one person could find a solution that is almost as good. As could be expected from the law of large numbers, the averaging mechanism provides decreasing returns: the improvement produced by adding a fixed number of individuals to the collective will become smaller as the size of the collective increases.

--

Francis Heylighen     
"Evolution, Complexity and Cognition" research group
Free University of Brussels
http://pespmc1.vub.ac.be/HEYL.html