Since my first attempt was so crummy and since I think you are still missing the point, let me try again re the Literary Digest and its fundamental lesson -- as I see it. :-)
I will adopt the conventional view (see, for example, Freedman, Pisani, and Purvees, Statistics, 3rd ed., Chap 19: Sample Surveys), but if you really hate this story, try M. C. Bryson, "The Literary Digest poll: making of a statistical myth," American Statistician, 1976 (which, to their credit, Freedman, et al. cite and I'm sure many cynics out there will really like -- it includes the charge that George Gallup grossly misrepresented how the Digest actually polled).
The Literary Digest was subscribed to by millions of people in 1936. This prestigious magazine had been predicting presidential elections -- correctly -- since 1916. They predicted Roosevelt would lose to Landon in 1936 by a wide margin. They said Roosevelt would get only 43% of the vote. In fact, FDR won by a landslide with 62%.
The reasons for the failure could be argued all day, but what makes this story (be it myth or not) powerful is that a young George Gallup predicted not only the election, but also the Digest's own result!
The Digest procedure was to mail questionnaires to 10 million subscribers. In 1936, 2.4 million were returned -- that's a HUGE sample. Gallup used a sample of 50,000 to predict a Roosevelt win with 56% of the vote and he used a sample of 3,000, chosen at random from the Digest's own list, to predict the Digest's own result before they had processed the full returns. (Gallup's prediction missed by a percentage point.)
So let's turn to two fundamental lessons of the Digest saga:
1) You do not need a HUGE sample to predict how tens or hundreds of millions of people will vote. This is counterintuitive. Most people think, "If I'm going to predict the result from a county with 40,000 people, a sample of 3,000 will be OK, but no way a sample of 3,000 could predict the result of an election with 100 million voters." This is wrong. It's hard to see that the size of the population fades in importance (to cover myself for the correction factor needed when sampling without replacement) rather quickly.
2) A HUGE sample does not guarantee accuracy. Most people would argue, "Geez, with 2.4 million responses, it's gotta be close to right. And if not, 2.4 million, no matter how chosen has to be better than 3,000!" This is way wrong. 3,000 truly randomly sampled blows away 2.4 million non-randomly sampled.
I hope that is a better summary and that you can better see my perspective on this. I want to communicate that polling is hard to do AND that it is remarkably effective. You can have great fun with the former. Here's an example that a colleague of mine found and that we use in our team-taught stats course:
A more extreme example is given by Timur Kuran. In 1990, a poll by Washington Post-ABC News predicted that Daniel Ortega of the Sandinista Party would win the Nicaraguan election. In fact, he lost to Chamorro 55% to 41%. Why was the poll so wrong? Kuran explains:
- An ingenious experiment by Katherine Bishoping and Howard Schuman
(1992) points to the source of the confusion. A few weeks before
the election, Bishoping and Schuman conducted 300 interviews in and
around Managua. In one-third of the interviews the interviewer used
a pen featuring the red and black colors of the Sandinista Party and the
inscription �Daniel Presidente.� In another third the interviewer used a
pen featuring the blue and white colors of the opposition and the
inscription �UNO.� And in the remaining third the interviewer used a
neutrally coloured pen with no lettering. Interviewers did not draw
attention to their pens or make claims about their own political
sympathies. Yet, the results show clearly that the pens influenced
the respondents. When the interviewer held a Sandinista pen, the
respondents voiced support for Ortega by a 63% to 37% margin.
Ortega also came out ahead in the neutral-pen condition, 60% to
40%. However, when the interviewer held a UNO pen, Chamorro was the
winner by 56% to 44%. The UNO-pen condition thus came close
to predicting the election outcome, whereas the Sandinista-pen condition
replicated the highly inaccurate pre-election polls.
Unfortunately, what this example -- and others that show the pitfalls of polling (back to phones!) -- does is leave the message that polling doesn't work. That's the wrong message. After every negative example I will tell my class, "This does not mean polling stinks -- it means it's hard to do well."
Humberto Barreto
x6315
