Posted by Jim Lindgren:
Roth on Guns and Why Historians Struggle Determining "which quantitative
studies are wrong and which are right."--
http://volokh.com/archives/archive_2007_09_16-2007_09_22.shtml#1190098384
In Reviews in American History the country�s leading historian of
homicide, Randolph Roth (OSU) has an article that deals with two
recent disputes in American history: how many guns were listed in
early American probate records and whether the Old West was violent (I
dealt with the latter issue in an [1]earlier post). On these two
related issues, revisionist historians had suggested that there were
few guns and relatively little violence.
Roth first indirectly addresses a question that puzzled me. How could
historians be so easily misled about the quality of Michael
Bellesiles�s fall 2000 book, Arming America?
Roth writes ([2]link may not work for many non-university readers):
As historians, we take pride in our openness to new ideas. That is
why, I think, so many of us were enthusiastic about Michael
Bellesiles�s Arming America. Its thesis�that few Americans used,
owned, or cared about guns before the mid-nineteenth century�was
certainly congenial to scholars who supported gun control, because
it suggested America�s homicide problem was caused by an increase
in gun ownership and the creation of a �gun culture.� But Arming
America had a more fundamental appeal for historians. It proved
that they could make important discoveries, that they could
confound received wisdom with bold hypotheses and careful research,
and that they could play a crucial role in public life.
Unfortunately, Arming America was wrong about early America.
Why do we as a profession have trouble deciding which quantitative
studies are sound and which are not? Why do we mistake faulty
studies for good ones and vice versa? Like everyone else, including
statisticians, we are not very good at judging probabilities.
Humans are proficient at many things, like counting, but as
psychologists have discovered time and again, we have trouble
figuring odds or risks or rates on the fly. Our common sense is a
hindrance. That is why we need tools to help us determine which
quantitative studies are wrong and which are right. Thanks to
statisticians, those tools are at our disposal. It�s simply a
matter of getting acquainted with them in a statistics course,
keeping a few textbooks at hand, and thumbing through them
occasionally for formulas that can help us answer quantitative
questions.
Bellesiles claimed, for instance, using evidence from probate
records, that only 15 to 21 percent of the wealth-holding
population owned guns in the late colonial and early national
period [even that estimate was an increase over Bellesiles's
original estimate in Arming America of 14.7%]. But James Lindgren
and Justin Heather, who reanalyzed a sample of probate records
gathered by Alice Hanson Jones, [3]found that half of all
wealth-holders had guns in 1774. Who is right? It takes only a few
minutes to decide, once the data are in hand and the right formula
is found.
The historical consensus today, even among many of Bellesiles's former
supporters, appears to be that Bellesiles was wrong on most of his
main points about guns in early America (see [4]Fall From Grace). Yet
I still occasionally encounter people who believe that, though
Bellesiles may have made some major mistakes or even may have acted
improperly, his significant errors were limited to the probate data,
leaving his primary thesis of very few guns and little or no gun
culture in early America basically intact. Yet in two widely read
articles ([5]one co-authored with a student, Justin Heather), I
detailed [6]hundreds of mistakes in Arming America that significantly
undercut nearly all of the major claims in the book.
Most historians just found it hard to believe that one of their own
had made such basic errors, especially once the book had been praised
by some of the country�s best historians in the New York Times and the
New York Review of Books (both reviewers later [7]changed their
opinions). And because we had [8]used sophisticated statistical
techniques, some historians were slow to realize that the errors we
identified were easily identifiable in the records and the raw data
itself.
I urged that, before entering the public discussion, historians should
spend an hour in a good research library checking the published
probate records of Providence that [9]Bellesiles had used for some of
his counts. They could quickly see that nearly [10]every claim he made
about those records was false (of course, we did not expect historians
to go into the many unpublished archives to read the original probate
records we also analyzed for other times and places). Although
Bellesiles viciously attacked our work on the Providence records in
the Wall Street Journal and online, he then quietly capitulated on
almost every one of our points regarding Providence, removing most of
his mistaken statements from his book in 2001.
In his 2007 article, Roth recounts guns in one of the sources [11]we
used, the 1774 inventories published by Alice Hanson Jones:
Nearly half of the 919 estates in her sample included guns, even
though at least 28 inventories in her sample were incomplete, and
81 estates were owned by women, 13 of whom owned guns. . . .
The formula shows that there is a 99% chance that the actual
proportion of wealth-holders who owned guns (�) was between .600
and .462. There is only a 1 in 200 chance that the proportion of
wealth-holders who owned guns was below 46.2 percent. Lindgren and
Heather are right: [12]probably half of all wealth-holders owned
guns.
Assuming that Jones�s sample was properly executed�and it has been
verified by subsequent scholars�how likely is it that the actual
proportion of wealth-holders who owned guns was only 15 to 21
percent, as Bellesiles claimed? . . .
Bellesiles�s estimate of 21 percent is 12.0 standard deviations
below the sample proportion (53.1 percent) and his lower estimate
of 15 percent is 14.2 below. That is an enormous distance,
statistically. The chance that Bellesiles�s estimates are correct
is nil�less than 1 in a zillion-zillion. . . .
Thanks to these statistics, we can reject Bellesiles�s findings
with confidence. We do not have to examine his methods or his
selection of counties. His estimates of gun ownership are
impossibly low.
To someone with even a modicum of statistical sense, Bellesiles's
estimates were [13]mathematically impossible.
It is gratifying to have [14]our work replicated and [15]confirmed, an
act for which I heartily thank Randy Roth.
By the way, two participants in the debate over Arming America
([16]Clayton Cramer and [17]Saul Cornell) have published books in the
last 15 months.
References
1. http://volokh.com/archives/archive_2007_08_26-2007_09_01.shtml#1188076990
2. http://muse.jhu.edu/journals/reviews_in_american_history/v035/35.2roth.pdf
3. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
4. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=692421
5. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
6. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=692421
7.
http://www.volokh.com/archives/archive_2005_01_02-2005_01_08.shtml#1104740820
8. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
9. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
10. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
11. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
12. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
13. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
14. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=268583
15. http://muse.jhu.edu/journals/reviews_in_american_history/v035/35.2roth.pdf
16.
http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?z=y&EAN=9781595550699&itm=1
17.
http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?z=y&EAN=9780195147865&itm=3
_______________________________________________
Volokh mailing list
[email protected]
http://lists.powerblogs.com/cgi-bin/mailman/listinfo/volokh
